1 files changed, 194 insertions, 0 deletions
diff --git a/notes/03_paper_outline.md b/notes/03_paper_outline.md
new file mode 100644
index 0000000..829226d
--- /dev/null
+++ b/notes/03_paper_outline.md
@@ -0,0 +1,194 @@
+# Paper Outline
+
+## Title
+
+Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons
+
+## Abstract Sketch
+
+Feedback alignment replaces exact backpropagation feedback weights with fixed random matrices. While this can support learning, it introduces an additional alignment burden: forward weights must simultaneously solve the task and remain compatible with random feedback pathways. We formalize this burden through angular, log-volume, and task-projected functional capacity. For MLPs, we derive exact distributional laws for random feedback alignment, scaling laws for multilayer capacity accumulation, and a prior-free minimax bound showing that no feedback initialization can beat isotropic random feedback without information about the forward weights or task. We validate these predictions through static distribution tests and large trajectory ensembles comparing FA and BP.
+
+## 1. Introduction
+
+Motivation:
+
+- BP has exact feedback symmetry.
+- FA replaces exact feedback with fixed random matrices.
+- FA can work, but the extra alignment burden is not quantitatively characterized.
+
+Core question:
+
+> How much capacity does random feedback alignment consume, how does it scale, and when does it become a functional FA/BP gap?
+
+Claims to avoid:
+
+- Do not claim architecture alone predicts final test accuracy.
+- Do not claim the capacity proxy is identical to generalization gap.
+
+Main contributions:
+
+1. Capacity formalization.
+2. Scaling law and redundancy-exhaustion phase transition.
+3. Prior-free minimax bound for feedback initialization.
+4. Distributional and trajectory-level empirical validation.
+
+## 2. Setup
+
+Define:
+
+- MLP architecture.
+- BP feedback operator.
+- FA feedback matrix.
+- Layerwise alignment statistic \(Q_l\).
+- Dense isotropic initialization assumptions.
+
+## 3. Distributional Alignment Law
+
+Theorem:
+
+\[
+Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
+\]
+
+Consequences:
+
+- \(\mathbb E[Q_l]=1/D_l\).
+- \(D_lQ_l\Rightarrow \chi_1^2\).
+- Tail probability via incomplete beta.
+
+## 4. Capacity Formalization
+
+Define:
+
+- Angular statistic.
+- Log-volume cost:
+
+\[
+C_l(q)=-\log\Pr(Q_l\ge q).
+\]
+
+- Functional capacity via task Jacobian \(J\).
+
+Explain why parameter-volume capacity and function capacity differ.
+
+## 5. Multilayer Scaling and Redundancy Exhaustion
+
+Multilayer:
+
+\[
+C_{\mathrm{all}}=\sum_l C_l(q_l).
+\]
+
+Equal-width scaling:
+
+\[
+C_{\mathrm{all}}=\Theta(Ln^2)
+\]
+
+for fixed \(q\).
+
+Functional phase transition:
+
+\[
+\Delta d_{\mathrm{hard}}
+=
+\max(0,k-(P-d)).
+\]
+
+Soft overlap:
+
+\[
+\mathbb E[\Delta d_{\mathrm{soft}}]\approx \frac{kd}{P}.
+\]
+
+## 6. Prior-Free Minimax Bound
+
+Theorem:
+
+\[
+\sup_\mu
+\inf_{\|a\|=1}
+\mathbb E_\mu[(a^\top \hat b)^2]
+=
+\frac1D.
+\]
+
+Interpretation:
+
+- Isotropic feedback is minimax optimal without prior information.
+- Scale affects norm dynamics, not angular bound.
+- Orthogonality affects conditioning, not the prior-free angular limit.
+- Sparsity and low rank trade coverage for efficiency.
+
+Prior-aware corollary:
+
+\[
+\mathbb E_{a,B}[(a^\top \hat b)^2]
+=
+\operatorname{tr}(\Sigma_A M_\mu).
+\]
+
+## 7. Trajectory Bridge
+
+Introduce local approximation:
+
+\[
+\epsilon_t(B)=\tilde g_t(B)-g_t.
+\]
+
+\[
+\delta\theta_T(B)
+\approx
+-\eta
+\sum_{t<T}
+\Phi_{T,t+1}\epsilon_t(B).
+\]
+
+\[
+\Delta L_T(B)
+\approx
+\frac12
+\delta\theta_T(B)^\top H_T\delta\theta_T(B).
+\]
+
+Use this to motivate trajectory experiments.
+
+## 8. Experiments
+
+Static distribution:
+
+- beta law validation.
+- initialization variants.
+
+Scaling:
+
+- depth and width sweeps.
+- threshold regimes.
+
+Trajectory:
+
+- FA vs BP on synthetic and simple vision MLPs.
+- \(Q_l(t)\), \(\Gamma_t\), and \(\Delta L_T\).
+- redundancy-exhaustion transition.
+
+## 9. Discussion
+
+Key points:
+
+- Random feedback imposes a measurable alignment burden.
+- Log-capacity cost scales linearly; raw feasible volume decays geometrically.
+- Overparameterization can hide parameter-volume cost until redundancy is exhausted.
+- Isotropic random feedback is prior-free minimax optimal for angular alignment.
+- Better initialization requires a prior over weights, task-sensitive directions, or desired conditioning.
+
+Limitations:
+
+- MLP focus.
+- Local functional capacity approximations.
+- Final test accuracy depends on data and optimization dynamics.
+
+## 10. Conclusion
+
+Return to the main claim:
+
+We do not exactly predict final FA accuracy from architecture alone. We provide distributional capacity bounds that quantify the extra alignment burden FA introduces and show when that burden should become visible as a functional gap.