path: root/notes/03_paper_outline.md

# Paper Outline

## Title

Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons

## Abstract Sketch

Feedback alignment replaces exact backpropagation feedback weights with fixed random matrices. While this can support learning, it introduces an additional alignment burden: forward weights must simultaneously solve the task and remain compatible with random feedback pathways. We formalize this burden through angular, log-volume, and task-projected functional capacity. For MLPs, we derive exact distributional laws for random feedback alignment, scaling laws for multilayer capacity accumulation, and a prior-free minimax bound showing that no feedback initialization can beat isotropic random feedback without information about the forward weights or task. We validate these predictions through static distribution tests and large trajectory ensembles comparing FA and BP.

## 1. Introduction

Motivation:

- BP has exact feedback symmetry.
- FA replaces exact feedback with fixed random matrices.
- FA can work, but the extra alignment burden is not quantitatively characterized.

Core question:

> How much capacity does random feedback alignment consume, how does it scale, and when does it become a functional FA/BP gap?

Claims to avoid:

- Do not claim architecture alone predicts final test accuracy.
- Do not claim the capacity proxy is identical to generalization gap.

Main contributions:

1. Capacity formalization.
2. Scaling law and redundancy-exhaustion phase transition.
3. Prior-free minimax bound for feedback initialization.
4. Distributional and trajectory-level empirical validation.

## 2. Setup

Define:

- MLP architecture.
- BP feedback operator.
- FA feedback matrix.
- Layerwise alignment statistic \(Q_l\).
- Dense isotropic initialization assumptions.

## 3. Distributional Alignment Law

Theorem:

\[
Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]

Consequences:

- \(\mathbb E[Q_l]=1/D_l\).
- \(D_lQ_l\Rightarrow \chi_1^2\).
- Tail probability via incomplete beta.

## 4. Capacity Formalization

Define:

- Angular statistic.
- Log-volume cost:

\[
C_l(q)=-\log\Pr(Q_l\ge q).
\]

- Functional capacity via task Jacobian \(J\).

Explain why parameter-volume capacity and function capacity differ.

## 5. Multilayer Scaling and Redundancy Exhaustion

Multilayer:

\[
C_{\mathrm{all}}=\sum_l C_l(q_l).
\]

Equal-width scaling:

\[
C_{\mathrm{all}}=\Theta(Ln^2)
\]

for fixed \(q\).

Functional phase transition:

\[
\Delta d_{\mathrm{hard}}
=
\max(0,k-(P-d)).
\]

Soft overlap:

\[
\mathbb E[\Delta d_{\mathrm{soft}}]\approx \frac{kd}{P}.
\]

## 6. Prior-Free Minimax Bound

Theorem:

\[
\sup_\mu
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]

Interpretation:

- Isotropic feedback is minimax optimal without prior information.
- Scale affects norm dynamics, not angular bound.
- Orthogonality affects conditioning, not the prior-free angular limit.
- Sparsity and low rank trade coverage for efficiency.

Prior-aware corollary:

\[
\mathbb E_{a,B}[(a^\top \hat b)^2]
=
\operatorname{tr}(\Sigma_A M_\mu).
\]

Distributional validation target:

\[
A(a)=a^\top M_\mu a
=
\frac{\sum_i \lambda_i G_i}{\sum_i G_i},
\qquad
G_i\sim\chi^2_1.
\]

Use this to compare predicted target-coverage distributions for isotropic,
subspace, axis, and anisotropic feedback initializations.

## 7. Trajectory Bridge

Introduce local approximation:

\[
\epsilon_t(B)=\tilde g_t(B)-g_t.
\]

\[
\delta\theta_T(B)
\approx
-\eta
\sum_{t<T}
\Phi_{T,t+1}\epsilon_t(B).
\]

\[
\Delta L_T(B)
\approx
\frac12
\delta\theta_T(B)^\top H_T\delta\theta_T(B).
\]

Use this to motivate trajectory experiments.

First bridge experiment:

- Predict a full loss-gap distribution by integrating FA/BP gradient mismatch along the BP path.
- Compare raw, standardized, and moment-matched predicted gap distributions against large FA trajectory ensembles.
- Current empirical status: standardized shape matches better than raw scale, motivating a response-operator refinement.

## 8. Experiments

Static distribution:

- beta law validation.
- initialization variants.

Scaling:

- depth and width sweeps.
- threshold regimes.
- large empirical tail calibration for \(C(q)=-\log P(Q\ge q)\).
- observed capacity distribution matching: \(S_l\sim\mathrm{Exp}(1)\), \(\sum_l S_l\sim\mathrm{Gamma}(L,1)\).

Trajectory:

- FA vs BP on synthetic and simple vision MLPs.
- \(Q_l(t)\), \(\Gamma_t\), and \(\Delta L_T\).
- redundancy-exhaustion transition.

Initial implementation note:

- Start with synthetic full-batch regression and no-bias MLPs.
- Record same-point gradient alignment by computing BP and FA gradients at the FA weights.
- Treat early trajectory runs as instrumentation checks before drawing distributional conclusions.
- Use within-architecture correlations as the first trajectory validation target; pooled correlations can be misleading because architecture-level task difficulty and BP final loss vary.

## 9. Discussion

Key points:

- Random feedback imposes a measurable alignment burden.
- Log-capacity cost scales linearly; raw feasible volume decays geometrically.
- Overparameterization can hide parameter-volume cost until redundancy is exhausted.
- Isotropic random feedback is prior-free minimax optimal for angular alignment.
- Better initialization requires a prior over weights, task-sensitive directions, or desired conditioning.

Limitations:

- MLP focus.
- Local functional capacity approximations.
- Final test accuracy depends on data and optimization dynamics.

## 10. Conclusion

Return to the main claim:

We do not exactly predict final FA accuracy from architecture alone. We provide distributional capacity bounds that quantify the extra alignment burden FA introduces and show when that burden should become visible as a functional gap.