path: root/notes/24_effective_capacity_literature_and_gap_theory.md

# Effective Capacity Literature and Gap Theory

This note collects literature that supports the intuition that large effective
capacity can absorb difficult training signals, then clarifies what our revised
theory should and should not claim.

## Literature to Position Against

### Zhang et al.: effective capacity and random labels

Zhang, Bengio, Hardt, Recht, and Vinyals use random-label tests to argue that
deep networks have enormous effective capacity. Their central observation is
that standard networks can fit random labels to zero training error.

Useful positioning:

```text
If neural networks can memorize arbitrary random labels, it is tempting to
believe that sufficiently overparameterized networks can also absorb random
feedback-alignment burden.
```

Our response:

```text
memorization capacity does not imply feedback-alignment burden is free;
it only means the network may still eventually fit the task.
```

Source:

```text
https://web.mit.edu/6.034/www/6.s966/2017_Zhang_et_al_T_Understanding_deep_learning_requires_rethinking_generalization.pdf
```

### NTK interpolation phase transition

Montanari and Zhong analyze interpolation under lazy training. In the
overparameterized NT regime, the empirical NT kernel becomes well-conditioned
enough to interpolate arbitrary labels.

Useful positioning:

```text
Classical overparameterization theory says that once the tangent kernel has
enough rank/eigenvalue mass, arbitrary labels can be fit.
```

Our response:

```text
FA changes the learning operator. BP interpolation capacity does not imply the
FA tangent operator has the same spectrum or convergence rate.
```

Source:

```text
https://arxiv.org/abs/2007.12826
```

### Lillicrap et al.: random feedback can work

Lillicrap et al. show that fixed random feedback weights can support learning
and that forward weights evolve to align with the feedback pathway.

Useful positioning:

```text
The original FA result supports the intuition that random feedback can be made
useful by the network.
```

Our response:

```text
useful does not mean cost-free. The relevant question is how much tangent
capacity or finite-time learning speed is lost relative to BP.
```

Source:

```text
https://www.nature.com/articles/ncomms13276
```

### Nøkland: DFA reaches zero training error

Nøkland's direct feedback alignment paper states that random feedback weights
can work because the network learns to make them useful, and reports zero
training error even in convolutional and very deep networks.

Useful positioning:

```text
DFA evidence makes the no-gap intuition plausible if one only looks at final
training error.
```

Our response:

```text
zero eventual train error is compatible with a nonzero finite-time operator gap.
We should measure the gap at the tangent-operator level, not only final
interpolation.
```

Source:

```text
https://papers.neurips.cc/paper/6441-direct-feedback-alignment-provides-learning-in-deep-neural-networks
```

### Song, Xu, and Lafferty: overparameterized FA convergence

Song et al. prove exponential error convergence for two-layer feedback
alignment in an overparameterized setting, and show that regularization is
needed for parameter alignment.

Useful positioning:

```text
This is the most direct theoretical counterweight: overparameterized FA can
converge.
```

Our response:

```text
our claim must not be "FA cannot fit." The correct claim is that fitting does
not imply BP-equivalent tangent capacity or BP-equivalent finite-time dynamics.
```

Source:

```text
https://proceedings.neurips.cc/paper/2021/hash/a576eafbce762079f7d1f77fca1c5cc2-Abstract.html
```

## Is E[K_FA | J] = (1-k/P) K_BP Theory or Fit?

It is theory under a random-subspace null model. It is not fitted from the
experiments.

Assumptions:

```text
P = parameter dimension
J = true output Jacobian on the training set
K_BP = J J^T
E = random k-dimensional alignment-constrained subspace in parameter space
P_E = projection onto E
Q = I - P_E = retained parameter projection
K_k = J Q J^T
E is Haar-random and independent of J
```

By rotational symmetry:

```text
E[P_E] = (k/P) I
```

because `E[P_E]` must commute with every rotation and its trace is `k`.

Therefore:

```text
E[Q] = (1 - k/P) I
```

and conditioning on `J`:

```text
E[K_k | J]
= J E[Q] J^T
= (1 - k/P) J J^T
= (1 - k/P) K_BP
```

So this equation is a clean theoretical baseline.

However, it is not yet a faithful FA theorem. Actual FA does not literally
remove a Haar-random `k`-dimensional subspace from all parameters. FA changes the
backward Jacobian and produces a generally non-symmetric tangent operator:

```text
K_FA = J J_tilde^T
```

where `J_tilde` is the surrogate FA update Jacobian. Therefore, the random
projection formula should be used as a null model or intuition, not as the main
final theorem unless we can justify the random-subspace approximation.

## Better Object for Measuring Gap

For squared loss:

```text
L = ||r||^2 / (2N)
```

BP update gives local residual dynamics:

```text
r_next_BP ≈ r - η K_BP r / N
K_BP = J J^T
```

FA update gives:

```text
r_next_FA ≈ r - η K_FA r / N
K_FA = J J_tilde^T
```

Since `K_FA` can be non-symmetric, only its symmetric part contributes to the
first-order loss decrease:

```text
S_FA = (K_FA + K_FA^T) / 2
```

One-step BP decrease:

```text
ΔL_BP ≈ η r^T K_BP r / N
```

One-step FA decrease:

```text
ΔL_FA ≈ η r^T S_FA r / N
```

Therefore the exact local one-step erosion is:

```text
e(r) = 1 - (r^T S_FA r) / (r^T K_BP r)
```

This is a better gap measure than raw hard capacity margin. It directly answers:

```text
how much BP learning speed is lost along the current residual direction?
```

For a distribution over feedback matrices:

```text
F_e(x) = P_B(e(r; B) <= x)
```

This is the distribution we should try to predict and validate.

## Finite-Time Gap from Operator Erosion

If the operators are frozen:

```text
r_T_BP = (I - η K_BP / N)^T r_0
r_T_FA = (I - η K_FA / N)^T r_0
```

The predicted train-gap is:

```text
gap_T = (||r_T_FA||^2 - ||r_T_BP||^2) / (2N)
```

If a simple scalar erosion approximation holds:

```text
S_FA ≈ ρ K_BP
```

then in the BP eigenbasis:

```text
gap_T ≈ (1 / 2N) sum_i c_i^2 [
  (1 - η ρ λ_i / N)^(2T)
  - (1 - η λ_i / N)^(2T)
]
```

where:

```text
K_BP v_i = λ_i v_i
c_i = v_i^T r_0
```

This gives a direct quantitative bridge from capacity/operator erosion to loss
gap. No fitted post-hoc scale is required if `ρ` is predicted from a structural
model or measured from the early tangent operator.

## Revised Main Theory Target

The paper should not primarily claim:

```text
hard capacity margin predicts a sharp FA/BP gap transition
```

The better target is:

```text
random feedback induces a distribution of residual-direction operator erosion
e(r); capacity controls the distribution of e(r); finite-time train-gap follows
from the product of the corresponding tangent operators.
```

This is compatible with:

```text
large enough FA can eventually fit;
large enough FA can have small raw final gap;
but FA is not BP-equivalent unless operator erosion vanishes.
```

## Scaling Claim

If the effective burden grows sublinearly:

```text
k_eff / P -> 0
```

then:

```text
operator erosion -> 0
```

and FA approaches BP in the soft-capacity sense.

If the effective burden is proportional to model size:

```text
k_eff / P -> α > 0
```

then:

```text
relative operator erosion persists
```

even though absolute task loss can still be small because both BP and FA may be
strong in the overparameterized regime.

The key unresolved theoretical task is to derive `k_eff` or `ρ` for actual FA
from architecture, feedback initialization, and early alignment dynamics.