Add effective capacity literature and gap theory note

1 files changed, 353 insertions, 0 deletions

diff --git a/notes/24_effective_capacity_literature_and_gap_theory.md b/notes/24_effective_capacity_literature_and_gap_theory.md
new file mode 100644
index 0000000..9ee424c
--- /dev/null
+++ b/notes/24_effective_capacity_literature_and_gap_theory.md
@@ -0,0 +1,353 @@
+# Effective Capacity Literature and Gap Theory
+
+This note collects literature that supports the intuition that large effective
+capacity can absorb difficult training signals, then clarifies what our revised
+theory should and should not claim.
+
+## Literature to Position Against
+
+### Zhang et al.: effective capacity and random labels
+
+Zhang, Bengio, Hardt, Recht, and Vinyals use random-label tests to argue that
+deep networks have enormous effective capacity. Their central observation is
+that standard networks can fit random labels to zero training error.
+
+Useful positioning:
+
+```text
+If neural networks can memorize arbitrary random labels, it is tempting to
+believe that sufficiently overparameterized networks can also absorb random
+feedback-alignment burden.
+```
+
+Our response:
+
+```text
+memorization capacity does not imply feedback-alignment burden is free;
+it only means the network may still eventually fit the task.
+```
+
+Source:
+
+```text
+https://web.mit.edu/6.034/www/6.s966/2017_Zhang_et_al_T_Understanding_deep_learning_requires_rethinking_generalization.pdf
+```
+
+### NTK interpolation phase transition
+
+Montanari and Zhong analyze interpolation under lazy training. In the
+overparameterized NT regime, the empirical NT kernel becomes well-conditioned
+enough to interpolate arbitrary labels.
+
+Useful positioning:
+
+```text
+Classical overparameterization theory says that once the tangent kernel has
+enough rank/eigenvalue mass, arbitrary labels can be fit.
+```
+
+Our response:
+
+```text
+FA changes the learning operator. BP interpolation capacity does not imply the
+FA tangent operator has the same spectrum or convergence rate.
+```
+
+Source:
+
+```text
+https://arxiv.org/abs/2007.12826
+```
+
+### Lillicrap et al.: random feedback can work
+
+Lillicrap et al. show that fixed random feedback weights can support learning
+and that forward weights evolve to align with the feedback pathway.
+
+Useful positioning:
+
+```text
+The original FA result supports the intuition that random feedback can be made
+useful by the network.
+```
+
+Our response:
+
+```text
+useful does not mean cost-free. The relevant question is how much tangent
+capacity or finite-time learning speed is lost relative to BP.
+```
+
+Source:
+
+```text
+https://www.nature.com/articles/ncomms13276
+```
+
+### Nøkland: DFA reaches zero training error
+
+Nøkland's direct feedback alignment paper states that random feedback weights
+can work because the network learns to make them useful, and reports zero
+training error even in convolutional and very deep networks.
+
+Useful positioning:
+
+```text
+DFA evidence makes the no-gap intuition plausible if one only looks at final
+training error.
+```
+
+Our response:
+
+```text
+zero eventual train error is compatible with a nonzero finite-time operator gap.
+We should measure the gap at the tangent-operator level, not only final
+interpolation.
+```
+
+Source:
+
+```text
+https://papers.neurips.cc/paper/6441-direct-feedback-alignment-provides-learning-in-deep-neural-networks
+```
+
+### Song, Xu, and Lafferty: overparameterized FA convergence
+
+Song et al. prove exponential error convergence for two-layer feedback
+alignment in an overparameterized setting, and show that regularization is
+needed for parameter alignment.
+
+Useful positioning:
+
+```text
+This is the most direct theoretical counterweight: overparameterized FA can
+converge.
+```
+
+Our response:
+
+```text
+our claim must not be "FA cannot fit." The correct claim is that fitting does
+not imply BP-equivalent tangent capacity or BP-equivalent finite-time dynamics.
+```
+
+Source:
+
+```text
+https://proceedings.neurips.cc/paper/2021/hash/a576eafbce762079f7d1f77fca1c5cc2-Abstract.html
+```
+
+## Is E[K_FA | J] = (1-k/P) K_BP Theory or Fit?
+
+It is theory under a random-subspace null model. It is not fitted from the
+experiments.
+
+Assumptions:
+
+```text
+P = parameter dimension
+J = true output Jacobian on the training set
+K_BP = J J^T
+E = random k-dimensional alignment-constrained subspace in parameter space
+P_E = projection onto E
+Q = I - P_E = retained parameter projection
+K_k = J Q J^T
+E is Haar-random and independent of J
+```
+
+By rotational symmetry:
+
+```text
+E[P_E] = (k/P) I
+```
+
+because `E[P_E]` must commute with every rotation and its trace is `k`.
+
+Therefore:
+
+```text
+E[Q] = (1 - k/P) I
+```
+
+and conditioning on `J`:
+
+```text
+E[K_k | J]
+= J E[Q] J^T
+= (1 - k/P) J J^T
+= (1 - k/P) K_BP
+```
+
+So this equation is a clean theoretical baseline.
+
+However, it is not yet a faithful FA theorem. Actual FA does not literally
+remove a Haar-random `k`-dimensional subspace from all parameters. FA changes the
+backward Jacobian and produces a generally non-symmetric tangent operator:
+
+```text
+K_FA = J J_tilde^T
+```
+
+where `J_tilde` is the surrogate FA update Jacobian. Therefore, the random
+projection formula should be used as a null model or intuition, not as the main
+final theorem unless we can justify the random-subspace approximation.
+
+## Better Object for Measuring Gap
+
+For squared loss:
+
+```text
+L = ||r||^2 / (2N)
+```
+
+BP update gives local residual dynamics:
+
+```text
+r_next_BP ≈ r - η K_BP r / N
+K_BP = J J^T
+```
+
+FA update gives:
+
+```text
+r_next_FA ≈ r - η K_FA r / N
+K_FA = J J_tilde^T
+```
+
+Since `K_FA` can be non-symmetric, only its symmetric part contributes to the
+first-order loss decrease:
+
+```text
+S_FA = (K_FA + K_FA^T) / 2
+```
+
+One-step BP decrease:
+
+```text
+ΔL_BP ≈ η r^T K_BP r / N
+```
+
+One-step FA decrease:
+
+```text
+ΔL_FA ≈ η r^T S_FA r / N
+```
+
+Therefore the exact local one-step erosion is:
+
+```text
+e(r) = 1 - (r^T S_FA r) / (r^T K_BP r)
+```
+
+This is a better gap measure than raw hard capacity margin. It directly answers:
+
+```text
+how much BP learning speed is lost along the current residual direction?
+```
+
+For a distribution over feedback matrices:
+
+```text
+F_e(x) = P_B(e(r; B) <= x)
+```
+
+This is the distribution we should try to predict and validate.
+
+## Finite-Time Gap from Operator Erosion
+
+If the operators are frozen:
+
+```text
+r_T_BP = (I - η K_BP / N)^T r_0
+r_T_FA = (I - η K_FA / N)^T r_0
+```
+
+The predicted train-gap is:
+
+```text
+gap_T = (||r_T_FA||^2 - ||r_T_BP||^2) / (2N)
+```
+
+If a simple scalar erosion approximation holds:
+
+```text
+S_FA ≈ ρ K_BP
+```
+
+then in the BP eigenbasis:
+
+```text
+gap_T ≈ (1 / 2N) sum_i c_i^2 [
+  (1 - η ρ λ_i / N)^(2T)
+  - (1 - η λ_i / N)^(2T)
+]
+```
+
+where:
+
+```text
+K_BP v_i = λ_i v_i
+c_i = v_i^T r_0
+```
+
+This gives a direct quantitative bridge from capacity/operator erosion to loss
+gap. No fitted post-hoc scale is required if `ρ` is predicted from a structural
+model or measured from the early tangent operator.
+
+## Revised Main Theory Target
+
+The paper should not primarily claim:
+
+```text
+hard capacity margin predicts a sharp FA/BP gap transition
+```
+
+The better target is:
+
+```text
+random feedback induces a distribution of residual-direction operator erosion
+e(r); capacity controls the distribution of e(r); finite-time train-gap follows
+from the product of the corresponding tangent operators.
+```
+
+This is compatible with:
+
+```text
+large enough FA can eventually fit;
+large enough FA can have small raw final gap;
+but FA is not BP-equivalent unless operator erosion vanishes.
+```
+
+## Scaling Claim
+
+If the effective burden grows sublinearly:
+
+```text
+k_eff / P -> 0
+```
+
+then:
+
+```text
+operator erosion -> 0
+```
+
+and FA approaches BP in the soft-capacity sense.
+
+If the effective burden is proportional to model size:
+
+```text
+k_eff / P -> α > 0
+```
+
+then:
+
+```text
+relative operator erosion persists
+```
+
+even though absolute task loss can still be small because both BP and FA may be
+strong in the overparameterized regime.
+
+The key unresolved theoretical task is to derive `k_eff` or `ρ` for actual FA
+from architecture, feedback initialization, and early alignment dynamics.