Position Song Xu Lafferty related work

1 files changed, 242 insertions, 0 deletions

diff --git a/notes/27_song_xu_lafferty_positioning.md b/notes/27_song_xu_lafferty_positioning.md
new file mode 100644
index 0000000..27946fb
--- /dev/null
+++ b/notes/27_song_xu_lafferty_positioning.md
@@ -0,0 +1,242 @@
+# Song, Xu, and Lafferty Positioning
+
+Paper:
+
+```text
+Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
+Ganlin Song, Ruitu Xu, John Lafferty
+NeurIPS 2021
+```
+
+Links:
+
+```text
+https://proceedings.neurips.cc/paper/2021/hash/a576eafbce762079f7d1f77fca1c5cc2-Abstract.html
+https://par.nsf.gov/servlets/purl/10301215
+```
+
+## What They Study
+
+They study two-layer scalar-output regression:
+
+```text
+f(x) = (1/sqrt(p)) sum_r beta_r psi(w_r^T x)
+```
+
+FA replaces the first-layer backward weight `beta_r` by a fixed random
+backward weight `b_r`:
+
+```text
+BP first-layer update uses beta_r
+FA first-layer update uses b_r
+```
+
+The second-layer weights `beta` are still trained normally.
+
+## Their Main Convergence Result
+
+Their Theorem 3.2 proves that in an overparameterized setting:
+
+```text
+||e(t+1)|| <= (1 - eta gamma / 4) ||e(t)||
+```
+
+under smooth bounded activation assumptions, a positive-definite `G` kernel
+assumption, and sufficiently large width.
+
+The key structural observation is:
+
+```text
+K_FA = G + H_FA
+```
+
+where:
+
+```text
+G = top-layer / beta kernel
+H_FA = first-layer FA contribution
+```
+
+They explicitly note:
+
+```text
+H_FA is not positive semidefinite and is close to 0 at initialization if the
+network is overparameterized.
+```
+
+Convergence is obtained because `G` is positive definite and `H_FA` remains
+small enough.
+
+## Their Alignment Result
+
+They distinguish convergence from alignment.
+
+Without regularization, in the overparameterized setting:
+
+```text
+cos angle(b, beta(t)) = O(n / sqrt(p))
+```
+
+So as width grows:
+
+```text
+beta(t) and b become nearly orthogonal
+```
+
+Thus, despite error convergence, actual alignment need not occur.
+
+With an L2 regularization schedule on `beta`, they prove alignment for linear
+networks:
+
+```text
+cos angle(b, beta(t)) >= c > 0
+```
+
+for sufficiently large time.
+
+## How This Relates to Us
+
+This paper does not refute our revised theory. It forces us to phrase it
+correctly.
+
+Wrong claim:
+
+```text
+Overparameterized FA cannot reach zero train error.
+```
+
+This is contradicted by Song/Xu/Lafferty in their two-layer setting.
+
+Correct claim:
+
+```text
+Overparameterized FA can converge, but convergence does not imply BP-equivalent
+tangent capacity or zero FA/BP operator erosion.
+```
+
+Their own theorem supports this distinction:
+
+1. FA convergence is driven by the positive-definite `G` component.
+2. The feedback-dependent first-layer component `H_FA` is non-PSD.
+3. In the wide lazy regime, `H_FA` is small.
+4. Alignment may not occur at all without regularization.
+
+So their result is compatible with:
+
+```text
+FA eventually fits because G is enough;
+FA still has a different learning operator from BP;
+FA can have finite-time or operator-level cost relative to BP.
+```
+
+## What We Should Not Say
+
+Do not write:
+
+```text
+we prove overparameterization cannot absorb FA
+```
+
+without qualification.
+
+Better:
+
+```text
+we prove overparameterization does not make the feedback-alignment burden
+operator-free; even when FA converges, the random feedback pathway can induce a
+nonzero relative tangent-operator erosion.
+```
+
+## What We Can Claim Against the Naive Interpretation
+
+If someone reads Song/Xu/Lafferty as:
+
+```text
+wide FA converges, therefore random feedback is free
+```
+
+then that interpretation is wrong.
+
+Their proof gives convergence because the top-layer kernel `G` remains
+positive-definite. It does not show:
+
+```text
+K_FA = K_BP
+```
+
+or:
+
+```text
+finite-time BP/FA gap = 0
+```
+
+or:
+
+```text
+feedback alignment cost disappears.
+```
+
+In fact, their no-regularization alignment result says the opposite of a naive
+alignment story:
+
+```text
+FA can converge without beta aligning with b.
+```
+
+This is useful for us: it supports the separation between:
+
+```text
+eventual fitting capacity
+```
+
+and:
+
+```text
+operator-level learning efficiency / capacity erosion.
+```
+
+## How To Position In The Paper
+
+Put them in related work as the closest theoretical predecessor:
+
+```text
+Song, Xu, and Lafferty prove exponential convergence of two-layer FA in an
+overparameterized regime and show that alignment need not occur without
+regularization. Their result demonstrates that random feedback can be sufficient
+for convergence. Our work asks a complementary question: even when convergence is
+possible, what is the distributional operator cost of using random feedback
+instead of BP?
+```
+
+Then cite their non-PSD kernel observation:
+
+```text
+They decompose the FA dynamics into a positive top-layer kernel G and a
+feedback-dependent term H_FA that is non-PSD and perturbative in the wide
+regime. Our tangent-operator erosion formalism generalizes this perspective by
+directly measuring the residual-direction loss in learning speed induced by the
+FA operator.
+```
+
+## Open Follow-Up
+
+We should still read their appendix more carefully because their proof may give
+a useful template for bounding:
+
+```text
+K_FA,t = G_t + H_FA,t
+```
+
+and showing when:
+
+```text
+H_FA,t
+```
+
+is perturbative. That could help us formalize `k_eff`:
+
+```text
+k_eff(B,t) = P * e_t(B)
+```
+
+or replace hard `k` with an operator-norm or residual-direction bound.