# Song, Xu, and Lafferty Positioning

Paper:

```text
Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
Ganlin Song, Ruitu Xu, John Lafferty
NeurIPS 2021
```

Links:

```text
https://proceedings.neurips.cc/paper/2021/hash/a576eafbce762079f7d1f77fca1c5cc2-Abstract.html
https://par.nsf.gov/servlets/purl/10301215
```

## What They Study

They study two-layer scalar-output regression:

```text
f(x) = (1/sqrt(p)) sum_r beta_r psi(w_r^T x)
```

FA replaces the first-layer backward weight `beta_r` by a fixed random
backward weight `b_r`:

```text
BP first-layer update uses beta_r
FA first-layer update uses b_r
```

The second-layer weights `beta` are still trained normally.

## Their Main Convergence Result

Their Theorem 3.2 proves that in an overparameterized setting:

```text
||e(t+1)|| <= (1 - eta gamma / 4) ||e(t)||
```

under smooth bounded activation assumptions, a positive-definite `G` kernel
assumption, and sufficiently large width.

The key structural observation is:

```text
K_FA = G + H_FA
```

where:

```text
G = top-layer / beta kernel
H_FA = first-layer FA contribution
```

They explicitly note:

```text
H_FA is not positive semidefinite and is close to 0 at initialization if the
network is overparameterized.
```

Convergence is obtained because `G` is positive definite and `H_FA` remains
small enough.

## Their Alignment Result

They distinguish convergence from alignment.

Without regularization, in the overparameterized setting:

```text
cos angle(b, beta(t)) = O(n / sqrt(p))
```

So as width grows:

```text
beta(t) and b become nearly orthogonal
```

Thus, despite error convergence, actual alignment need not occur.

With an L2 regularization schedule on `beta`, they prove alignment for linear
networks:

```text
cos angle(b, beta(t)) >= c > 0
```

for sufficiently large time.

## How This Relates to Us

This paper does not refute our revised theory. It forces us to phrase it
correctly.

Wrong claim:

```text
Overparameterized FA cannot reach zero train error.
```

This is contradicted by Song/Xu/Lafferty in their two-layer setting.

Correct claim:

```text
Overparameterized FA can converge, but convergence does not imply BP-equivalent
tangent capacity or zero FA/BP operator erosion.
```

Their own theorem supports this distinction:

1. FA convergence is driven by the positive-definite `G` component.
2. The feedback-dependent first-layer component `H_FA` is non-PSD.
3. In the wide lazy regime, `H_FA` is small.
4. Alignment may not occur at all without regularization.

So their result is compatible with:

```text
FA eventually fits because G is enough;
FA still has a different learning operator from BP;
FA can have finite-time or operator-level cost relative to BP.
```

## What We Should Not Say

Do not write:

```text
we prove overparameterization cannot absorb FA
```

without qualification.

Better:

```text
we prove overparameterization does not make the feedback-alignment burden
operator-free; even when FA converges, the random feedback pathway can induce a
nonzero relative tangent-operator erosion.
```

## What We Can Claim Against the Naive Interpretation

If someone reads Song/Xu/Lafferty as:

```text
wide FA converges, therefore random feedback is free
```

then that interpretation is wrong.

Their proof gives convergence because the top-layer kernel `G` remains
positive-definite. It does not show:

```text
K_FA = K_BP
```

or:

```text
finite-time BP/FA gap = 0
```

or:

```text
feedback alignment cost disappears.
```

In fact, their no-regularization alignment result says the opposite of a naive
alignment story:

```text
FA can converge without beta aligning with b.
```

This is useful for us: it supports the separation between:

```text
eventual fitting capacity
```

and:

```text
operator-level learning efficiency / capacity erosion.
```

## How To Position In The Paper

Put them in related work as the closest theoretical predecessor:

```text
Song, Xu, and Lafferty prove exponential convergence of two-layer FA in an
overparameterized regime and show that alignment need not occur without
regularization. Their result demonstrates that random feedback can be sufficient
for convergence. Our work asks a complementary question: even when convergence is
possible, what is the distributional operator cost of using random feedback
instead of BP?
```

Then cite their non-PSD kernel observation:

```text
They decompose the FA dynamics into a positive top-layer kernel G and a
feedback-dependent term H_FA that is non-PSD and perturbative in the wide
regime. Our tangent-operator erosion formalism generalizes this perspective by
directly measuring the residual-direction loss in learning speed induced by the
FA operator.
```

## Open Follow-Up

We should still read their appendix more carefully because their proof may give
a useful template for bounding:

```text
K_FA,t = G_t + H_FA,t
```

and showing when:

```text
H_FA,t
```

is perturbative. That could help us formalize `k_eff`:

```text
k_eff(B,t) = P * e_t(B)
```

or replace hard `k` with an operator-norm or residual-direction bound.