1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
|
# Song, Xu, and Lafferty Positioning
Paper:
```text
Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
Ganlin Song, Ruitu Xu, John Lafferty
NeurIPS 2021
```
Links:
```text
https://proceedings.neurips.cc/paper/2021/hash/a576eafbce762079f7d1f77fca1c5cc2-Abstract.html
https://par.nsf.gov/servlets/purl/10301215
```
## What They Study
They study two-layer scalar-output regression:
```text
f(x) = (1/sqrt(p)) sum_r beta_r psi(w_r^T x)
```
FA replaces the first-layer backward weight `beta_r` by a fixed random
backward weight `b_r`:
```text
BP first-layer update uses beta_r
FA first-layer update uses b_r
```
The second-layer weights `beta` are still trained normally.
## Their Main Convergence Result
Their Theorem 3.2 proves that in an overparameterized setting:
```text
||e(t+1)|| <= (1 - eta gamma / 4) ||e(t)||
```
under smooth bounded activation assumptions, a positive-definite `G` kernel
assumption, and sufficiently large width.
The key structural observation is:
```text
K_FA = G + H_FA
```
where:
```text
G = top-layer / beta kernel
H_FA = first-layer FA contribution
```
They explicitly note:
```text
H_FA is not positive semidefinite and is close to 0 at initialization if the
network is overparameterized.
```
Convergence is obtained because `G` is positive definite and `H_FA` remains
small enough.
## Their Alignment Result
They distinguish convergence from alignment.
Without regularization, in the overparameterized setting:
```text
cos angle(b, beta(t)) = O(n / sqrt(p))
```
So as width grows:
```text
beta(t) and b become nearly orthogonal
```
Thus, despite error convergence, actual alignment need not occur.
With an L2 regularization schedule on `beta`, they prove alignment for linear
networks:
```text
cos angle(b, beta(t)) >= c > 0
```
for sufficiently large time.
## How This Relates to Us
This paper does not refute our revised theory. It forces us to phrase it
correctly.
Wrong claim:
```text
Overparameterized FA cannot reach zero train error.
```
This is contradicted by Song/Xu/Lafferty in their two-layer setting.
Correct claim:
```text
Overparameterized FA can converge, but convergence does not imply BP-equivalent
tangent capacity or zero FA/BP operator erosion.
```
Their own theorem supports this distinction:
1. FA convergence is driven by the positive-definite `G` component.
2. The feedback-dependent first-layer component `H_FA` is non-PSD.
3. In the wide lazy regime, `H_FA` is small.
4. Alignment may not occur at all without regularization.
So their result is compatible with:
```text
FA eventually fits because G is enough;
FA still has a different learning operator from BP;
FA can have finite-time or operator-level cost relative to BP.
```
## What We Should Not Say
Do not write:
```text
we prove overparameterization cannot absorb FA
```
without qualification.
Better:
```text
we prove overparameterization does not make the feedback-alignment burden
operator-free; even when FA converges, the random feedback pathway can induce a
nonzero relative tangent-operator erosion.
```
## What We Can Claim Against the Naive Interpretation
If someone reads Song/Xu/Lafferty as:
```text
wide FA converges, therefore random feedback is free
```
then that interpretation is wrong.
Their proof gives convergence because the top-layer kernel `G` remains
positive-definite. It does not show:
```text
K_FA = K_BP
```
or:
```text
finite-time BP/FA gap = 0
```
or:
```text
feedback alignment cost disappears.
```
In fact, their no-regularization alignment result says the opposite of a naive
alignment story:
```text
FA can converge without beta aligning with b.
```
This is useful for us: it supports the separation between:
```text
eventual fitting capacity
```
and:
```text
operator-level learning efficiency / capacity erosion.
```
## How To Position In The Paper
Put them in related work as the closest theoretical predecessor:
```text
Song, Xu, and Lafferty prove exponential convergence of two-layer FA in an
overparameterized regime and show that alignment need not occur without
regularization. Their result demonstrates that random feedback can be sufficient
for convergence. Our work asks a complementary question: even when convergence is
possible, what is the distributional operator cost of using random feedback
instead of BP?
```
Then cite their non-PSD kernel observation:
```text
They decompose the FA dynamics into a positive top-layer kernel G and a
feedback-dependent term H_FA that is non-PSD and perturbative in the wide
regime. Our tangent-operator erosion formalism generalizes this perspective by
directly measuring the residual-direction loss in learning speed induced by the
FA operator.
```
## Open Follow-Up
We should still read their appendix more carefully because their proof may give
a useful template for bounding:
```text
K_FA,t = G_t + H_FA,t
```
and showing when:
```text
H_FA,t
```
is perturbative. That could help us formalize `k_eff`:
```text
k_eff(B,t) = P * e_t(B)
```
or replace hard `k` with an operator-norm or residual-direction bound.
|
