1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
|
# Soft Capacity Erosion Revision
The dense long-training sweep rules out the clean hard-exhaustion story as the
main theory.
The previous hard-rank claim was:
```text
Δd_hard = max(0, k - (P - d))
```
This is still a useful sanity check for exact rank collapse, but it is too
coarse for FA/BP train-gap prediction. It predicts no functional loss before
redundant directions are exhausted. The dense sweep shows the gap changes
smoothly with capacity margin instead.
## Correct Interpretation
Capacity is eroded continuously.
Alignment constraints do not wait until all redundant directions are exhausted.
They project into task-relevant directions with nonzero expected overlap from
the beginning. Overparameterization reduces the overlap, but does not make it
zero.
So the right statement is not:
```text
enough redundant capacity -> no FA/BP bias
```
The right statement is:
```text
more redundant capacity -> smaller expected FA/BP bias
```
## Random-Subspace Soft Erosion
Let:
```text
P = parameter dimension
d = task-sensitive dimension
k = alignment constraint dimension
S = task-sensitive parameter subspace
E = alignment constraint subspace
P_S = projection onto S
P_E = projection onto E
```
The soft overlap is:
```text
T_k = tr(P_E P_S)
```
For a random `k`-dimensional alignment subspace:
```text
E[T_k] = k d / P
```
This is nonzero for every `k > 0`. Therefore, at the effective-capacity level,
there is no exact zero-loss region before hard exhaustion. There is only a
small-overlap region when `P` is large.
The hard-rank loss:
```text
max(0, k - (P - d))
```
is a threshold for exact rank collapse, not a threshold for finite-time loss,
conditioning loss, or effective-dimension loss.
## Tangent-Kernel Version
Let `J` be the training-set output Jacobian. BP uses:
```text
K_BP = J J^T
```
If alignment removes a random `k`-dimensional parameter subspace and leaves
projection `Q`, then the restricted tangent kernel is:
```text
K_k = J Q J^T
```
For a random retained subspace of dimension `P-k`:
```text
E[Q] = (1 - k/P) I
```
Therefore:
```text
E[K_k | J] = (1 - k/P) K_BP
```
This is the cleanest correction. Before any hard rank collapse, the expected
tangent operator is already weakened by a factor proportional to `k/P`.
For squared-loss gradient descent with residual `r_t`, each BP eigenmode with
eigenvalue `λ_i` contracts roughly as:
```text
BP: (1 - η λ_i / N)^T
soft FA: (1 - η (1-k/P) λ_i / N)^T
```
So finite-time gap is positive for any `k > 0`, although it shrinks as `k/P`
shrinks.
## Penalized Alignment Version
If alignment is a soft penalty rather than a hard projection:
```text
H_align = C^T C
K_λ = J (M + λ H_align)^(-1) J^T
```
For small `λ`:
```text
K_λ = K_0 - λ J M^(-1) H_align M^(-1) J^T + higher-order terms
```
Again, there is no hard threshold. Any nonzero alignment pressure reduces
effective tangent capacity in task-relevant directions unless it is exactly
orthogonal to the task subspace.
## Revised Contribution 2
Old wording:
```text
FA/BP gap opens when redundant directions are exhausted.
```
New wording:
```text
Feedback alignment induces a soft capacity erosion. The expected task-relevant
overlap of alignment constraints scales as k d / P, so redundancy suppresses but
does not eliminate the FA/BP gap. Hard rank exhaustion remains a limiting
boundary for exact rank collapse, while empirical train gaps follow a smooth
capacity-controlled ramp.
```
## What The Dense Experiment Supports
Dense `T=30000` sweep:
```text
margin -190 -> gap 0.1468
margin -158 -> gap 0.1005
margin -126 -> gap 0.0465
margin -94 -> gap 0.0297
margin -62 -> gap 0.0142
margin -30 -> gap 0.0075
margin 2 -> gap 0.0037
margin 34 -> gap 0.0021
margin 66 -> gap 0.0008
margin 98 -> gap 0.0006
margin 130 -> gap 0.0002
```
This is exactly the qualitative shape expected from soft erosion: monotone,
smooth, and close to log-linear over the tested range.
## Paper Consequence
Contribution 2 should be renamed from:
```text
Scaling Law and Redundancy Exhaustion
```
to something like:
```text
Scaling Law and Soft Capacity Erosion
```
Hard exhaustion should appear as a corollary or limiting case, not the main
claim.
|
