1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
|
# Soft-Erosion Theory Distribution Validation Attempt
We tested whether the revised soft-erosion theory can predict the empirical
FA/BP train-gap distribution from the dense `T=30000` sweep.
## Data
Empirical run:
```text
outputs/phase_transition_dense_T30000_352traj
```
Validation output:
```text
outputs/soft_erosion_theory_vs_empirical
```
Main plots:
```text
outputs/soft_erosion_theory_vs_empirical/gap_distribution_theory_vs_empirical.png
outputs/soft_erosion_theory_vs_empirical/local_erosion_distribution_theory_vs_empirical.png
outputs/soft_erosion_theory_vs_empirical/local_operator_scalar_gap_theory_vs_empirical.png
```
Script:
```text
scripts/soft_erosion_theory_vs_empirical.py
```
## Theory 1: Hard-k Random-Subspace Gap Distribution
This test uses the null model:
```text
P = parameter count
k = hard FA constraint count
rho ~ Beta((P-k)/2, k/2)
K_FA ≈ rho K_BP
```
Then it predicts finite-time loss from the BP initial tangent spectrum:
```text
r_T_BP = exp(-eta T K_BP / N) r_0
r_T_FA = exp(-eta T rho K_BP / N) r_0
gap_T = (||r_T_FA||^2 - ||r_T_BP||^2) / (2N)
```
This is not fitted. The only random quantity is the random-subspace theory
distribution for `rho`.
Result:
```text
gap_distribution_theory_vs_empirical.png
```
The hard-k random-subspace theory strongly overpredicts the empirical gap,
especially on the positive-margin side.
Examples:
| width | margin | hard-k theory mean | empirical mean |
|---:|---:|---:|---:|
| 40 | 130 | 0.040374 | 0.000162 |
| 36 | 66 | 0.050133 | 0.000791 |
| 32 | 2 | 0.064534 | 0.003729 |
| 28 | -62 | 0.103630 | 0.014195 |
| 24 | -126 | 0.121998 | 0.046464 |
| 20 | -190 | 0.151140 | 0.146803 |
So the hard constraint count `k` is not the effective burden. It is too
pessimistic.
## Theory 2: Local One-Step Erosion
We measured the actual initial local operator erosion for each feedback seed:
```text
speed_ratio = <g_BP, g_FA> / <g_BP, g_BP>
erosion = 1 - speed_ratio
```
This directly estimates:
```text
1 - (r^T S_FA r) / (r^T K_BP r)
```
where:
```text
S_FA = (K_FA + K_FA^T) / 2
```
The comparison:
```text
local_erosion_distribution_theory_vs_empirical.png
```
shows that real FA local erosion is much smaller than the hard-k Beta null.
Examples:
| width | empirical local erosion mean | hard-k Beta erosion mean |
|---:|---:|---:|
| 24 | 0.3907 | about 0.634 |
| 32 | 0.2338 | about 0.691 |
| 40 | 0.3381 | about 0.733 |
Again, hard `k` overcounts effective burden.
## Theory 3: Local-Operator Scalar Gap Prediction
We then used the measured initial speed ratio as a no-fit scalar operator
prediction:
```text
K_FA ≈ speed_ratio(B) K_BP
```
and propagated the BP tangent spectrum for `T=30000`.
Result:
```text
local_operator_scalar_gap_theory_vs_empirical.png
```
This improves over hard-k random subspace but is still not sufficient.
Examples:
| width | margin | local-operator theory mean | empirical mean |
|---:|---:|---:|---:|
| 40 | 130 | 0.002701 | 0.000162 |
| 36 | 66 | 0.004612 | 0.000791 |
| 32 | 2 | 0.005276 | 0.003729 |
| 28 | -62 | 0.026043 | 0.014195 |
| 24 | -126 | 0.047863 | 0.046464 |
| 22 | -158 | 0.037249 | 0.100526 |
| 20 | -190 | 0.071783 | 0.146803 |
It matches around `width=24` and is roughly right near `width=32`, but:
1. it overpredicts the positive-margin tail;
2. it underpredicts the more negative-margin regime;
3. it is not monotone enough because a single initial scalar cannot capture
training-time alignment and feature drift.
## Conclusion
The current result is not a successful validation of the hard-k random-subspace
theory.
It gives a sharper diagnosis:
```text
hard FA constraint count k is not k_eff.
```
The empirical gap is controlled by an effective operator erosion that evolves
during training:
```text
K_FA,t = J_t J_tilde_t^T
```
not by a fixed random deletion of a `k`-dimensional subspace.
## Revised Theory Target
The next theory should predict a distribution over:
```text
e_t(B) = 1 - (r_t^T S_FA,t r_t) / (r_t^T K_BP,t r_t)
```
or its cumulative finite-time form:
```text
r_T_FA = product_t (I - eta K_FA,t / N) r_0
r_T_BP = product_t (I - eta K_BP,t / N) r_0
```
The right effective capacity is therefore:
```text
k_eff(B,t) = P * e_t(B)
```
or a spectral version of this quantity, not the raw hard matrix constraint
count.
## Relation to Song, Xu, and Lafferty
`Convergence and Alignment of Gradient Descent with Random Backpropagation
Weights` is highly relevant and must be read carefully next.
Current hypothesis:
```text
Their overparameterized convergence theorem is compatible with our findings.
It shows FA can converge; it does not imply BP-equivalent finite-time tangent
capacity.
```
We need to check whether their alignment dynamics imply a decay law for
`e_t(B)` or `k_eff(B,t)`.
|
