1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
|
# Evidence Ledger (Writing Base for the AAAI Paper)
Authoritative consolidation of every result, number, and figure source in this
project, compiled directly from notes 00-35, the scripts, and the raw outputs.
**Write the paper from this file**, not from any earlier draft: the earlier
LaTeX draft was written without a careful pass over the experimental record and
omits much of the evidence chain below. Status of every claim is marked.
Numbers updated 2026-06-09 after the wrap-up corrections (note 37): the
closed-form comparison is now data/init-matched, and the e0-distribution
sampler runs the real FA backward pass.
---
## 0. Scope and claim discipline
- Setting: MLPs, squared loss `L = ||r||^2/(2N)`, full-batch training,
feedback alignment = fixed random backward matrices `B_l` replacing
`W_{l+1}^T`. Synthetic random-label / teacher tasks + MNIST-subset
validation (note 38).
- The paper quantifies the **optimization (train-loss) gap** FA pays relative
to BP. It does NOT claim FA fails to converge (compatible with Song-Xu-
Lafferty), does not predict test accuracy, and the estimator is
**conditional on two early kernel snapshots** — not architecture-only.
- "No fit" everywhere below means: no post-hoc scalar was tuned to match the
reported data. Predictions are computed from initialization (and, for the
estimator, one early snapshot).
## 1. Theorems (proof status)
| # | statement | status | source note |
|---|---|---|---|
| T1 | `Q = cos^2(a,b) ~ Beta(1/2,(D-1)/2)`, `E[Q]=1/D` for independent isotropic directions | exact | 00/01 |
| T2 | log-volume costs add: `C_all = sum_l C_l = Theta(L n^2)`; surprisal `S_l ~ Exp(1)`, `sum ~ Gamma(L,1)` | exact | 01 |
| T3 | prior-free minimax: `sup_mu inf_a E[(a^T b)^2] = 1/D`, attained by isotropic | exact | 01 |
| T4 | null model: Haar-random k-dim deletion gives `e(r) ~ Beta(k/2,(P-k)/2)`, `E[e]=k/P`, `P(e=0)=0` | exact (within null model) | 26 |
| T5 | **initial-erosion theorem**: `E_B[e_0 | W,r] = 1 - ||g_out^BP||^2 / sum_l ||g_l^BP||^2` (hidden speed share), any depth, any independent zero-mean feedback | exact | 29 |
| T6 | one-hidden-layer Gaussian FA: `e_0 | W,r` exactly Gaussian with explicit mean/std (linear in B) | exact | 32 |
| T7 | closed-form gap under A1 (lazy) + A2 (scalar erosion `S_FA ~= rho K_BP`): `gap_T = (1/2N) sum_i c_i^2 [(1-eta rho lam_i/N)^{2T} - (1-eta lam_i/N)^{2T}]`; analytic => soft ramp, no threshold unless `rho -> 0`; converged regime `log gap_T ~= const - (2 eta T/N) rho lam_min` | exact under A1+A2 | 34 |
A2 is exact in the residual direction by T5 (`rho = 1 - E_B[e_0]` = output
share); extending it to all eigenmodes is the one modeling step. A1 is the
standard lazy approximation whose breakdown is *measured* (item 3c below).
## 2. Static geometry: validation record
| claim | evidence | numbers | output dir |
|---|---|---|---|
| Beta law | 700k samples, D = 64..4096 | KS stat 0.002-0.004, p 0.17-0.99; mean/q99 match to 4 digits | `capacity_empirical_validation` |
| capacity tail cost | same run | max smoothed cost error 0.20 nats, mean 0.019; multilayer product max err 0.46 nats | same |
| `Gamma(L,1)` surprisal | 100k per (D,L), D in {64..4096}, L in {1..16} | max KS **0.0042**, mean abs mean-error 0.0042 | `multilayer_capacity_distribution` |
| minimax | 5 schemes, D=32/128 | isotropic lam_min 0.0291 vs 1/32=0.0313; rademacher 0.0289; anisotropic 0.0061; subspace/axis 0 | `minimax_initialization`, `initialization_distribution_matching` |
| functional overlap | P=96,d=24, trials=100 | hard loss exact (0 until k>P-d, then k-(P-d)); soft overlap kd/P to 3 digits | `functional_capacity_overlap` |
## 3. The gap pillar: four-level evidence chain
### (a) Null model + its honest failure (negative result, keep in paper)
- T4 proves nonzero soft cost in the cleanest model; `P(e=0)=0`.
- **Hard-k overpredicts real FA**: predicted gap 0.040 at margin +130 vs
measured 0.00016 (250x); ~2x on local erosion (measured local erosion 0.23-
0.39 vs Beta-mean 0.63-0.73) (note 25, `soft_erosion_theory_vs_empirical`).
Role in paper: motivates the *operator* object; hard constraint count is
not the effective burden.
### (b) Exact at t=0 (T5 + T6)
- Mean theorem, 2-hidden-layer, real FA backward, 6 widths x 6 inits x 512
draws: `E_B[e_0] = 0.4378 +/- 0.008` vs theory `0.4403`; per-width errors
0.0003-0.008 (note 29, `actual_fa_initial_operator_moments`).
- Independent t=0 anchor from the (superseded) recovery probe: 0.4026+/-0.0066
vs 0.4071 (run log, note 37).
- Full distribution (T6), **sampler = real FA backward after note-37 fix**:
4 widths x 3 inits x 4096 draws, KS p in [0.13, 0.999], all means/stds match
to 3-4 digits; linear identity `<C,B>` verified against the real backward to
max err 8.9e-16 (`actual_fa_initial_erosion_distribution`).
- **MNIST (note 38)**: 16 configs (width {64,256} x depth {1,2} x 4 inits,
512 real-backward draws each): hidden share spans **0.39-0.82** and
`E_B[e_0]` lands on it with |diff| 0.0001-0.0089 (all within ~2 stderr) —
the theorem holds on real data with no fit
(`real_data_validation_mnist/e0_mnist_rows.csv`).
### (c) Finite time: the lazy chain (previous draft omitted ALL of this)
1. **T=5 local exactness** (note 10): frozen K(0), 256 traj, 8 N values:
corr **0.99947**, MAE **0.00083**, mean ratio 0.993. Horizon table: ratio
1.028/0.987/0.942/0.884/0.814 at T=1/5/10/20/50. => the operator object is
right; freezing is a *local* theorem.
2. **Drift is the whole error** (note 13): at T=50, oracle time-varying K(t)
product cuts gap MAE 0.0277 -> **0.000273** (100x); BP loss MAE 0.0254 ->
0.00024. => no higher-order Taylor mystery; model K_t and you are done.
3. **Estimator** `K_hat_t = K_0 + (t/s)(K_s - K_0)` (notes 15-17): 256 traj,
T=50, s=20: MAE **0.00189**, bias -0.0015, corr **0.99885**; fixed-K(0)
MAE 0.0185 (10x worse); re-tangent MAE 0.0049 biased high => **no-fit
bracket** `linear <= empirical <= retangent`.
Stress grid (7 settings): depth 1/2/3, width 32/64/96, horizon 25/50/100:
linear MAE 0.00023-0.0057, corr 0.994-0.99989
(`operator_stress_grid_summary`).
4. **Bias structure** (note 17): residual -0.0021 gap bias = BP-loss
overprediction (+0.0026), FA near-unbiased (+0.0005); cause: drift velocity
decays after s (alpha_t < t/s; e.g. true alpha_50 = 1.46 (BP) / 1.17 (FA)
vs linear 2.5). Honest error story, no fitted correction.
5. **MNIST replication (note 38)**: same protocol, widths {128,256} x
N {128,256,512} x 2 inits x 8 feedbacks = 96 traj, lazy step
(`eta lam_max/N = 0.05`): velocity MAE **0.00196** (= 3.9% of mean gap),
bias -0.0006, corr 0.911 (gap range only 0.040-0.063, so MAE/bias are the
operative stats); fixed K(0) MAE 0.0061 biased high — same bracket as
synthetic. At eta lam_max/N = 0.3 the lazy chain breaks as expected
(scope boundary, recorded in note 38).
### (d) The soft ramp + closed form (T7)
- Dense sweep (note 22): T=30000, 11 widths (20..40), 352 traj: gap
0.1468 -> 0.000162 across margin -190 -> +130, smooth, log-linear
(`log gap ~ -0.0209*margin`, R^2 **0.993**), **no kink at margin 0,
nonzero at positive margin** => refutes the redundancy-exhaustion phase
transition (and notes 19-21 document how the kink hypothesis died as T grew:
near-margin-0 gap 0.297 at T=3000, 0.051 at T=10^4, 0.0037 at T=3x10^4).
- Closed form vs measured (**matched data + init after note-37 fix**):
- matched single init: corr(log,log) = **0.933**;
- 5-init geometric mean (same data): corr **0.982**; ensemble-mean
lam_min(w) rises 0.026 -> 0.763 and explains the ramp at corr **0.987**;
rho stable 0.68-0.79 across widths.
- the previously reported corr 0.977 came from a comparison computed on the
wrong data seed and init seed; retire that number (note 37).
- frozen-init compresses the range (under-predicts large-gap end, over-
predicts small-gap tail) — exactly the drift the estimator corrects;
state this as mechanism-vs-magnitude.
(`closed_form_soft_ramp/closed_form_vs_measured.csv`, `closed_form_per_seed.csv`)
## 4. Negative results worth reporting (reviewer-proofing)
| failed approach | number | note |
|---|---|---|
| hard-k as gap predictor | overpredicts 2x-250x | 25 |
| infinitesimal derivative extrapolation `K_0 + t dK_0` | MAE **270** vs 0.024 baseline (diverges) | 31 |
| scalar directional-gain model across N | corr 0.124 (vs operator 0.999) | 14 |
| trajectory bridge on long Adam horizons | shape corr 0.97 but 2-3x scale error | 02/08 |
| phase-transition (kink) hypothesis | dissolves as T grows; dense T=30000 smooth | 19-22 |
These define why the final objects (operator erosion + early velocity) are
what they are; (31) and (25) deserve explicit mention in the paper.
## 5. Real-data and generalization coverage (new, note 38)
- MNIST e0 + estimator: `outputs/real_data_validation_mnist/` (script
`scripts/real_data_validation.py`, self-test verifies kernels vs autograd
at 1e-15).
- Teacher-task test gap (SGD, widths 8..96): `outputs/teacher_test_gap_sgd_T8000/`
— addresses "what about generalization": train-gap story plus a descriptive
test-MSE companion. Random-label sweeps measure optimization gap by design.
## 6. Figure plan (sources verified on disk)
| fig | content | source |
|---|---|---|
| F1 | static law: Beta hist + QQ (+ Gamma(L,1) inset or appendix) | `capacity_empirical_validation` |
| F2 | dense soft ramp + closed form (matched + 5-init mean + band) + lam_min ramp law | `closed_form_soft_ramp/closed_form_vs_measured_ramp.png` |
| F3 | e0: moment calibration + Gaussian distribution panels (real backward) | `actual_fa_initial_operator_moments`, `actual_fa_initial_erosion_distribution` |
| F4 | estimator: predicted-vs-empirical scatter, 256 traj + stress grid summary | `compressed_operator_s20_256traj_T50_width64_plots`, `operator_stress_grid_summary` |
| F5 | MNIST: e0 calibration + estimator scatter (real data) | `real_data_validation_mnist` |
| F6 (optional/appendix) | T=5 exactness + oracle-K(t) diagnostic (the lazy chain) | notes 10/13 outputs |
## 7. Main text vs appendix split
- Main: T1-T3 (brief), T4 + failure (a), T5 (centerpiece theorem), T7 +
Corollaries, estimator + stress grid, dense ramp, MNIST.
- Appendix: full proofs (T1-T7 from note 33 + 34), T6 derivation + KS table,
the finite-time evidence chain details (T=5 table, oracle diagnostic, bias
decomposition, alpha_t curvature), negative results (31, 25), experiment
configs/seeds, teacher test-gap figure.
## 8. Open problems (state, do not solve)
Architecture-only finite-time theory: predict `rho` and the drift `K_s - K_0`
from initialization statistics (infinite-width recursions / deep-linear order
parameters); per-mode spectral erosion `rho_i` refining A2. (note 35)
## 9. Known booby traps for the writing pass
- Do not cite corr 0.977 for the closed form (mismatched; see note 37).
- Do not present the estimator as architecture-only; it conditions on (K_0,K_s).
- Do not present hard-k as a gap predictor; it is the null model only.
- `E[e]=k/P` and `rho` are different objects (null model vs measured share).
- The e0-distribution figure must be the post-fix (real-backward) rerun.
- Dense-run measured gaps live in `width_summary.csv` (train_gap_mean), 1 init
x 32 feedback seeds — say so in the caption.
- All speeds/losses use `L = ||r||^2/(2N)`; `e_0` is defined from speeds, not
losses.
|
