diff options
Diffstat (limited to 'notes/36_evidence_ledger.md')
| -rw-r--r-- | notes/36_evidence_ledger.md | 184 |
1 files changed, 184 insertions, 0 deletions
diff --git a/notes/36_evidence_ledger.md b/notes/36_evidence_ledger.md new file mode 100644 index 0000000..63e797d --- /dev/null +++ b/notes/36_evidence_ledger.md @@ -0,0 +1,184 @@ +# Evidence Ledger (Writing Base for the AAAI Paper) + +Authoritative consolidation of every result, number, and figure source in this +project, compiled directly from notes 00-35, the scripts, and the raw outputs. +**Write the paper from this file**, not from any earlier draft: the earlier +LaTeX draft was written without a careful pass over the experimental record and +omits much of the evidence chain below. Status of every claim is marked. + +Numbers updated 2026-06-09 after the wrap-up corrections (note 37): the +closed-form comparison is now data/init-matched, and the e0-distribution +sampler runs the real FA backward pass. + +--- + +## 0. Scope and claim discipline + +- Setting: MLPs, squared loss `L = ||r||^2/(2N)`, full-batch training, + feedback alignment = fixed random backward matrices `B_l` replacing + `W_{l+1}^T`. Synthetic random-label / teacher tasks + MNIST-subset + validation (note 38). +- The paper quantifies the **optimization (train-loss) gap** FA pays relative + to BP. It does NOT claim FA fails to converge (compatible with Song-Xu- + Lafferty), does not predict test accuracy, and the estimator is + **conditional on two early kernel snapshots** — not architecture-only. +- "No fit" everywhere below means: no post-hoc scalar was tuned to match the + reported data. Predictions are computed from initialization (and, for the + estimator, one early snapshot). + +## 1. Theorems (proof status) + +| # | statement | status | source note | +|---|---|---|---| +| T1 | `Q = cos^2(a,b) ~ Beta(1/2,(D-1)/2)`, `E[Q]=1/D` for independent isotropic directions | exact | 00/01 | +| T2 | log-volume costs add: `C_all = sum_l C_l = Theta(L n^2)`; surprisal `S_l ~ Exp(1)`, `sum ~ Gamma(L,1)` | exact | 01 | +| T3 | prior-free minimax: `sup_mu inf_a E[(a^T b)^2] = 1/D`, attained by isotropic | exact | 01 | +| T4 | null model: Haar-random k-dim deletion gives `e(r) ~ Beta(k/2,(P-k)/2)`, `E[e]=k/P`, `P(e=0)=0` | exact (within null model) | 26 | +| T5 | **initial-erosion theorem**: `E_B[e_0 | W,r] = 1 - ||g_out^BP||^2 / sum_l ||g_l^BP||^2` (hidden speed share), any depth, any independent zero-mean feedback | exact | 29 | +| T6 | one-hidden-layer Gaussian FA: `e_0 | W,r` exactly Gaussian with explicit mean/std (linear in B) | exact | 32 | +| T7 | closed-form gap under A1 (lazy) + A2 (scalar erosion `S_FA ~= rho K_BP`): `gap_T = (1/2N) sum_i c_i^2 [(1-eta rho lam_i/N)^{2T} - (1-eta lam_i/N)^{2T}]`; analytic => soft ramp, no threshold unless `rho -> 0`; converged regime `log gap_T ~= const - (2 eta T/N) rho lam_min` | exact under A1+A2 | 34 | + +A2 is exact in the residual direction by T5 (`rho = 1 - E_B[e_0]` = output +share); extending it to all eigenmodes is the one modeling step. A1 is the +standard lazy approximation whose breakdown is *measured* (item 3c below). + +## 2. Static geometry: validation record + +| claim | evidence | numbers | output dir | +|---|---|---|---| +| Beta law | 700k samples, D = 64..4096 | KS stat 0.002-0.004, p 0.17-0.99; mean/q99 match to 4 digits | `capacity_empirical_validation` | +| capacity tail cost | same run | max smoothed cost error 0.20 nats, mean 0.019; multilayer product max err 0.46 nats | same | +| `Gamma(L,1)` surprisal | 100k per (D,L), D in {64..4096}, L in {1..16} | max KS **0.0042**, mean abs mean-error 0.0042 | `multilayer_capacity_distribution` | +| minimax | 5 schemes, D=32/128 | isotropic lam_min 0.0291 vs 1/32=0.0313; rademacher 0.0289; anisotropic 0.0061; subspace/axis 0 | `minimax_initialization`, `initialization_distribution_matching` | +| functional overlap | P=96,d=24, trials=100 | hard loss exact (0 until k>P-d, then k-(P-d)); soft overlap kd/P to 3 digits | `functional_capacity_overlap` | + +## 3. The gap pillar: four-level evidence chain + +### (a) Null model + its honest failure (negative result, keep in paper) + +- T4 proves nonzero soft cost in the cleanest model; `P(e=0)=0`. +- **Hard-k overpredicts real FA**: predicted gap 0.040 at margin +130 vs + measured 0.00016 (250x); ~2x on local erosion (measured local erosion 0.23- + 0.39 vs Beta-mean 0.63-0.73) (note 25, `soft_erosion_theory_vs_empirical`). + Role in paper: motivates the *operator* object; hard constraint count is + not the effective burden. + +### (b) Exact at t=0 (T5 + T6) + +- Mean theorem, 2-hidden-layer, real FA backward, 6 widths x 6 inits x 512 + draws: `E_B[e_0] = 0.4378 +/- 0.008` vs theory `0.4403`; per-width errors + 0.0003-0.008 (note 29, `actual_fa_initial_operator_moments`). +- Independent t=0 anchor from the (superseded) recovery probe: 0.4026+/-0.0066 + vs 0.4071 (run log, note 37). +- Full distribution (T6), **sampler = real FA backward after note-37 fix**: + 4 widths x 3 inits x 4096 draws, KS p in [0.13, 0.999], all means/stds match + to 3-4 digits; linear identity `<C,B>` verified against the real backward to + max err 8.9e-16 (`actual_fa_initial_erosion_distribution`). +- **MNIST (note 38)**: 16 configs (width {64,256} x depth {1,2} x 4 inits, + 512 real-backward draws each): hidden share spans **0.39-0.82** and + `E_B[e_0]` lands on it with |diff| 0.0001-0.0089 (all within ~2 stderr) — + the theorem holds on real data with no fit + (`real_data_validation_mnist/e0_mnist_rows.csv`). + +### (c) Finite time: the lazy chain (previous draft omitted ALL of this) + +1. **T=5 local exactness** (note 10): frozen K(0), 256 traj, 8 N values: + corr **0.99947**, MAE **0.00083**, mean ratio 0.993. Horizon table: ratio + 1.028/0.987/0.942/0.884/0.814 at T=1/5/10/20/50. => the operator object is + right; freezing is a *local* theorem. +2. **Drift is the whole error** (note 13): at T=50, oracle time-varying K(t) + product cuts gap MAE 0.0277 -> **0.000273** (100x); BP loss MAE 0.0254 -> + 0.00024. => no higher-order Taylor mystery; model K_t and you are done. +3. **Estimator** `K_hat_t = K_0 + (t/s)(K_s - K_0)` (notes 15-17): 256 traj, + T=50, s=20: MAE **0.00189**, bias -0.0015, corr **0.99885**; fixed-K(0) + MAE 0.0185 (10x worse); re-tangent MAE 0.0049 biased high => **no-fit + bracket** `linear <= empirical <= retangent`. + Stress grid (7 settings): depth 1/2/3, width 32/64/96, horizon 25/50/100: + linear MAE 0.00023-0.0057, corr 0.994-0.99989 + (`operator_stress_grid_summary`). +4. **Bias structure** (note 17): residual -0.0021 gap bias = BP-loss + overprediction (+0.0026), FA near-unbiased (+0.0005); cause: drift velocity + decays after s (alpha_t < t/s; e.g. true alpha_50 = 1.46 (BP) / 1.17 (FA) + vs linear 2.5). Honest error story, no fitted correction. +5. **MNIST replication (note 38)**: same protocol, width 256, depth 2, + N in {128,256,512}, 48 traj: see `real_data_validation_mnist/`. + +### (d) The soft ramp + closed form (T7) + +- Dense sweep (note 22): T=30000, 11 widths (20..40), 352 traj: gap + 0.1468 -> 0.000162 across margin -190 -> +130, smooth, log-linear + (`log gap ~ -0.0209*margin`, R^2 **0.993**), **no kink at margin 0, + nonzero at positive margin** => refutes the redundancy-exhaustion phase + transition (and notes 19-21 document how the kink hypothesis died as T grew: + near-margin-0 gap 0.297 at T=3000, 0.051 at T=10^4, 0.0037 at T=3x10^4). +- Closed form vs measured (**matched data + init after note-37 fix**): + - matched single init: corr(log,log) = **0.933**; + - 5-init geometric mean (same data): corr **0.982**; ensemble-mean + lam_min(w) rises 0.026 -> 0.763 and explains the ramp at corr **0.987**; + rho stable 0.68-0.79 across widths. + - the previously reported corr 0.977 came from a comparison computed on the + wrong data seed and init seed; retire that number (note 37). + - frozen-init compresses the range (under-predicts large-gap end, over- + predicts small-gap tail) — exactly the drift the estimator corrects; + state this as mechanism-vs-magnitude. + (`closed_form_soft_ramp/closed_form_vs_measured.csv`, `closed_form_per_seed.csv`) + +## 4. Negative results worth reporting (reviewer-proofing) + +| failed approach | number | note | +|---|---|---| +| hard-k as gap predictor | overpredicts 2x-250x | 25 | +| infinitesimal derivative extrapolation `K_0 + t dK_0` | MAE **270** vs 0.024 baseline (diverges) | 31 | +| scalar directional-gain model across N | corr 0.124 (vs operator 0.999) | 14 | +| trajectory bridge on long Adam horizons | shape corr 0.97 but 2-3x scale error | 02/08 | +| phase-transition (kink) hypothesis | dissolves as T grows; dense T=30000 smooth | 19-22 | + +These define why the final objects (operator erosion + early velocity) are +what they are; (31) and (25) deserve explicit mention in the paper. + +## 5. Real-data and generalization coverage (new, note 38) + +- MNIST e0 + estimator: `outputs/real_data_validation_mnist/` (script + `scripts/real_data_validation.py`, self-test verifies kernels vs autograd + at 1e-15). +- Teacher-task test gap (SGD, widths 8..96): `outputs/teacher_test_gap_sgd_T8000/` + — addresses "what about generalization": train-gap story plus a descriptive + test-MSE companion. Random-label sweeps measure optimization gap by design. + +## 6. Figure plan (sources verified on disk) + +| fig | content | source | +|---|---|---| +| F1 | static law: Beta hist + QQ (+ Gamma(L,1) inset or appendix) | `capacity_empirical_validation` | +| F2 | dense soft ramp + closed form (matched + 5-init mean + band) + lam_min ramp law | `closed_form_soft_ramp/closed_form_vs_measured_ramp.png` | +| F3 | e0: moment calibration + Gaussian distribution panels (real backward) | `actual_fa_initial_operator_moments`, `actual_fa_initial_erosion_distribution` | +| F4 | estimator: predicted-vs-empirical scatter, 256 traj + stress grid summary | `compressed_operator_s20_256traj_T50_width64_plots`, `operator_stress_grid_summary` | +| F5 | MNIST: e0 calibration + estimator scatter (real data) | `real_data_validation_mnist` | +| F6 (optional/appendix) | T=5 exactness + oracle-K(t) diagnostic (the lazy chain) | notes 10/13 outputs | + +## 7. Main text vs appendix split + +- Main: T1-T3 (brief), T4 + failure (a), T5 (centerpiece theorem), T7 + + Corollaries, estimator + stress grid, dense ramp, MNIST. +- Appendix: full proofs (T1-T7 from note 33 + 34), T6 derivation + KS table, + the finite-time evidence chain details (T=5 table, oracle diagnostic, bias + decomposition, alpha_t curvature), negative results (31, 25), experiment + configs/seeds, teacher test-gap figure. + +## 8. Open problems (state, do not solve) + +Architecture-only finite-time theory: predict `rho` and the drift `K_s - K_0` +from initialization statistics (infinite-width recursions / deep-linear order +parameters); per-mode spectral erosion `rho_i` refining A2. (note 35) + +## 9. Known booby traps for the writing pass + +- Do not cite corr 0.977 for the closed form (mismatched; see note 37). +- Do not present the estimator as architecture-only; it conditions on (K_0,K_s). +- Do not present hard-k as a gap predictor; it is the null model only. +- `E[e]=k/P` and `rho` are different objects (null model vs measured share). +- The e0-distribution figure must be the post-fix (real-backward) rerun. +- Dense-run measured gaps live in `width_summary.csv` (train_gap_mean), 1 init + x 32 feedback seeds — say so in the caption. +- All speeds/losses use `L = ||r||^2/(2N)`; `e_0` is defined from speeds, not + losses. |