diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-06-09 21:35:51 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-06-09 21:35:51 -0500 |
| commit | f860ba2ffb409dc1b7dfab9534ed05cb25692269 (patch) | |
| tree | d30029bb8bb754a7c3c5f44fd13869e4e4178b84 | |
| parent | 84fa70be406248e660488ddc4c56ca5ea0034189 (diff) | |
Organize repo and add AAAI-27 writing plan
Root README orients the repo; notes/README reindexed by status
(writing pipeline / authoritative note per result / six historical
phases with superseded markers); scripts/README gains a status table.
Note 40 = reproduction manifest (claim -> command -> outputs -> note;
everything unlisted is not citable). Note 41 = the binding paper plan:
AAAI-27 constraints (7pp + refs, appendix not guaranteed read),
narrative, section/page budget, frozen theorem and figure numbering,
related-work list with per-cite rationale, writing rules, weekly
schedule to the 07-20/07-27 deadlines, and risk register.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| -rw-r--r-- | README.md | 38 | ||||
| -rw-r--r-- | notes/36_evidence_ledger.md | 6 | ||||
| -rw-r--r-- | notes/40_reproduction_manifest.md | 83 | ||||
| -rw-r--r-- | notes/41_paper_plan.md | 176 | ||||
| -rw-r--r-- | notes/README.md | 91 | ||||
| -rw-r--r-- | scripts/README.md | 38 |
6 files changed, 409 insertions, 23 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..171bd24 --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# facap — What Does Random Feedback Cost? + +Theory + experiments quantifying the training cost of **feedback alignment** +(FA: fixed random backward matrices instead of transposed weights) relative to +backpropagation in MLPs. Target venue: **AAAI-27** (abstract 2026-07-20, +paper 2026-07-27). + +## Results in one table + +| claim | headline number | +|---|---| +| static alignment is an exact Beta(1/2,(D-1)/2) law; log-volume cost adds to Theta(Ln^2) | 700k samples, KS p 0.17-0.99; Gamma(L,1) max KS 0.0042 | +| no prior-free feedback initialization beats isotropic (minimax 1/D) | exact theorem + lam_min table | +| initial FA/BP speed gap = hidden-layer BP speed share, exactly, any depth | synthetic 0.4378 vs 0.4403; MNIST diff <= 0.0089 over share 0.39-0.82 | +| one-hidden-layer e0 is exactly Gaussian | KS p 0.13-0.999, real-backward sampler | +| the finite-time gap is a closed-form **soft ramp** — no phase transition | dense T=30000: log-linear R^2 0.993; closed form corr 0.933 matched / 0.982 ensemble; lam_min corr 0.987 | +| a two-snapshot early-velocity estimator predicts the gap with no fit | 256 traj: MAE 0.0019, corr 0.999; stress grid 0.994-0.99989; MNIST MAE 0.0020 (3.9% rel) | +| optimization cost is invariant; test-side effect is task-dependent | teacher task: test gap sign flip (-0.120+/-0.013 at w=32) | + +## Layout + +- `notes/` — project memory. Entry points: `notes/41_paper_plan.md` (plan), + `notes/36_evidence_ledger.md` (numbers), `notes/40_reproduction_manifest.md` + (runs). `notes/README.md` indexes all 41 notes with status. +- `scripts/` — all experiments (python, float64, CPU; deterministic seeds). + `scripts/README.md` has the status table and recipes. +- `outputs/` — gitignored, regenerable. Paper-grade runs are exactly those in + `notes/40_reproduction_manifest.md`; the rest is exploratory history. +- `paper/` — AAAI-27 scaffolding (`main.tex` needs AuthorKit 27; + `preview.tex` compiles standalone via tectonic). `body.tex` is being + rewritten from the evidence ledger. + +## Setup + +```bash +pip install -r requirements.txt # numpy scipy matplotlib torch (+ torchvision for MNIST) +python scripts/real_data_validation.py --self-test # sanity: kernels vs autograd +``` diff --git a/notes/36_evidence_ledger.md b/notes/36_evidence_ledger.md index ed952bf..1028f48 100644 --- a/notes/36_evidence_ledger.md +++ b/notes/36_evidence_ledger.md @@ -155,6 +155,9 @@ what they are; (31) and (25) deserve explicit mention in the paper. ## 6. Figure plan (sources verified on disk) +> Final figure selection and placement now live in note 41 §5 (4 main +> figures); this table remains the source-of-truth for what exists on disk. + | fig | content | source | |---|---|---| | F1 | static law: Beta hist + QQ (+ Gamma(L,1) inset or appendix) | `capacity_empirical_validation` | @@ -166,6 +169,9 @@ what they are; (31) and (25) deserve explicit mention in the paper. ## 7. Main text vs appendix split +> Superseded by the binding section/page plan in note 41 §3/§8; kept as the +> first-pass allocation. + - Main: T1-T3 (brief), T4 + failure (a), T5 (centerpiece theorem), T7 + Corollaries, estimator + stress grid, dense ramp, MNIST. - Appendix: full proofs (T1-T7 from note 33 + 34), T6 derivation + KS table, diff --git a/notes/40_reproduction_manifest.md b/notes/40_reproduction_manifest.md new file mode 100644 index 0000000..7e18d98 --- /dev/null +++ b/notes/40_reproduction_manifest.md @@ -0,0 +1,83 @@ +# Reproduction Manifest (paper-grade runs only) + +One row per number/figure that can appear in the paper: the claim, the exact +command (or the note that records it), the output directory, and the note that +interprets it. `outputs/` is gitignored — these runs are the regenerable +ground truth behind the ledger (note 36). Anything in `outputs/` NOT listed +here is an exploratory/superseded/smoke run and must not be cited. + +Environment: python 3.13, torch (CPU, float64), `requirements.txt`. All runs +deterministic given the seeds shown. Long runs use `nohup`. + +## Static geometry (T1-T3 + overlap) + +| claim | command | outputs | note | +|---|---|---|---| +| Beta law, tails, multilayer product (700k samples, D=64..4096) | `python scripts/capacity_empirical_validation.py --dimensions 64 128 256 512 1024 2048 4096 --samples 100000 --batch-size 2048 --seed 123 --plot` | `capacity_empirical_validation/` | 02 | +| surprisal `S~Exp(1)`, `sum S ~ Gamma(L,1)` (max KS 0.0042) | `python scripts/multilayer_capacity_distribution.py --dimensions 64 256 1024 4096 --layers 1 2 4 8 16 --samples 100000 --batch-size 8192 --seed 456 --plot` | `multilayer_capacity_distribution/` | 02 | +| minimax: isotropic attains 1/D, alternatives below | `python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot` | `minimax_initialization/` | 02 | +| coverage distributions per scheme | `python scripts/initialization_distribution_matching.py --dimension 128 --target-samples 100000 --feedback-samples 100000 --batch-size 8192 --seed 2026 --subspace-dim 8 --anisotropy 64 --plot` | `initialization_distribution_matching/` | 02 | +| hard rank loss + kd/P overlap | `python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot` | `functional_capacity_overlap/` | 02 | + +## The soft ramp (dense sweep) + closed form (T7) + +| claim | command | outputs | note | +|---|---|---|---| +| dense T=30000 ramp: gap 0.1468 -> 0.000162, smooth, log-linear R^2=0.993 | `python scripts/downstream_capacity_sweep.py --task random --widths 20 22 24 26 28 30 32 34 36 38 40 --train-samples 128 --test-samples 1024 --probe-samples 64 --steps 30000 --lr 0.01 --optimizer sgd --init-seeds 1 --feedback-seeds 32 --data-seed 0 --skip-jacobian --plot --outdir outputs/phase_transition_dense_T30000_352traj` (config preserved in its `summary.json`) | `phase_transition_dense_T30000_352traj/` | 22 | +| closed form, matched data+init: corr 0.933 matched / 0.982 5-init geo mean; lam_min ensemble corr 0.987 | `python scripts/closed_form_soft_ramp.py --torch-threads 8` (reads the dense dir) | `closed_form_soft_ramp/` | 34, 37 | +| hard-k null model overpredicts (2x-250x) | `python scripts/soft_erosion_theory_vs_empirical.py` (reads the dense dir; see note 25 for the recorded run) | `soft_erosion_theory_vs_empirical/` | 25 | + +## Initial erosion (T5 + T6) + +| claim | command | outputs | note | +|---|---|---|---| +| `E_B[e_0]=0.4378+/-0.008` vs hidden share 0.4403; per-width errors <= 0.008 | `python scripts/actual_fa_initial_operator_moments.py --widths 16 24 32 48 64 96 --init-seeds 6 --feedback-samples 512 --torch-threads 8 --outdir outputs/actual_fa_initial_operator_moments` | `actual_fa_initial_operator_moments/` | 29 | +| e0 exactly Gaussian; KS p 0.13-0.999; sampler = real FA backward, linear identity <= 8.9e-16 | `python scripts/actual_fa_initial_erosion_distribution.py --widths 16 32 64 128 --init-seeds 3 --feedback-samples 4096 --torch-threads 8 --outdir outputs/actual_fa_initial_erosion_distribution` | `actual_fa_initial_erosion_distribution/` (pre-fix snapshot kept in `*_pre_fix_backup/`) | 32, 37 | + +## Finite-time chain (lazy validity -> drift -> estimator) + +| claim | command | outputs | note | +|---|---|---|---| +| T=5 frozen-K(0) local exactness: corr 0.99947, MAE 0.00083; horizon table T=1..50 | recorded invocation in note 10 (fixed width 64, N in {64..320}, SGD lr 1e-3, 4 init x 8 feedback) | `downstream_capacity_fixed_width64_sgd_T5_N*/`, `fa_tangent_kernel_fixed_width64_sgd_T5_*` | 10 | +| oracle time-varying K(t) removes T=50 error: gap MAE 0.0277 -> 0.000273 | `python scripts/finite_time_kernel_diagnostic.py --width 64 --train-samples 128 --steps 50 --lr 1e-3 --init-seeds 2 --feedback-seeds 4` | `finite_time_kernel_probe_T50_N128_8runs/` | 13 | +| scalar early predictors fail across N (corr 0.124) | `python scripts/early_kernel_predictors.py ...` (recorded in note 14) | `early_directional_predictors_T50_width64_N*/` | 14 | +| estimator, 256 traj: velocity MAE 0.00189, corr 0.99885; fixed K(0) 10x worse; retangent upper bracket | `python scripts/compressed_operator_predictor.py --width 64 --hidden-layers 2 --train-samples <N> --target-steps 50 --early-steps 20 --lr 1e-3 --init-seeds 4 --feedback-seeds 8` for N in {64,96,128,160,192,224,256,320}, then `python scripts/plot_compressed_operator_results.py` | `compressed_operator_s20_256traj_T50_width64_N*/`, `*_plots/` | 15 | +| stress grid: corr 0.994-0.99989 across depth {1,2,3} x width {32,64,96} x horizon {25,50,100} | per-setting runs recorded in note 16, then `python scripts/summarize_operator_stress_grid.py` + `python scripts/plot_operator_overlap_grid.py` | `stress_depth*_*/`, `operator_stress_grid_summary/` | 16, 17 | +| first-order derivative extrapolation diverges (MAE 270) | `python scripts/fa_tangent_hierarchy_derivative_probe.py` (eps 0.05; recorded in note 31) | `fa_tangent_hierarchy_derivative_probe_eps005/` | 31 | + +## Real data + generalization + +| claim | command | outputs | note | +|---|---|---|---| +| MNIST e0: 16 configs, share 0.39-0.82, abs diff <= 0.0089 | `python scripts/real_data_validation.py --part a --torch-threads 16` | `real_data_validation_mnist/e0_mnist_*` | 38 | +| MNIST estimator: 96 traj, velocity MAE 0.00196 (3.9% rel), bias -0.0006, corr 0.911; fixed MAE 0.0061 | `python scripts/real_data_validation.py --part b --est-widths 128 256 --lr-safety 0.05 --torch-threads 16` | `real_data_validation_mnist/estimator_mnist_*` | 38 | +| kernel code self-test vs autograd (<= 5.3e-15) | `python scripts/real_data_validation.py --self-test` | stdout | 38 | +| teacher task: train gap 0.249->0.0069 soft; test gap sign flip (-0.120+/-0.013 at w=32) | `python scripts/downstream_capacity_sweep.py --task mlp --teacher-width 64 --teacher-hidden-layers 2 --normalize-targets --widths 8 12 16 24 32 48 64 96 --train-samples 256 --test-samples 2048 --probe-samples 32 --steps 8000 --lr 0.01 --optimizer sgd --init-seeds 3 --feedback-seeds 8 --data-seed 7 --skip-jacobian --plot --outdir outputs/teacher_test_gap_sgd_T8000` then `python scripts/plot_teacher_test_gap.py` | `teacher_test_gap_sgd_T8000/` | 39 | + +## Figure -> file map (current best versions) + +| paper figure (plan in note 41) | file | +|---|---| +| soft ramp + closed form + ramp law | `closed_form_soft_ramp/closed_form_vs_measured_ramp.png` | +| e0 moment calibration (synthetic) | `actual_fa_initial_operator_moments/predicted_vs_empirical_initial_erosion_mean.png` | +| e0 Gaussian panels | `actual_fa_initial_erosion_distribution/e0_distribution_theory_vs_empirical.png` | +| estimator scatter (synthetic 256 traj) | `compressed_operator_s20_256traj_T50_width64_plots/T50_s20_prediction_scatter.png` | +| estimator trajectories + band | `compressed_operator_s20_256traj_T50_width64_plots/T50_s20_velocity_theory_band_empirical_trajectories.png` | +| MNIST e0 calibration | `real_data_validation_mnist/e0_mnist_calibration.png` | +| MNIST estimator scatter | `real_data_validation_mnist/estimator_mnist_scatter.png` | +| teacher train vs test gap | `teacher_test_gap_sgd_T8000/teacher_train_vs_test_gap.png` | +| static beta (appendix) | `capacity_empirical_validation/` plots | +| stress grid (appendix) | `operator_stress_grid_summary/stress_grid_*` | + +All paper figures will be re-rendered to AAAI column dimensions during the +writing pass (note 41, week 2); the files above are the data-correct sources. + +## Not paper-grade (do not cite) + +Everything else under `outputs/`, in particular: all `*smoke*`, all +`downstream_capacity_random_*` / `downstream_capacity_sweep*` Adam-era sweeps +(superseded by SGD runs; notes 05-07), `phase_transition_sgd_*` short-T probes +(superseded by the dense run), `compressed_operator_T50_*` and +`*_retangent_*` intermediate variants (subsumed by the s20 256-traj set + +stress grid), `trajectory_*` bridge-era runs (notes 02/08; historical), and +`fa_tangent_hierarchy_first_order_try/` (negative result recorded in note 31). diff --git a/notes/41_paper_plan.md b/notes/41_paper_plan.md new file mode 100644 index 0000000..2e6a4e2 --- /dev/null +++ b/notes/41_paper_plan.md @@ -0,0 +1,176 @@ +# AAAI-27 Paper Plan + +The single planning document for the writing phase. Inputs: evidence ledger +(note 36), reproduction manifest (note 40), corrections (note 37). Supersedes +note 03 and the figure list in note 35. + +## 0. Venue constraints (verified 2026-06-09) + +- **AAAI-27**: abstract due **2026-07-20**, full paper **2026-07-27** + (23:59:59 UTC-12); supplementary (technical appendix / code+data) due 3 days + after the paper deadline. Conference 2027-02-16..23, Montreal. +- **7 pages of technical content + unlimited reference-only pages.** +- Reviewers are NOT required to read supplementary material => every + load-bearing statement, number, and figure goes in the 7 pages; the appendix + carries proofs and extended tables only. +- Mandatory reproducibility checklist; plan to submit code+configs as + supplementary zip (manifest note 40 makes this mechanical). +- Format: AuthorKit 27 (`aaai27.sty`, not on CTAN — download required), + anonymous submission. + +## 1. Title + +Working: **"What Does Random Feedback Cost? The Exact Price of Feedback +Alignment in MLPs"** + +Alternates (decide at polish time): +- "What Does Random Feedback Cost? A Distributional Capacity and Operator-Gap + Analysis of Feedback Alignment" (current scaffolding title; long) +- "The Soft Price of Random Feedback: Exact Capacity and Operator-Gap Laws for + Feedback Alignment" + +## 2. The pitch (frozen narrative) + +A decade of FA work answered "can random feedback learn?" — we answer "what +does it cost?", exactly and with no fitted parameters. Three results: (i) the +static geometry of random feedback is an exact Beta law whose log-volume cost +adds to Theta(Ln^2), and (ii) no prior-free initialization can beat it +(minimax 1/D). (iii) The training cost it induces is *always nonzero, soft, +and computable*: at initialization the expected FA/BP speed gap equals exactly +the hidden-layer share of BP's learning speed (one BP backward pass predicts +it; verified on synthetic nets and MNIST); over training the gap follows a +closed-form spectral soft ramp (no phase transition — we refute the natural +redundancy-exhaustion story), and a two-snapshot early-velocity estimator +predicts the finite-time gap to ~2-4% with zero fitting. Capacity dilutes the +cost (k/P, lam_min growth); it never deletes it. + +One-line summary: **random feedback is not free; its cost is irreducible but +soft, and now exactly priced.** + +## 3. Section plan and page budget (7.00 pages) + +| sec | pages | content (all numbers from notes 36/40 only) | +|---|---|---| +| 1 Introduction | 0.90 | cost-not-capability framing; the wrong natural guess (phase transition) and what replaces it; 3 contributions; explicit claim-scope sentence (optimization gap; estimator is snapshot-conditional) | +| 2 Setup | 0.55 | BP/FA backward, speeds, `e_0`, `K_BP`, `K_FA`, `S_FA`; loss convention `L=||r||^2/2N`; "output layer identical" fact | +| 3 Static cost (T1+P1) | 0.65 | Beta law (Thm 1, 2-line proof in main), log-volume cost, additivity + Theta(Ln^2) (Prop 1, proof appendix); validation sentence: 700k samples KS table + Gamma(L,1) max KS 0.0042 | +| 4 No free lunch at init (T3) | 0.40 | minimax Thm 2 + 3-line proof + the "anisotropy helps some directions only by hurting others" reading; empirical lam_min table sentence | +| 5 The gap | 2.30 | (a) null model Prop 2 (`Beta(k/2,(P-k)/2)`, `P(e=0)=0`) + **its honest 2x-250x overprediction** => motivates operator object; (b) Thm 3 `E_B[e_0]=hidden share` + proof + Gaussian Prop 3; MNIST share-0.39-0.82 confirmation sentence; (c) A1/A2 stated plainly, Thm 4 closed form + Cor 1 (soft, analytic, no threshold) + Cor 2 (ramp law `log gap ~ -c rho lam_min`); (d) the lazy chain in one paragraph WITH numbers (T=5 corr 0.99947 -> oracle K(t) MAE 0.000273 -> estimator) + estimator definition + bracket (velocity low / retangent high) + bias origin (drift-velocity decay) | +| 6 Experiments | 1.50 | F1-F4 + numbers table; soft ramp (refutation), closed form matched 0.933 / ensemble 0.982 / lam_min 0.987; e0 0.4378 vs 0.4403 + KS; estimator 256-traj MAE 0.00189 corr 0.99885 + stress grid range; MNIST both parts; teacher test-gap sign flip (2 sentences + pointer to F4/appendix) | +| 7 Related work | 0.45 | four paragraphs: FA empirics; FA theory (Song-Xu-Lafferty complementarity via `E_B[H_FA]=0`); NTK/lazy; capacity/memorization | +| 8 Limitations + conclusion | 0.25 | scope (MLP, full-batch, squared loss; MNIST-subset real data; conv/transformers open); estimator conditional; architecture-only drift = open problem | + +Figure area is counted inside sections 5-6 (~1.6 pages of the budget). + +## 4. Theorem numbering (frozen) + +- **Thm 1** alignment Beta law; **Prop 1** multilayer additivity/scaling +- **Thm 2** prior-free minimax +- **Prop 2** random-subspace null model (erosion Beta, `E=k/P`) +- **Thm 3** initial erosion = hidden speed share; **Prop 3** exact Gaussian e0 + (one hidden layer) +- **Thm 4** closed-form gap (A1+A2); **Cor 1** soft/no-threshold; **Cor 2** + ramp law +- Estimator: definition + empirical claims only — never call it a theorem. + +## 5. Figures (4 main, AAAI two-column specs) + +| # | panels | source (note 40 map) | placement | +|---|---|---|---| +| F1 | soft ramp vs closed form (log-y, error bars, matched + ensemble + band) ; ramp law vs lam_min | `closed_form_soft_ramp/` | sec 6, full width | +| F2 | e0 calibration: synthetic widths ; MNIST configs (share 0.39-0.82) | `actual_fa_initial_operator_moments/` + `real_data_validation_mnist/` | sec 5b/6, column | +| F3 | estimator: synthetic 256-traj scatter ; MNIST 96-traj scatter | `compressed_operator_*_plots/` + `real_data_validation_mnist/` | sec 6, full width | +| F4 | teacher task: train gap (soft) ; test gap (sign flip) | `teacher_test_gap_sgd_T8000/` | sec 6, column | + +Appendix figures: static Beta/QQ + Gamma(L,1); e0 Gaussian panels; stress-grid +panels; T=5 horizon table-figure; oracle-K(t) diagnostic; estimator +trajectory band. All sources exist; re-render at column width with matched +fonts (week-2 task; one `paper/figures.py` script regenerating every figure +from the manifest runs, no manual edits). + +## 6. Related work (citation list with one-line why) + +FA empirics: +1. Lillicrap et al. 2016, Nat. Comms — FA works; alignment emerges. (origin) +2. Nokland 2016 — DFA in deep nets. (variant breadth) +3. Liao, Leibo, Poggio 2017 — sign symmetry; how important is weight symmetry. (initialization variants our Thm 2 prices) +4. Xiao et al. 2019 — sign-symmetry at ImageNet scale. (scaling empirics) +5. Bartunov et al. 2018 — bio-plausible methods stress-tested; FA degrades at scale. (the cost is visible empirically) +6. Launay et al. 2020 — DFA scales to modern architectures. (counterpoint; cost vs usability) +7. Akrout et al. 2019 — weight mirrors; Kolen-Pollack 1994 — learned transport. (ways to *pay* the cost rather than avoid it; contrast with fixed-B scope) +8. Crick 1989 / Grossberg 1987 — weight transport problem. (motivation, one cite) +9. Lillicrap et al. 2020, Nat Rev Neuro — backprop and the brain survey. (context) +10. Frenkel et al. 2021 — DRTP. (fixed-feedback family) + +FA theory: +11. Song, Xu, Lafferty 2021 — two-layer FA convergence; `K_FA=G+H_FA`. (the complement: our Thm 3 = `E_B[H_FA]=0` in residual direction; convergence != operator equality — note 27 wording) +12. Refinetti, d'Ascoli, Ohana, Goldt 2021 — align-then-memorise dynamics. (their alignment phase = our drift `K_s-K_0`) +13. Saxe et al. 2014 — deep linear dynamics. (route to architecture-only theory, future work) + +Kernels / lazy: +14. Jacot et al. 2018 — NTK. (the operator lens) +15. Lee et al. 2019 — wide nets as linear models. (lazy residual recursion) +16. Chizat, Oyallon, Bach 2019 — lazy training. (A1 scope) +17. Huang & Yau 2020 — neural tangent hierarchy. (the K_t-dynamics program our estimator instantiates empirically; note 30) + +Capacity / memorization: +18. Zhang et al. 2017 — fitting random labels. (the "capacity absorbs it" intuition we refute softly) +19. Montanari & Zhong 2022 — interpolation phase transition. (why a kink was a reasonable guess) +20. (optional) Arora et al. 2019 exact NTK computation; cite only if space. + +Rule: every citation earns a clause about how it relates to a *result* of +ours; no laundry lists. Bibliography moves to `aaai27.bst`-compatible `.bib`. + +## 7. Writing rules (binding) + +1. Numbers only from notes 36/40; verify against the CSV when transcribing. +2. Never cite the retired 0.977; closed form = 0.933 matched / 0.982 ensemble. +3. Estimator = "snapshot-conditional, no fitted parameters", never + architecture-only; state the lazy-regime scope (MNIST eta-sensitivity). +4. Hard-k appears only as a null model + reported failure. +5. Negative results (derivative blow-up MAE 270; scalar-gain corr 0.124) get + one sentence each in sec 5d/appendix — they justify the estimator's form. +6. Every theorem states its assumptions inline (independence, zero-mean, + output-layer-exact, A1/A2). +7. Prose style: every formula followed by a reading sentence; no unexplained + jargon ("erosion", "speed" defined once in sec 2). +8. The teacher test-gap sign flip is reported as scoping evidence, not as a + contribution claim. + +## 8. Appendix plan (supplementary PDF) + +A. Full proofs T1, P1, T2, P2, T3, P3 (source: note 33 sections; rewrite to + match main-text notation), T4 + corollaries (note 34). +B. The finite-time chain: T=5 horizon table, oracle-K(t) diagnostic, bias + decomposition + alpha_t curvature (notes 10/13/17). +C. Stress grid full table (note 16) + MNIST per-config table (note 38). +D. Negative results: hard-k overprediction table (25), derivative probe (31), + scalar gain (14). +E. Experiment details: all configs/seeds (= manifest note 40), hardware, + runtime; reproducibility checklist answers. +F. Teacher test-gap discussion + figure (note 39). + +## 9. Schedule (today = Mon 2026-06-09) + +| week | deliverable | exit criterion | +|---|---|---| +| 06-09..06-15 | wrap-up audit + this plan (**done**); download AuthorKit; `paper/figures.py` regenerating F1-F4 at AAAI specs | `main.tex` compiles with kit; 4 figures render | +| 06-16..06-22 | full body rewrite from ledger (sec 1-6), figures embedded | 7-page draft, every number traced to manifest | +| 06-23..06-29 | related work + `.bib`; appendix A-F; reproducibility checklist; code-release zip | complete submission package v1 | +| 06-30..07-06 | claims audit (read against ledger §9 + rules §7); number audit (against CSVs); style pass; external read if available | v2; no unverified number | +| 07-07..07-13 | buffer for surprises; abstract polish; figure freeze | v3 frozen except typos | +| 07-14..07-20 | abstract registration (**07-20**); final proof pass | submitted abstract | +| 07-21..07-27 | paper submission (**07-27**); supplementary (+3 days) | submitted | + +## 10. Risk register + +| risk | mitigation | +|---|---| +| "MLPs/synthetic only" | MNIST validation in main text (F2/F3); limitations sentence: conv/transformers open | +| "train gap — who cares about generalization?" | F4 sign flip + scoping paragraph: optimization cost is the invariant; test effect task-dependent | +| "estimator is conditional, not a theory" | stated upfront in intro + limitations; closed form supplies the mechanism level; open-problem paragraph | +| novelty vs Song-Xu-Lafferty | complementarity paragraph via `E_B[H_FA]=0` (note 27); we quantify cost, they prove convergence | +| MNIST estimator corr 0.911 looks weak vs 0.999 | report MAE-first (3.9% relative); explain narrow gap range; per-config table in appendix | +| 7-page overflow | appendix offload order: stress-grid table -> e0 Gaussian panels -> teacher figure -> Prop 3 proof sketch | +| AuthorKit/compile friction | week-1 task, not week-4 | +| reviewer asks for per-mode rho_i or architecture-only drift | named open problem with a concrete route (NTK recursion / deep-linear), framed as future work | diff --git a/notes/README.md b/notes/README.md index 76d8e46..ee608b6 100644 --- a/notes/README.md +++ b/notes/README.md @@ -1,37 +1,82 @@ # Notes -This folder collects working notes for the project tentatively titled: +Project memory for: **What Does Random Feedback Cost? (Feedback Alignment +capacity & operator-gap analysis)** — target AAAI-27. -**Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons** +## Start here (the writing pipeline) -Use this directory as the project memory. Keep rough ideas here before promoting them into a paper draft or experiments. +| doc | role | +|---|---| +| [41_paper_plan.md](41_paper_plan.md) | **the plan**: venue rules, narrative, section/page budget, theorem & figure numbering, related-work list, schedule, risks | +| [36_evidence_ledger.md](36_evidence_ledger.md) | **the numbers**: every claim with its corrected headline figure, claim discipline, booby traps | +| [40_reproduction_manifest.md](40_reproduction_manifest.md) | **the runs**: claim -> exact command -> output dir -> note; everything not listed is not citable | +| [37_wrapup_corrections.md](37_wrapup_corrections.md) | audit record: e0 sampler de-circularized; closed-form comparison matched (0.977 retired -> 0.933/0.982) | -## Start here (current state) +## Authoritative note per result -- [36_evidence_ledger.md](36_evidence_ledger.md): **the writing base** — full - evidence chain, corrected headline numbers, figure sources, claim discipline. -- [38_mnist_real_data_validation.md](38_mnist_real_data_validation.md): e0 - theorem + estimator replicated on MNIST (no fit). -- [39_teacher_test_gap.md](39_teacher_test_gap.md): teacher-task test gap — - optimization cost invariant, test effect task-dependent (sign flip). -- [37_wrapup_corrections.md](37_wrapup_corrections.md): pre-writing audit - fixes (e0 sampler now real FA backward; closed-form comparison now - data/init-matched — corr 0.977 retired, 0.933 matched / 0.982 ensemble). -- [35_current_results_consolidated.md](35_current_results_consolidated.md): contribution snapshot (numbers corrected per note 37). -- [34_closed_form_soft_ramp_theorem.md](34_closed_form_soft_ramp_theorem.md): closed-form gap law that quantifies the burden and explains the soft ramp. -- [18_contribution_roadmap.md](18_contribution_roadmap.md): contribution roadmap (includes the tangent-operator estimator). +| result | note | status | +|---|---|---| +| T1 Beta alignment law + capacity cost | 01 (statements/proofs), 02 (validation) | proven + validated | +| P1 multilayer additivity, Theta(Ln^2), Gamma(L,1) | 01, 02 | proven + validated | +| T2 prior-free minimax 1/D | 01, 02 | proven + validated | +| P2 random-subspace null model (and its honest failure as a predictor) | 26 (theorem), 25 (failure) | proven; overpredicts real FA | +| T3 initial erosion = hidden BP speed share | 29 | proven + validated (synthetic 0.4378/0.4403; MNIST note 38) | +| P3 exact Gaussian e0 (one hidden layer) | 32 + fix in 37 | proven + validated (real backward) | +| T4 closed-form soft-ramp gap law | 34 + correction in 37 | proven under A1+A2; corr 0.933 matched / 0.982 ensemble | +| early-velocity estimator (+ bracket, bias structure) | 15, 16, 17 | no-fit, snapshot-conditional; MAE 0.0019, corr 0.999 | +| dense soft ramp (refutes phase transition) | 22 | measured, 352 traj | +| finite-time validity chain (T=5 local; oracle K(t)) | 10, 13 | measured | +| MNIST real-data validation | 38 | measured | +| teacher-task test gap (sign flip) | 39 | measured | +| positioning (Song-Xu-Lafferty; effective-capacity lit; methods for K_t) | 27, 24, 30 | reference | +| proof-vs-estimator-vs-fit discipline | 28 | reference | +| appendix proof source material | 33 | reference (framing partly stale) | -## Files +## Historical chain (chronological; superseded items marked) -- [00_project_overview.md](00_project_overview.md): high-level thesis, contribution map, and scope. -- [01_theory_notes.md](01_theory_notes.md): notation, theorem candidates, proof sketches, and open theoretical gaps. -- [02_experiment_notes.md](02_experiment_notes.md): validation plan, metrics, datasets, and plotting ideas. -- [03_paper_outline.md](03_paper_outline.md): draft paper structure and section-level writing plan. +Phase 1 — static geometry (validated, still paper-grade): +- 00 project overview — **superseded framing** (banner added); 01 theory + notes — T1-T3 statements and proofs; 02 experiment notes — static + validation record (Beta/Gamma/minimax/overlap) + early trajectory probes; + 03 paper outline — **superseded** (banner added). -## Note Conventions +Phase 2 — capacity-margin era (Adam sweeps; superseded by SGD work): +- 04 downstream design; 05 random-task results; 06 peak-shift analysis — + all **superseded** by the Jacobian/NTK view of 07. + +Phase 3 — operator view and finite time: +- 07 BP capacity theory (the pivot to tangent operators); 08 SGD T=50 + validation (corr 0.97, 2x scale -> diagnosis); 09 fixed-width probe; + 10 **T=5 local exactness** (paper-grade); 11 finite-time roadmap; + 12 error decomposition (BP-too-strong/FA-too-weak); 13 **oracle K(t) + diagnostic** (paper-grade: drift is the whole error); 14 scalar predictors + fail (negative, kept); 15 **the estimator** (paper-grade); 16 **stress + grid** (paper-grade); 17 **bias structure** (paper-grade); 18 contribution + roadmap. + +Phase 4 — the phase-transition story dies: +- 19/20/21 kink-era analyses — **refuted** by longer training; 22 **dense + T=30000 soft ramp** (paper-grade); 23 soft-erosion revision. + +Phase 5 — exact theory of the burden: +- 24 literature + gap-theory positioning; 25 hard-k overprediction + (negative, kept); 26 null-model theorem + claim discipline; 27 + Song-Xu-Lafferty positioning; 28 proof-vs-fit status; 29 **initial-moment + theorem T3**; 30 methods survey for K_t dynamics; 31 derivative + extrapolation blows up (negative, kept); 32 **e0 Gaussian validation**; + 33 complete markdown draft (appendix source); 34 **closed-form law T4**; + 35 consolidated snapshot. + +Phase 6 — wrap-up for AAAI-27 (2026-06-09): +- 36 evidence ledger; 37 corrections; 38 **MNIST validation**; 39 **teacher + test gap**; 40 reproduction manifest; 41 paper plan. + +## Note conventions - Prefer formulas and precise claims over prose-only ideas. - Mark unproven statements as `Conjecture` or `Hypothesis`. -- Mark results ready for paper text as `Candidate theorem`, `Candidate proposition`, or `Candidate corollary`. +- Mark results ready for paper text as `Candidate theorem/proposition/corollary`. - Keep experiment ideas close to the metric they test. - When a note depends on an assumption, write the assumption explicitly. +- Never edit a historical note's results in place: append a dated correction + block and record the fix in a new note (as done in 34/35/37). diff --git a/scripts/README.md b/scripts/README.md index af529a3..51dcbe3 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,5 +1,43 @@ # Scripts +Status index (paper-grade commands live in `notes/40_reproduction_manifest.md`; +the recipes further down this file cover the static-geometry scripts). + +**Core experiment scripts (paper-grade):** + +| script | what it validates | note | +|---|---|---| +| `capacity_empirical_validation.py` | Beta law, tails, multilayer product | 02 | +| `multilayer_capacity_distribution.py` | surprisal Exp(1) / Gamma(L,1) | 02 | +| `minimax_initialization.py`, `initialization_distribution_matching.py` | minimax 1/D + coverage per scheme | 02 | +| `functional_capacity_overlap.py` | hard rank loss + kd/P overlap | 02 | +| `downstream_capacity_sweep.py` | BP/FA sweep engine (dense ramp run, teacher run) + shared MLP/grad infra | 22, 39 | +| `closed_form_soft_ramp.py` | closed-form gap law vs dense ramp (matched data/init + init band) | 34, 37 | +| `actual_fa_initial_operator_moments.py` | T3: E_B[e0] = hidden speed share | 29 | +| `actual_fa_initial_erosion_distribution.py` | P3: exact Gaussian e0 (real-backward sampler) | 32, 37 | +| `compressed_operator_predictor.py` + `plot_compressed_operator_results.py` | early-velocity estimator, 256-traj set | 15 | +| `summarize_operator_stress_grid.py` + `plot_operator_overlap_grid.py` | estimator stress grid | 16, 17 | +| `finite_time_kernel_diagnostic.py` | oracle time-varying K(t) vs fixed K(0) | 13 | +| `real_data_validation.py` | MNIST: T3 + estimator (has `--self-test`) | 38 | +| `plot_teacher_test_gap.py` | teacher train-vs-test gap figure | 39 | + +**Diagnostics / negative results (kept for the record):** +`fa_tangent_kernel_capacity.py` (kernel capacity probe, note 07), +`early_kernel_predictors.py` (scalar predictors fail, note 14), +`fa_tangent_hierarchy_derivative_probe.py` (derivative extrapolation blows +up, note 31), `soft_erosion_theory_vs_empirical.py` (hard-k overpredicts, +note 25), `static_alignment_beta.py` + `capacity_scaling.py` (small static +demos subsumed by the large validation). + +**Historical (bridge/ensemble era, superseded):** +`trajectory_mlp_fa.py`, `trajectory_ensemble.py`, +`trajectory_gap_distribution.py` (notes 02/08), +`plot_phase_transition*.py`, `plot_dense_phase_transition.py`, +`plot_capacity_transition_overlay.py` (kink-era visualizations, notes 19-22), +`plot_tangent_hierarchy_first_order_try.py` (note 31). + +--- + ## Static Alignment Beta Law Run: |
