summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-06-09 21:35:51 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-06-09 21:35:51 -0500
commitf860ba2ffb409dc1b7dfab9534ed05cb25692269 (patch)
treed30029bb8bb754a7c3c5f44fd13869e4e4178b84
parent84fa70be406248e660488ddc4c56ca5ea0034189 (diff)
Organize repo and add AAAI-27 writing plan
Root README orients the repo; notes/README reindexed by status (writing pipeline / authoritative note per result / six historical phases with superseded markers); scripts/README gains a status table. Note 40 = reproduction manifest (claim -> command -> outputs -> note; everything unlisted is not citable). Note 41 = the binding paper plan: AAAI-27 constraints (7pp + refs, appendix not guaranteed read), narrative, section/page budget, frozen theorem and figure numbering, related-work list with per-cite rationale, writing rules, weekly schedule to the 07-20/07-27 deadlines, and risk register. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-rw-r--r--README.md38
-rw-r--r--notes/36_evidence_ledger.md6
-rw-r--r--notes/40_reproduction_manifest.md83
-rw-r--r--notes/41_paper_plan.md176
-rw-r--r--notes/README.md91
-rw-r--r--scripts/README.md38
6 files changed, 409 insertions, 23 deletions
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..171bd24
--- /dev/null
+++ b/README.md
@@ -0,0 +1,38 @@
+# facap — What Does Random Feedback Cost?
+
+Theory + experiments quantifying the training cost of **feedback alignment**
+(FA: fixed random backward matrices instead of transposed weights) relative to
+backpropagation in MLPs. Target venue: **AAAI-27** (abstract 2026-07-20,
+paper 2026-07-27).
+
+## Results in one table
+
+| claim | headline number |
+|---|---|
+| static alignment is an exact Beta(1/2,(D-1)/2) law; log-volume cost adds to Theta(Ln^2) | 700k samples, KS p 0.17-0.99; Gamma(L,1) max KS 0.0042 |
+| no prior-free feedback initialization beats isotropic (minimax 1/D) | exact theorem + lam_min table |
+| initial FA/BP speed gap = hidden-layer BP speed share, exactly, any depth | synthetic 0.4378 vs 0.4403; MNIST diff <= 0.0089 over share 0.39-0.82 |
+| one-hidden-layer e0 is exactly Gaussian | KS p 0.13-0.999, real-backward sampler |
+| the finite-time gap is a closed-form **soft ramp** — no phase transition | dense T=30000: log-linear R^2 0.993; closed form corr 0.933 matched / 0.982 ensemble; lam_min corr 0.987 |
+| a two-snapshot early-velocity estimator predicts the gap with no fit | 256 traj: MAE 0.0019, corr 0.999; stress grid 0.994-0.99989; MNIST MAE 0.0020 (3.9% rel) |
+| optimization cost is invariant; test-side effect is task-dependent | teacher task: test gap sign flip (-0.120+/-0.013 at w=32) |
+
+## Layout
+
+- `notes/` — project memory. Entry points: `notes/41_paper_plan.md` (plan),
+ `notes/36_evidence_ledger.md` (numbers), `notes/40_reproduction_manifest.md`
+ (runs). `notes/README.md` indexes all 41 notes with status.
+- `scripts/` — all experiments (python, float64, CPU; deterministic seeds).
+ `scripts/README.md` has the status table and recipes.
+- `outputs/` — gitignored, regenerable. Paper-grade runs are exactly those in
+ `notes/40_reproduction_manifest.md`; the rest is exploratory history.
+- `paper/` — AAAI-27 scaffolding (`main.tex` needs AuthorKit 27;
+ `preview.tex` compiles standalone via tectonic). `body.tex` is being
+ rewritten from the evidence ledger.
+
+## Setup
+
+```bash
+pip install -r requirements.txt # numpy scipy matplotlib torch (+ torchvision for MNIST)
+python scripts/real_data_validation.py --self-test # sanity: kernels vs autograd
+```
diff --git a/notes/36_evidence_ledger.md b/notes/36_evidence_ledger.md
index ed952bf..1028f48 100644
--- a/notes/36_evidence_ledger.md
+++ b/notes/36_evidence_ledger.md
@@ -155,6 +155,9 @@ what they are; (31) and (25) deserve explicit mention in the paper.
## 6. Figure plan (sources verified on disk)
+> Final figure selection and placement now live in note 41 §5 (4 main
+> figures); this table remains the source-of-truth for what exists on disk.
+
| fig | content | source |
|---|---|---|
| F1 | static law: Beta hist + QQ (+ Gamma(L,1) inset or appendix) | `capacity_empirical_validation` |
@@ -166,6 +169,9 @@ what they are; (31) and (25) deserve explicit mention in the paper.
## 7. Main text vs appendix split
+> Superseded by the binding section/page plan in note 41 §3/§8; kept as the
+> first-pass allocation.
+
- Main: T1-T3 (brief), T4 + failure (a), T5 (centerpiece theorem), T7 +
Corollaries, estimator + stress grid, dense ramp, MNIST.
- Appendix: full proofs (T1-T7 from note 33 + 34), T6 derivation + KS table,
diff --git a/notes/40_reproduction_manifest.md b/notes/40_reproduction_manifest.md
new file mode 100644
index 0000000..7e18d98
--- /dev/null
+++ b/notes/40_reproduction_manifest.md
@@ -0,0 +1,83 @@
+# Reproduction Manifest (paper-grade runs only)
+
+One row per number/figure that can appear in the paper: the claim, the exact
+command (or the note that records it), the output directory, and the note that
+interprets it. `outputs/` is gitignored — these runs are the regenerable
+ground truth behind the ledger (note 36). Anything in `outputs/` NOT listed
+here is an exploratory/superseded/smoke run and must not be cited.
+
+Environment: python 3.13, torch (CPU, float64), `requirements.txt`. All runs
+deterministic given the seeds shown. Long runs use `nohup`.
+
+## Static geometry (T1-T3 + overlap)
+
+| claim | command | outputs | note |
+|---|---|---|---|
+| Beta law, tails, multilayer product (700k samples, D=64..4096) | `python scripts/capacity_empirical_validation.py --dimensions 64 128 256 512 1024 2048 4096 --samples 100000 --batch-size 2048 --seed 123 --plot` | `capacity_empirical_validation/` | 02 |
+| surprisal `S~Exp(1)`, `sum S ~ Gamma(L,1)` (max KS 0.0042) | `python scripts/multilayer_capacity_distribution.py --dimensions 64 256 1024 4096 --layers 1 2 4 8 16 --samples 100000 --batch-size 8192 --seed 456 --plot` | `multilayer_capacity_distribution/` | 02 |
+| minimax: isotropic attains 1/D, alternatives below | `python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot` | `minimax_initialization/` | 02 |
+| coverage distributions per scheme | `python scripts/initialization_distribution_matching.py --dimension 128 --target-samples 100000 --feedback-samples 100000 --batch-size 8192 --seed 2026 --subspace-dim 8 --anisotropy 64 --plot` | `initialization_distribution_matching/` | 02 |
+| hard rank loss + kd/P overlap | `python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot` | `functional_capacity_overlap/` | 02 |
+
+## The soft ramp (dense sweep) + closed form (T7)
+
+| claim | command | outputs | note |
+|---|---|---|---|
+| dense T=30000 ramp: gap 0.1468 -> 0.000162, smooth, log-linear R^2=0.993 | `python scripts/downstream_capacity_sweep.py --task random --widths 20 22 24 26 28 30 32 34 36 38 40 --train-samples 128 --test-samples 1024 --probe-samples 64 --steps 30000 --lr 0.01 --optimizer sgd --init-seeds 1 --feedback-seeds 32 --data-seed 0 --skip-jacobian --plot --outdir outputs/phase_transition_dense_T30000_352traj` (config preserved in its `summary.json`) | `phase_transition_dense_T30000_352traj/` | 22 |
+| closed form, matched data+init: corr 0.933 matched / 0.982 5-init geo mean; lam_min ensemble corr 0.987 | `python scripts/closed_form_soft_ramp.py --torch-threads 8` (reads the dense dir) | `closed_form_soft_ramp/` | 34, 37 |
+| hard-k null model overpredicts (2x-250x) | `python scripts/soft_erosion_theory_vs_empirical.py` (reads the dense dir; see note 25 for the recorded run) | `soft_erosion_theory_vs_empirical/` | 25 |
+
+## Initial erosion (T5 + T6)
+
+| claim | command | outputs | note |
+|---|---|---|---|
+| `E_B[e_0]=0.4378+/-0.008` vs hidden share 0.4403; per-width errors <= 0.008 | `python scripts/actual_fa_initial_operator_moments.py --widths 16 24 32 48 64 96 --init-seeds 6 --feedback-samples 512 --torch-threads 8 --outdir outputs/actual_fa_initial_operator_moments` | `actual_fa_initial_operator_moments/` | 29 |
+| e0 exactly Gaussian; KS p 0.13-0.999; sampler = real FA backward, linear identity <= 8.9e-16 | `python scripts/actual_fa_initial_erosion_distribution.py --widths 16 32 64 128 --init-seeds 3 --feedback-samples 4096 --torch-threads 8 --outdir outputs/actual_fa_initial_erosion_distribution` | `actual_fa_initial_erosion_distribution/` (pre-fix snapshot kept in `*_pre_fix_backup/`) | 32, 37 |
+
+## Finite-time chain (lazy validity -> drift -> estimator)
+
+| claim | command | outputs | note |
+|---|---|---|---|
+| T=5 frozen-K(0) local exactness: corr 0.99947, MAE 0.00083; horizon table T=1..50 | recorded invocation in note 10 (fixed width 64, N in {64..320}, SGD lr 1e-3, 4 init x 8 feedback) | `downstream_capacity_fixed_width64_sgd_T5_N*/`, `fa_tangent_kernel_fixed_width64_sgd_T5_*` | 10 |
+| oracle time-varying K(t) removes T=50 error: gap MAE 0.0277 -> 0.000273 | `python scripts/finite_time_kernel_diagnostic.py --width 64 --train-samples 128 --steps 50 --lr 1e-3 --init-seeds 2 --feedback-seeds 4` | `finite_time_kernel_probe_T50_N128_8runs/` | 13 |
+| scalar early predictors fail across N (corr 0.124) | `python scripts/early_kernel_predictors.py ...` (recorded in note 14) | `early_directional_predictors_T50_width64_N*/` | 14 |
+| estimator, 256 traj: velocity MAE 0.00189, corr 0.99885; fixed K(0) 10x worse; retangent upper bracket | `python scripts/compressed_operator_predictor.py --width 64 --hidden-layers 2 --train-samples <N> --target-steps 50 --early-steps 20 --lr 1e-3 --init-seeds 4 --feedback-seeds 8` for N in {64,96,128,160,192,224,256,320}, then `python scripts/plot_compressed_operator_results.py` | `compressed_operator_s20_256traj_T50_width64_N*/`, `*_plots/` | 15 |
+| stress grid: corr 0.994-0.99989 across depth {1,2,3} x width {32,64,96} x horizon {25,50,100} | per-setting runs recorded in note 16, then `python scripts/summarize_operator_stress_grid.py` + `python scripts/plot_operator_overlap_grid.py` | `stress_depth*_*/`, `operator_stress_grid_summary/` | 16, 17 |
+| first-order derivative extrapolation diverges (MAE 270) | `python scripts/fa_tangent_hierarchy_derivative_probe.py` (eps 0.05; recorded in note 31) | `fa_tangent_hierarchy_derivative_probe_eps005/` | 31 |
+
+## Real data + generalization
+
+| claim | command | outputs | note |
+|---|---|---|---|
+| MNIST e0: 16 configs, share 0.39-0.82, abs diff <= 0.0089 | `python scripts/real_data_validation.py --part a --torch-threads 16` | `real_data_validation_mnist/e0_mnist_*` | 38 |
+| MNIST estimator: 96 traj, velocity MAE 0.00196 (3.9% rel), bias -0.0006, corr 0.911; fixed MAE 0.0061 | `python scripts/real_data_validation.py --part b --est-widths 128 256 --lr-safety 0.05 --torch-threads 16` | `real_data_validation_mnist/estimator_mnist_*` | 38 |
+| kernel code self-test vs autograd (<= 5.3e-15) | `python scripts/real_data_validation.py --self-test` | stdout | 38 |
+| teacher task: train gap 0.249->0.0069 soft; test gap sign flip (-0.120+/-0.013 at w=32) | `python scripts/downstream_capacity_sweep.py --task mlp --teacher-width 64 --teacher-hidden-layers 2 --normalize-targets --widths 8 12 16 24 32 48 64 96 --train-samples 256 --test-samples 2048 --probe-samples 32 --steps 8000 --lr 0.01 --optimizer sgd --init-seeds 3 --feedback-seeds 8 --data-seed 7 --skip-jacobian --plot --outdir outputs/teacher_test_gap_sgd_T8000` then `python scripts/plot_teacher_test_gap.py` | `teacher_test_gap_sgd_T8000/` | 39 |
+
+## Figure -> file map (current best versions)
+
+| paper figure (plan in note 41) | file |
+|---|---|
+| soft ramp + closed form + ramp law | `closed_form_soft_ramp/closed_form_vs_measured_ramp.png` |
+| e0 moment calibration (synthetic) | `actual_fa_initial_operator_moments/predicted_vs_empirical_initial_erosion_mean.png` |
+| e0 Gaussian panels | `actual_fa_initial_erosion_distribution/e0_distribution_theory_vs_empirical.png` |
+| estimator scatter (synthetic 256 traj) | `compressed_operator_s20_256traj_T50_width64_plots/T50_s20_prediction_scatter.png` |
+| estimator trajectories + band | `compressed_operator_s20_256traj_T50_width64_plots/T50_s20_velocity_theory_band_empirical_trajectories.png` |
+| MNIST e0 calibration | `real_data_validation_mnist/e0_mnist_calibration.png` |
+| MNIST estimator scatter | `real_data_validation_mnist/estimator_mnist_scatter.png` |
+| teacher train vs test gap | `teacher_test_gap_sgd_T8000/teacher_train_vs_test_gap.png` |
+| static beta (appendix) | `capacity_empirical_validation/` plots |
+| stress grid (appendix) | `operator_stress_grid_summary/stress_grid_*` |
+
+All paper figures will be re-rendered to AAAI column dimensions during the
+writing pass (note 41, week 2); the files above are the data-correct sources.
+
+## Not paper-grade (do not cite)
+
+Everything else under `outputs/`, in particular: all `*smoke*`, all
+`downstream_capacity_random_*` / `downstream_capacity_sweep*` Adam-era sweeps
+(superseded by SGD runs; notes 05-07), `phase_transition_sgd_*` short-T probes
+(superseded by the dense run), `compressed_operator_T50_*` and
+`*_retangent_*` intermediate variants (subsumed by the s20 256-traj set +
+stress grid), `trajectory_*` bridge-era runs (notes 02/08; historical), and
+`fa_tangent_hierarchy_first_order_try/` (negative result recorded in note 31).
diff --git a/notes/41_paper_plan.md b/notes/41_paper_plan.md
new file mode 100644
index 0000000..2e6a4e2
--- /dev/null
+++ b/notes/41_paper_plan.md
@@ -0,0 +1,176 @@
+# AAAI-27 Paper Plan
+
+The single planning document for the writing phase. Inputs: evidence ledger
+(note 36), reproduction manifest (note 40), corrections (note 37). Supersedes
+note 03 and the figure list in note 35.
+
+## 0. Venue constraints (verified 2026-06-09)
+
+- **AAAI-27**: abstract due **2026-07-20**, full paper **2026-07-27**
+ (23:59:59 UTC-12); supplementary (technical appendix / code+data) due 3 days
+ after the paper deadline. Conference 2027-02-16..23, Montreal.
+- **7 pages of technical content + unlimited reference-only pages.**
+- Reviewers are NOT required to read supplementary material => every
+ load-bearing statement, number, and figure goes in the 7 pages; the appendix
+ carries proofs and extended tables only.
+- Mandatory reproducibility checklist; plan to submit code+configs as
+ supplementary zip (manifest note 40 makes this mechanical).
+- Format: AuthorKit 27 (`aaai27.sty`, not on CTAN — download required),
+ anonymous submission.
+
+## 1. Title
+
+Working: **"What Does Random Feedback Cost? The Exact Price of Feedback
+Alignment in MLPs"**
+
+Alternates (decide at polish time):
+- "What Does Random Feedback Cost? A Distributional Capacity and Operator-Gap
+ Analysis of Feedback Alignment" (current scaffolding title; long)
+- "The Soft Price of Random Feedback: Exact Capacity and Operator-Gap Laws for
+ Feedback Alignment"
+
+## 2. The pitch (frozen narrative)
+
+A decade of FA work answered "can random feedback learn?" — we answer "what
+does it cost?", exactly and with no fitted parameters. Three results: (i) the
+static geometry of random feedback is an exact Beta law whose log-volume cost
+adds to Theta(Ln^2), and (ii) no prior-free initialization can beat it
+(minimax 1/D). (iii) The training cost it induces is *always nonzero, soft,
+and computable*: at initialization the expected FA/BP speed gap equals exactly
+the hidden-layer share of BP's learning speed (one BP backward pass predicts
+it; verified on synthetic nets and MNIST); over training the gap follows a
+closed-form spectral soft ramp (no phase transition — we refute the natural
+redundancy-exhaustion story), and a two-snapshot early-velocity estimator
+predicts the finite-time gap to ~2-4% with zero fitting. Capacity dilutes the
+cost (k/P, lam_min growth); it never deletes it.
+
+One-line summary: **random feedback is not free; its cost is irreducible but
+soft, and now exactly priced.**
+
+## 3. Section plan and page budget (7.00 pages)
+
+| sec | pages | content (all numbers from notes 36/40 only) |
+|---|---|---|
+| 1 Introduction | 0.90 | cost-not-capability framing; the wrong natural guess (phase transition) and what replaces it; 3 contributions; explicit claim-scope sentence (optimization gap; estimator is snapshot-conditional) |
+| 2 Setup | 0.55 | BP/FA backward, speeds, `e_0`, `K_BP`, `K_FA`, `S_FA`; loss convention `L=||r||^2/2N`; "output layer identical" fact |
+| 3 Static cost (T1+P1) | 0.65 | Beta law (Thm 1, 2-line proof in main), log-volume cost, additivity + Theta(Ln^2) (Prop 1, proof appendix); validation sentence: 700k samples KS table + Gamma(L,1) max KS 0.0042 |
+| 4 No free lunch at init (T3) | 0.40 | minimax Thm 2 + 3-line proof + the "anisotropy helps some directions only by hurting others" reading; empirical lam_min table sentence |
+| 5 The gap | 2.30 | (a) null model Prop 2 (`Beta(k/2,(P-k)/2)`, `P(e=0)=0`) + **its honest 2x-250x overprediction** => motivates operator object; (b) Thm 3 `E_B[e_0]=hidden share` + proof + Gaussian Prop 3; MNIST share-0.39-0.82 confirmation sentence; (c) A1/A2 stated plainly, Thm 4 closed form + Cor 1 (soft, analytic, no threshold) + Cor 2 (ramp law `log gap ~ -c rho lam_min`); (d) the lazy chain in one paragraph WITH numbers (T=5 corr 0.99947 -> oracle K(t) MAE 0.000273 -> estimator) + estimator definition + bracket (velocity low / retangent high) + bias origin (drift-velocity decay) |
+| 6 Experiments | 1.50 | F1-F4 + numbers table; soft ramp (refutation), closed form matched 0.933 / ensemble 0.982 / lam_min 0.987; e0 0.4378 vs 0.4403 + KS; estimator 256-traj MAE 0.00189 corr 0.99885 + stress grid range; MNIST both parts; teacher test-gap sign flip (2 sentences + pointer to F4/appendix) |
+| 7 Related work | 0.45 | four paragraphs: FA empirics; FA theory (Song-Xu-Lafferty complementarity via `E_B[H_FA]=0`); NTK/lazy; capacity/memorization |
+| 8 Limitations + conclusion | 0.25 | scope (MLP, full-batch, squared loss; MNIST-subset real data; conv/transformers open); estimator conditional; architecture-only drift = open problem |
+
+Figure area is counted inside sections 5-6 (~1.6 pages of the budget).
+
+## 4. Theorem numbering (frozen)
+
+- **Thm 1** alignment Beta law; **Prop 1** multilayer additivity/scaling
+- **Thm 2** prior-free minimax
+- **Prop 2** random-subspace null model (erosion Beta, `E=k/P`)
+- **Thm 3** initial erosion = hidden speed share; **Prop 3** exact Gaussian e0
+ (one hidden layer)
+- **Thm 4** closed-form gap (A1+A2); **Cor 1** soft/no-threshold; **Cor 2**
+ ramp law
+- Estimator: definition + empirical claims only — never call it a theorem.
+
+## 5. Figures (4 main, AAAI two-column specs)
+
+| # | panels | source (note 40 map) | placement |
+|---|---|---|---|
+| F1 | soft ramp vs closed form (log-y, error bars, matched + ensemble + band) ; ramp law vs lam_min | `closed_form_soft_ramp/` | sec 6, full width |
+| F2 | e0 calibration: synthetic widths ; MNIST configs (share 0.39-0.82) | `actual_fa_initial_operator_moments/` + `real_data_validation_mnist/` | sec 5b/6, column |
+| F3 | estimator: synthetic 256-traj scatter ; MNIST 96-traj scatter | `compressed_operator_*_plots/` + `real_data_validation_mnist/` | sec 6, full width |
+| F4 | teacher task: train gap (soft) ; test gap (sign flip) | `teacher_test_gap_sgd_T8000/` | sec 6, column |
+
+Appendix figures: static Beta/QQ + Gamma(L,1); e0 Gaussian panels; stress-grid
+panels; T=5 horizon table-figure; oracle-K(t) diagnostic; estimator
+trajectory band. All sources exist; re-render at column width with matched
+fonts (week-2 task; one `paper/figures.py` script regenerating every figure
+from the manifest runs, no manual edits).
+
+## 6. Related work (citation list with one-line why)
+
+FA empirics:
+1. Lillicrap et al. 2016, Nat. Comms — FA works; alignment emerges. (origin)
+2. Nokland 2016 — DFA in deep nets. (variant breadth)
+3. Liao, Leibo, Poggio 2017 — sign symmetry; how important is weight symmetry. (initialization variants our Thm 2 prices)
+4. Xiao et al. 2019 — sign-symmetry at ImageNet scale. (scaling empirics)
+5. Bartunov et al. 2018 — bio-plausible methods stress-tested; FA degrades at scale. (the cost is visible empirically)
+6. Launay et al. 2020 — DFA scales to modern architectures. (counterpoint; cost vs usability)
+7. Akrout et al. 2019 — weight mirrors; Kolen-Pollack 1994 — learned transport. (ways to *pay* the cost rather than avoid it; contrast with fixed-B scope)
+8. Crick 1989 / Grossberg 1987 — weight transport problem. (motivation, one cite)
+9. Lillicrap et al. 2020, Nat Rev Neuro — backprop and the brain survey. (context)
+10. Frenkel et al. 2021 — DRTP. (fixed-feedback family)
+
+FA theory:
+11. Song, Xu, Lafferty 2021 — two-layer FA convergence; `K_FA=G+H_FA`. (the complement: our Thm 3 = `E_B[H_FA]=0` in residual direction; convergence != operator equality — note 27 wording)
+12. Refinetti, d'Ascoli, Ohana, Goldt 2021 — align-then-memorise dynamics. (their alignment phase = our drift `K_s-K_0`)
+13. Saxe et al. 2014 — deep linear dynamics. (route to architecture-only theory, future work)
+
+Kernels / lazy:
+14. Jacot et al. 2018 — NTK. (the operator lens)
+15. Lee et al. 2019 — wide nets as linear models. (lazy residual recursion)
+16. Chizat, Oyallon, Bach 2019 — lazy training. (A1 scope)
+17. Huang & Yau 2020 — neural tangent hierarchy. (the K_t-dynamics program our estimator instantiates empirically; note 30)
+
+Capacity / memorization:
+18. Zhang et al. 2017 — fitting random labels. (the "capacity absorbs it" intuition we refute softly)
+19. Montanari & Zhong 2022 — interpolation phase transition. (why a kink was a reasonable guess)
+20. (optional) Arora et al. 2019 exact NTK computation; cite only if space.
+
+Rule: every citation earns a clause about how it relates to a *result* of
+ours; no laundry lists. Bibliography moves to `aaai27.bst`-compatible `.bib`.
+
+## 7. Writing rules (binding)
+
+1. Numbers only from notes 36/40; verify against the CSV when transcribing.
+2. Never cite the retired 0.977; closed form = 0.933 matched / 0.982 ensemble.
+3. Estimator = "snapshot-conditional, no fitted parameters", never
+ architecture-only; state the lazy-regime scope (MNIST eta-sensitivity).
+4. Hard-k appears only as a null model + reported failure.
+5. Negative results (derivative blow-up MAE 270; scalar-gain corr 0.124) get
+ one sentence each in sec 5d/appendix — they justify the estimator's form.
+6. Every theorem states its assumptions inline (independence, zero-mean,
+ output-layer-exact, A1/A2).
+7. Prose style: every formula followed by a reading sentence; no unexplained
+ jargon ("erosion", "speed" defined once in sec 2).
+8. The teacher test-gap sign flip is reported as scoping evidence, not as a
+ contribution claim.
+
+## 8. Appendix plan (supplementary PDF)
+
+A. Full proofs T1, P1, T2, P2, T3, P3 (source: note 33 sections; rewrite to
+ match main-text notation), T4 + corollaries (note 34).
+B. The finite-time chain: T=5 horizon table, oracle-K(t) diagnostic, bias
+ decomposition + alpha_t curvature (notes 10/13/17).
+C. Stress grid full table (note 16) + MNIST per-config table (note 38).
+D. Negative results: hard-k overprediction table (25), derivative probe (31),
+ scalar gain (14).
+E. Experiment details: all configs/seeds (= manifest note 40), hardware,
+ runtime; reproducibility checklist answers.
+F. Teacher test-gap discussion + figure (note 39).
+
+## 9. Schedule (today = Mon 2026-06-09)
+
+| week | deliverable | exit criterion |
+|---|---|---|
+| 06-09..06-15 | wrap-up audit + this plan (**done**); download AuthorKit; `paper/figures.py` regenerating F1-F4 at AAAI specs | `main.tex` compiles with kit; 4 figures render |
+| 06-16..06-22 | full body rewrite from ledger (sec 1-6), figures embedded | 7-page draft, every number traced to manifest |
+| 06-23..06-29 | related work + `.bib`; appendix A-F; reproducibility checklist; code-release zip | complete submission package v1 |
+| 06-30..07-06 | claims audit (read against ledger §9 + rules §7); number audit (against CSVs); style pass; external read if available | v2; no unverified number |
+| 07-07..07-13 | buffer for surprises; abstract polish; figure freeze | v3 frozen except typos |
+| 07-14..07-20 | abstract registration (**07-20**); final proof pass | submitted abstract |
+| 07-21..07-27 | paper submission (**07-27**); supplementary (+3 days) | submitted |
+
+## 10. Risk register
+
+| risk | mitigation |
+|---|---|
+| "MLPs/synthetic only" | MNIST validation in main text (F2/F3); limitations sentence: conv/transformers open |
+| "train gap — who cares about generalization?" | F4 sign flip + scoping paragraph: optimization cost is the invariant; test effect task-dependent |
+| "estimator is conditional, not a theory" | stated upfront in intro + limitations; closed form supplies the mechanism level; open-problem paragraph |
+| novelty vs Song-Xu-Lafferty | complementarity paragraph via `E_B[H_FA]=0` (note 27); we quantify cost, they prove convergence |
+| MNIST estimator corr 0.911 looks weak vs 0.999 | report MAE-first (3.9% relative); explain narrow gap range; per-config table in appendix |
+| 7-page overflow | appendix offload order: stress-grid table -> e0 Gaussian panels -> teacher figure -> Prop 3 proof sketch |
+| AuthorKit/compile friction | week-1 task, not week-4 |
+| reviewer asks for per-mode rho_i or architecture-only drift | named open problem with a concrete route (NTK recursion / deep-linear), framed as future work |
diff --git a/notes/README.md b/notes/README.md
index 76d8e46..ee608b6 100644
--- a/notes/README.md
+++ b/notes/README.md
@@ -1,37 +1,82 @@
# Notes
-This folder collects working notes for the project tentatively titled:
+Project memory for: **What Does Random Feedback Cost? (Feedback Alignment
+capacity & operator-gap analysis)** — target AAAI-27.
-**Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons**
+## Start here (the writing pipeline)
-Use this directory as the project memory. Keep rough ideas here before promoting them into a paper draft or experiments.
+| doc | role |
+|---|---|
+| [41_paper_plan.md](41_paper_plan.md) | **the plan**: venue rules, narrative, section/page budget, theorem & figure numbering, related-work list, schedule, risks |
+| [36_evidence_ledger.md](36_evidence_ledger.md) | **the numbers**: every claim with its corrected headline figure, claim discipline, booby traps |
+| [40_reproduction_manifest.md](40_reproduction_manifest.md) | **the runs**: claim -> exact command -> output dir -> note; everything not listed is not citable |
+| [37_wrapup_corrections.md](37_wrapup_corrections.md) | audit record: e0 sampler de-circularized; closed-form comparison matched (0.977 retired -> 0.933/0.982) |
-## Start here (current state)
+## Authoritative note per result
-- [36_evidence_ledger.md](36_evidence_ledger.md): **the writing base** — full
- evidence chain, corrected headline numbers, figure sources, claim discipline.
-- [38_mnist_real_data_validation.md](38_mnist_real_data_validation.md): e0
- theorem + estimator replicated on MNIST (no fit).
-- [39_teacher_test_gap.md](39_teacher_test_gap.md): teacher-task test gap —
- optimization cost invariant, test effect task-dependent (sign flip).
-- [37_wrapup_corrections.md](37_wrapup_corrections.md): pre-writing audit
- fixes (e0 sampler now real FA backward; closed-form comparison now
- data/init-matched — corr 0.977 retired, 0.933 matched / 0.982 ensemble).
-- [35_current_results_consolidated.md](35_current_results_consolidated.md): contribution snapshot (numbers corrected per note 37).
-- [34_closed_form_soft_ramp_theorem.md](34_closed_form_soft_ramp_theorem.md): closed-form gap law that quantifies the burden and explains the soft ramp.
-- [18_contribution_roadmap.md](18_contribution_roadmap.md): contribution roadmap (includes the tangent-operator estimator).
+| result | note | status |
+|---|---|---|
+| T1 Beta alignment law + capacity cost | 01 (statements/proofs), 02 (validation) | proven + validated |
+| P1 multilayer additivity, Theta(Ln^2), Gamma(L,1) | 01, 02 | proven + validated |
+| T2 prior-free minimax 1/D | 01, 02 | proven + validated |
+| P2 random-subspace null model (and its honest failure as a predictor) | 26 (theorem), 25 (failure) | proven; overpredicts real FA |
+| T3 initial erosion = hidden BP speed share | 29 | proven + validated (synthetic 0.4378/0.4403; MNIST note 38) |
+| P3 exact Gaussian e0 (one hidden layer) | 32 + fix in 37 | proven + validated (real backward) |
+| T4 closed-form soft-ramp gap law | 34 + correction in 37 | proven under A1+A2; corr 0.933 matched / 0.982 ensemble |
+| early-velocity estimator (+ bracket, bias structure) | 15, 16, 17 | no-fit, snapshot-conditional; MAE 0.0019, corr 0.999 |
+| dense soft ramp (refutes phase transition) | 22 | measured, 352 traj |
+| finite-time validity chain (T=5 local; oracle K(t)) | 10, 13 | measured |
+| MNIST real-data validation | 38 | measured |
+| teacher-task test gap (sign flip) | 39 | measured |
+| positioning (Song-Xu-Lafferty; effective-capacity lit; methods for K_t) | 27, 24, 30 | reference |
+| proof-vs-estimator-vs-fit discipline | 28 | reference |
+| appendix proof source material | 33 | reference (framing partly stale) |
-## Files
+## Historical chain (chronological; superseded items marked)
-- [00_project_overview.md](00_project_overview.md): high-level thesis, contribution map, and scope.
-- [01_theory_notes.md](01_theory_notes.md): notation, theorem candidates, proof sketches, and open theoretical gaps.
-- [02_experiment_notes.md](02_experiment_notes.md): validation plan, metrics, datasets, and plotting ideas.
-- [03_paper_outline.md](03_paper_outline.md): draft paper structure and section-level writing plan.
+Phase 1 — static geometry (validated, still paper-grade):
+- 00 project overview — **superseded framing** (banner added); 01 theory
+ notes — T1-T3 statements and proofs; 02 experiment notes — static
+ validation record (Beta/Gamma/minimax/overlap) + early trajectory probes;
+ 03 paper outline — **superseded** (banner added).
-## Note Conventions
+Phase 2 — capacity-margin era (Adam sweeps; superseded by SGD work):
+- 04 downstream design; 05 random-task results; 06 peak-shift analysis —
+ all **superseded** by the Jacobian/NTK view of 07.
+
+Phase 3 — operator view and finite time:
+- 07 BP capacity theory (the pivot to tangent operators); 08 SGD T=50
+ validation (corr 0.97, 2x scale -> diagnosis); 09 fixed-width probe;
+ 10 **T=5 local exactness** (paper-grade); 11 finite-time roadmap;
+ 12 error decomposition (BP-too-strong/FA-too-weak); 13 **oracle K(t)
+ diagnostic** (paper-grade: drift is the whole error); 14 scalar predictors
+ fail (negative, kept); 15 **the estimator** (paper-grade); 16 **stress
+ grid** (paper-grade); 17 **bias structure** (paper-grade); 18 contribution
+ roadmap.
+
+Phase 4 — the phase-transition story dies:
+- 19/20/21 kink-era analyses — **refuted** by longer training; 22 **dense
+ T=30000 soft ramp** (paper-grade); 23 soft-erosion revision.
+
+Phase 5 — exact theory of the burden:
+- 24 literature + gap-theory positioning; 25 hard-k overprediction
+ (negative, kept); 26 null-model theorem + claim discipline; 27
+ Song-Xu-Lafferty positioning; 28 proof-vs-fit status; 29 **initial-moment
+ theorem T3**; 30 methods survey for K_t dynamics; 31 derivative
+ extrapolation blows up (negative, kept); 32 **e0 Gaussian validation**;
+ 33 complete markdown draft (appendix source); 34 **closed-form law T4**;
+ 35 consolidated snapshot.
+
+Phase 6 — wrap-up for AAAI-27 (2026-06-09):
+- 36 evidence ledger; 37 corrections; 38 **MNIST validation**; 39 **teacher
+ test gap**; 40 reproduction manifest; 41 paper plan.
+
+## Note conventions
- Prefer formulas and precise claims over prose-only ideas.
- Mark unproven statements as `Conjecture` or `Hypothesis`.
-- Mark results ready for paper text as `Candidate theorem`, `Candidate proposition`, or `Candidate corollary`.
+- Mark results ready for paper text as `Candidate theorem/proposition/corollary`.
- Keep experiment ideas close to the metric they test.
- When a note depends on an assumption, write the assumption explicitly.
+- Never edit a historical note's results in place: append a dated correction
+ block and record the fix in a new note (as done in 34/35/37).
diff --git a/scripts/README.md b/scripts/README.md
index af529a3..51dcbe3 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -1,5 +1,43 @@
# Scripts
+Status index (paper-grade commands live in `notes/40_reproduction_manifest.md`;
+the recipes further down this file cover the static-geometry scripts).
+
+**Core experiment scripts (paper-grade):**
+
+| script | what it validates | note |
+|---|---|---|
+| `capacity_empirical_validation.py` | Beta law, tails, multilayer product | 02 |
+| `multilayer_capacity_distribution.py` | surprisal Exp(1) / Gamma(L,1) | 02 |
+| `minimax_initialization.py`, `initialization_distribution_matching.py` | minimax 1/D + coverage per scheme | 02 |
+| `functional_capacity_overlap.py` | hard rank loss + kd/P overlap | 02 |
+| `downstream_capacity_sweep.py` | BP/FA sweep engine (dense ramp run, teacher run) + shared MLP/grad infra | 22, 39 |
+| `closed_form_soft_ramp.py` | closed-form gap law vs dense ramp (matched data/init + init band) | 34, 37 |
+| `actual_fa_initial_operator_moments.py` | T3: E_B[e0] = hidden speed share | 29 |
+| `actual_fa_initial_erosion_distribution.py` | P3: exact Gaussian e0 (real-backward sampler) | 32, 37 |
+| `compressed_operator_predictor.py` + `plot_compressed_operator_results.py` | early-velocity estimator, 256-traj set | 15 |
+| `summarize_operator_stress_grid.py` + `plot_operator_overlap_grid.py` | estimator stress grid | 16, 17 |
+| `finite_time_kernel_diagnostic.py` | oracle time-varying K(t) vs fixed K(0) | 13 |
+| `real_data_validation.py` | MNIST: T3 + estimator (has `--self-test`) | 38 |
+| `plot_teacher_test_gap.py` | teacher train-vs-test gap figure | 39 |
+
+**Diagnostics / negative results (kept for the record):**
+`fa_tangent_kernel_capacity.py` (kernel capacity probe, note 07),
+`early_kernel_predictors.py` (scalar predictors fail, note 14),
+`fa_tangent_hierarchy_derivative_probe.py` (derivative extrapolation blows
+up, note 31), `soft_erosion_theory_vs_empirical.py` (hard-k overpredicts,
+note 25), `static_alignment_beta.py` + `capacity_scaling.py` (small static
+demos subsumed by the large validation).
+
+**Historical (bridge/ensemble era, superseded):**
+`trajectory_mlp_fa.py`, `trajectory_ensemble.py`,
+`trajectory_gap_distribution.py` (notes 02/08),
+`plot_phase_transition*.py`, `plot_dense_phase_transition.py`,
+`plot_capacity_transition_overlay.py` (kink-era visualizations, notes 19-22),
+`plot_tangent_hierarchy_first_order_try.py` (note 31).
+
+---
+
## Static Alignment Beta Law
Run: