diff options
Diffstat (limited to 'notes/41_paper_plan.md')
| -rw-r--r-- | notes/41_paper_plan.md | 176 |
1 files changed, 176 insertions, 0 deletions
diff --git a/notes/41_paper_plan.md b/notes/41_paper_plan.md new file mode 100644 index 0000000..2e6a4e2 --- /dev/null +++ b/notes/41_paper_plan.md @@ -0,0 +1,176 @@ +# AAAI-27 Paper Plan + +The single planning document for the writing phase. Inputs: evidence ledger +(note 36), reproduction manifest (note 40), corrections (note 37). Supersedes +note 03 and the figure list in note 35. + +## 0. Venue constraints (verified 2026-06-09) + +- **AAAI-27**: abstract due **2026-07-20**, full paper **2026-07-27** + (23:59:59 UTC-12); supplementary (technical appendix / code+data) due 3 days + after the paper deadline. Conference 2027-02-16..23, Montreal. +- **7 pages of technical content + unlimited reference-only pages.** +- Reviewers are NOT required to read supplementary material => every + load-bearing statement, number, and figure goes in the 7 pages; the appendix + carries proofs and extended tables only. +- Mandatory reproducibility checklist; plan to submit code+configs as + supplementary zip (manifest note 40 makes this mechanical). +- Format: AuthorKit 27 (`aaai27.sty`, not on CTAN — download required), + anonymous submission. + +## 1. Title + +Working: **"What Does Random Feedback Cost? The Exact Price of Feedback +Alignment in MLPs"** + +Alternates (decide at polish time): +- "What Does Random Feedback Cost? A Distributional Capacity and Operator-Gap + Analysis of Feedback Alignment" (current scaffolding title; long) +- "The Soft Price of Random Feedback: Exact Capacity and Operator-Gap Laws for + Feedback Alignment" + +## 2. The pitch (frozen narrative) + +A decade of FA work answered "can random feedback learn?" — we answer "what +does it cost?", exactly and with no fitted parameters. Three results: (i) the +static geometry of random feedback is an exact Beta law whose log-volume cost +adds to Theta(Ln^2), and (ii) no prior-free initialization can beat it +(minimax 1/D). (iii) The training cost it induces is *always nonzero, soft, +and computable*: at initialization the expected FA/BP speed gap equals exactly +the hidden-layer share of BP's learning speed (one BP backward pass predicts +it; verified on synthetic nets and MNIST); over training the gap follows a +closed-form spectral soft ramp (no phase transition — we refute the natural +redundancy-exhaustion story), and a two-snapshot early-velocity estimator +predicts the finite-time gap to ~2-4% with zero fitting. Capacity dilutes the +cost (k/P, lam_min growth); it never deletes it. + +One-line summary: **random feedback is not free; its cost is irreducible but +soft, and now exactly priced.** + +## 3. Section plan and page budget (7.00 pages) + +| sec | pages | content (all numbers from notes 36/40 only) | +|---|---|---| +| 1 Introduction | 0.90 | cost-not-capability framing; the wrong natural guess (phase transition) and what replaces it; 3 contributions; explicit claim-scope sentence (optimization gap; estimator is snapshot-conditional) | +| 2 Setup | 0.55 | BP/FA backward, speeds, `e_0`, `K_BP`, `K_FA`, `S_FA`; loss convention `L=||r||^2/2N`; "output layer identical" fact | +| 3 Static cost (T1+P1) | 0.65 | Beta law (Thm 1, 2-line proof in main), log-volume cost, additivity + Theta(Ln^2) (Prop 1, proof appendix); validation sentence: 700k samples KS table + Gamma(L,1) max KS 0.0042 | +| 4 No free lunch at init (T3) | 0.40 | minimax Thm 2 + 3-line proof + the "anisotropy helps some directions only by hurting others" reading; empirical lam_min table sentence | +| 5 The gap | 2.30 | (a) null model Prop 2 (`Beta(k/2,(P-k)/2)`, `P(e=0)=0`) + **its honest 2x-250x overprediction** => motivates operator object; (b) Thm 3 `E_B[e_0]=hidden share` + proof + Gaussian Prop 3; MNIST share-0.39-0.82 confirmation sentence; (c) A1/A2 stated plainly, Thm 4 closed form + Cor 1 (soft, analytic, no threshold) + Cor 2 (ramp law `log gap ~ -c rho lam_min`); (d) the lazy chain in one paragraph WITH numbers (T=5 corr 0.99947 -> oracle K(t) MAE 0.000273 -> estimator) + estimator definition + bracket (velocity low / retangent high) + bias origin (drift-velocity decay) | +| 6 Experiments | 1.50 | F1-F4 + numbers table; soft ramp (refutation), closed form matched 0.933 / ensemble 0.982 / lam_min 0.987; e0 0.4378 vs 0.4403 + KS; estimator 256-traj MAE 0.00189 corr 0.99885 + stress grid range; MNIST both parts; teacher test-gap sign flip (2 sentences + pointer to F4/appendix) | +| 7 Related work | 0.45 | four paragraphs: FA empirics; FA theory (Song-Xu-Lafferty complementarity via `E_B[H_FA]=0`); NTK/lazy; capacity/memorization | +| 8 Limitations + conclusion | 0.25 | scope (MLP, full-batch, squared loss; MNIST-subset real data; conv/transformers open); estimator conditional; architecture-only drift = open problem | + +Figure area is counted inside sections 5-6 (~1.6 pages of the budget). + +## 4. Theorem numbering (frozen) + +- **Thm 1** alignment Beta law; **Prop 1** multilayer additivity/scaling +- **Thm 2** prior-free minimax +- **Prop 2** random-subspace null model (erosion Beta, `E=k/P`) +- **Thm 3** initial erosion = hidden speed share; **Prop 3** exact Gaussian e0 + (one hidden layer) +- **Thm 4** closed-form gap (A1+A2); **Cor 1** soft/no-threshold; **Cor 2** + ramp law +- Estimator: definition + empirical claims only — never call it a theorem. + +## 5. Figures (4 main, AAAI two-column specs) + +| # | panels | source (note 40 map) | placement | +|---|---|---|---| +| F1 | soft ramp vs closed form (log-y, error bars, matched + ensemble + band) ; ramp law vs lam_min | `closed_form_soft_ramp/` | sec 6, full width | +| F2 | e0 calibration: synthetic widths ; MNIST configs (share 0.39-0.82) | `actual_fa_initial_operator_moments/` + `real_data_validation_mnist/` | sec 5b/6, column | +| F3 | estimator: synthetic 256-traj scatter ; MNIST 96-traj scatter | `compressed_operator_*_plots/` + `real_data_validation_mnist/` | sec 6, full width | +| F4 | teacher task: train gap (soft) ; test gap (sign flip) | `teacher_test_gap_sgd_T8000/` | sec 6, column | + +Appendix figures: static Beta/QQ + Gamma(L,1); e0 Gaussian panels; stress-grid +panels; T=5 horizon table-figure; oracle-K(t) diagnostic; estimator +trajectory band. All sources exist; re-render at column width with matched +fonts (week-2 task; one `paper/figures.py` script regenerating every figure +from the manifest runs, no manual edits). + +## 6. Related work (citation list with one-line why) + +FA empirics: +1. Lillicrap et al. 2016, Nat. Comms — FA works; alignment emerges. (origin) +2. Nokland 2016 — DFA in deep nets. (variant breadth) +3. Liao, Leibo, Poggio 2017 — sign symmetry; how important is weight symmetry. (initialization variants our Thm 2 prices) +4. Xiao et al. 2019 — sign-symmetry at ImageNet scale. (scaling empirics) +5. Bartunov et al. 2018 — bio-plausible methods stress-tested; FA degrades at scale. (the cost is visible empirically) +6. Launay et al. 2020 — DFA scales to modern architectures. (counterpoint; cost vs usability) +7. Akrout et al. 2019 — weight mirrors; Kolen-Pollack 1994 — learned transport. (ways to *pay* the cost rather than avoid it; contrast with fixed-B scope) +8. Crick 1989 / Grossberg 1987 — weight transport problem. (motivation, one cite) +9. Lillicrap et al. 2020, Nat Rev Neuro — backprop and the brain survey. (context) +10. Frenkel et al. 2021 — DRTP. (fixed-feedback family) + +FA theory: +11. Song, Xu, Lafferty 2021 — two-layer FA convergence; `K_FA=G+H_FA`. (the complement: our Thm 3 = `E_B[H_FA]=0` in residual direction; convergence != operator equality — note 27 wording) +12. Refinetti, d'Ascoli, Ohana, Goldt 2021 — align-then-memorise dynamics. (their alignment phase = our drift `K_s-K_0`) +13. Saxe et al. 2014 — deep linear dynamics. (route to architecture-only theory, future work) + +Kernels / lazy: +14. Jacot et al. 2018 — NTK. (the operator lens) +15. Lee et al. 2019 — wide nets as linear models. (lazy residual recursion) +16. Chizat, Oyallon, Bach 2019 — lazy training. (A1 scope) +17. Huang & Yau 2020 — neural tangent hierarchy. (the K_t-dynamics program our estimator instantiates empirically; note 30) + +Capacity / memorization: +18. Zhang et al. 2017 — fitting random labels. (the "capacity absorbs it" intuition we refute softly) +19. Montanari & Zhong 2022 — interpolation phase transition. (why a kink was a reasonable guess) +20. (optional) Arora et al. 2019 exact NTK computation; cite only if space. + +Rule: every citation earns a clause about how it relates to a *result* of +ours; no laundry lists. Bibliography moves to `aaai27.bst`-compatible `.bib`. + +## 7. Writing rules (binding) + +1. Numbers only from notes 36/40; verify against the CSV when transcribing. +2. Never cite the retired 0.977; closed form = 0.933 matched / 0.982 ensemble. +3. Estimator = "snapshot-conditional, no fitted parameters", never + architecture-only; state the lazy-regime scope (MNIST eta-sensitivity). +4. Hard-k appears only as a null model + reported failure. +5. Negative results (derivative blow-up MAE 270; scalar-gain corr 0.124) get + one sentence each in sec 5d/appendix — they justify the estimator's form. +6. Every theorem states its assumptions inline (independence, zero-mean, + output-layer-exact, A1/A2). +7. Prose style: every formula followed by a reading sentence; no unexplained + jargon ("erosion", "speed" defined once in sec 2). +8. The teacher test-gap sign flip is reported as scoping evidence, not as a + contribution claim. + +## 8. Appendix plan (supplementary PDF) + +A. Full proofs T1, P1, T2, P2, T3, P3 (source: note 33 sections; rewrite to + match main-text notation), T4 + corollaries (note 34). +B. The finite-time chain: T=5 horizon table, oracle-K(t) diagnostic, bias + decomposition + alpha_t curvature (notes 10/13/17). +C. Stress grid full table (note 16) + MNIST per-config table (note 38). +D. Negative results: hard-k overprediction table (25), derivative probe (31), + scalar gain (14). +E. Experiment details: all configs/seeds (= manifest note 40), hardware, + runtime; reproducibility checklist answers. +F. Teacher test-gap discussion + figure (note 39). + +## 9. Schedule (today = Mon 2026-06-09) + +| week | deliverable | exit criterion | +|---|---|---| +| 06-09..06-15 | wrap-up audit + this plan (**done**); download AuthorKit; `paper/figures.py` regenerating F1-F4 at AAAI specs | `main.tex` compiles with kit; 4 figures render | +| 06-16..06-22 | full body rewrite from ledger (sec 1-6), figures embedded | 7-page draft, every number traced to manifest | +| 06-23..06-29 | related work + `.bib`; appendix A-F; reproducibility checklist; code-release zip | complete submission package v1 | +| 06-30..07-06 | claims audit (read against ledger §9 + rules §7); number audit (against CSVs); style pass; external read if available | v2; no unverified number | +| 07-07..07-13 | buffer for surprises; abstract polish; figure freeze | v3 frozen except typos | +| 07-14..07-20 | abstract registration (**07-20**); final proof pass | submitted abstract | +| 07-21..07-27 | paper submission (**07-27**); supplementary (+3 days) | submitted | + +## 10. Risk register + +| risk | mitigation | +|---|---| +| "MLPs/synthetic only" | MNIST validation in main text (F2/F3); limitations sentence: conv/transformers open | +| "train gap — who cares about generalization?" | F4 sign flip + scoping paragraph: optimization cost is the invariant; test effect task-dependent | +| "estimator is conditional, not a theory" | stated upfront in intro + limitations; closed form supplies the mechanism level; open-problem paragraph | +| novelty vs Song-Xu-Lafferty | complementarity paragraph via `E_B[H_FA]=0` (note 27); we quantify cost, they prove convergence | +| MNIST estimator corr 0.911 looks weak vs 0.999 | report MAE-first (3.9% relative); explain narrow gap range; per-config table in appendix | +| 7-page overflow | appendix offload order: stress-grid table -> e0 Gaussian panels -> teacher figure -> Prop 3 proof sketch | +| AuthorKit/compile friction | week-1 task, not week-4 | +| reviewer asks for per-mode rho_i or architecture-only drift | named open problem with a concrete route (NTK recursion / deep-linear), framed as future work | |