diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/40_reproduction_manifest.md | 13 | ||||
| -rw-r--r-- | notes/41_paper_plan.md | 6 | ||||
| -rw-r--r-- | notes/42_master_technical_reference.md | 6 | ||||
| -rw-r--r-- | notes/43_aaai_accept_bar_experiments.md | 272 | ||||
| -rw-r--r-- | notes/README.md | 6 |
5 files changed, 303 insertions, 0 deletions
diff --git a/notes/40_reproduction_manifest.md b/notes/40_reproduction_manifest.md index 009f058..d7390cb 100644 --- a/notes/40_reproduction_manifest.md +++ b/notes/40_reproduction_manifest.md @@ -54,10 +54,23 @@ deterministic given the seeds shown. Long runs use `nohup`. | kernel code self-test vs autograd (<= 5.3e-15) | `python scripts/real_data_validation.py --self-test` | stdout | 38 | | teacher task: train gap 0.249->0.0069 soft; test gap sign flip (-0.120+/-0.013 at w=32) | `python scripts/downstream_capacity_sweep.py --task mlp --teacher-width 64 --teacher-hidden-layers 2 --normalize-targets --widths 8 12 16 24 32 48 64 96 --train-samples 256 --test-samples 2048 --probe-samples 32 --steps 8000 --lr 0.01 --optimizer sgd --init-seeds 3 --feedback-seeds 8 --data-seed 7 --skip-jacobian --plot --outdir outputs/teacher_test_gap_sgd_T8000` then `python scripts/plot_teacher_test_gap.py` | `teacher_test_gap_sgd_T8000/` | 39 | +## Focused-paper controls added 2026-07-21 + +| claim | command | outputs | note | +|---|---|---|---| +| FA and DFA exact initialization calibration through depth 6: 60 rows, max abs error 0.009164 | `python scripts/aaai_depth_experiments.py --part init --depths 1 2 3 4 6 --width 64 --train-samples 64 --init-seeds 6 --feedback-draws 256 --torch-threads 16 --outdir outputs/aaai_depth_experiments` | `aaai_depth_experiments/initialization_rows.csv` | 43 | +| FA finite-time depth x width grid: matrix-alignment proxy drops 18.06 orders while width-64 gap grows 0.0421 -> 0.3461; frozen MAE 0.06693; two-snapshot MAE 0.01043 / 0.01394 | `python scripts/aaai_depth_experiments.py --part finite --depths 1 2 3 4 6 --finite-widths 32 64 96 --train-samples 64 --finite-init-seeds 4 --finite-feedback-seeds 8 --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 --outdir outputs/aaai_depth_experiments`, then `--part summarize` | `aaai_depth_experiments/finite_time_*`, `mismatch_vs_cost_by_depth.*` | 43 | +| parameter-matched depth control around P=5376: widths 269/64/47/39/31; gap is nonmonotone and peaks at depth 4 | `python scripts/parameter_matched_depth_control.py --depths 1 2 3 4 6 --reference-depth 2 --reference-width 64 --train-samples 64 --init-seeds 4 --feedback-seeds 8 --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 --outdir outputs/parameter_matched_depth_control` | `parameter_matched_depth_control/` | 43 | +| initialization-rate compensation reduces but does not remove the gap (29-51% reduction); separate stability-confirmed tuning preserves a gap | `python scripts/learning_rate_compensation.py --depths 1 2 3 4 6 --width 64 --train-samples 128 --steps 200 --base-lr 0.001 --tune-init-seeds 4 --tune-feedback-seeds 4 --eval-init-seeds 6 --eval-feedback-seeds 8 --torch-threads 16 --outdir outputs/learning_rate_compensation` | `learning_rate_compensation/` | 43 | +| cross-entropy CNN initialization identity for FA and DFA: max abs error 0.008309, max standardized error 1.308 | `python scripts/cnn_initialization_validation.py --train-samples 128 --image-size 8 --channels 8 12 --init-seeds 4 --feedback-draws 2048 --torch-threads 8 --outdir outputs/cnn_initialization_validation` | `cnn_initialization_validation/` | 43 | +| two focused three-panel figures | `python scripts/plot_aaai_main_figures.py --outdir outputs/aaai_main_figures` | `aaai_main_figures/` | 43 | + ## Figure -> file map (current best versions) | paper figure (plan in note 41) | file | |---|---| +| focused Figure 1: mismatch vs exact initial cost | `aaai_main_figures/figure1_mismatch_vs_initial_cost.{png,pdf}` | +| focused Figure 2: later cost and operator evolution | `aaai_main_figures/figure2_training_dynamics.{png,pdf}` | | soft ramp + closed form + ramp law | `closed_form_soft_ramp/closed_form_vs_measured_ramp.png` | | e0 moment calibration (synthetic) | `actual_fa_initial_operator_moments/predicted_vs_empirical_initial_erosion_mean.png` | | e0 Gaussian panels | `actual_fa_initial_erosion_distribution/e0_distribution_theory_vs_empirical.png` | diff --git a/notes/41_paper_plan.md b/notes/41_paper_plan.md index d3bd12f..41f6a05 100644 --- a/notes/41_paper_plan.md +++ b/notes/41_paper_plan.md @@ -1,5 +1,11 @@ # AAAI-27 Paper Plan +> **Superseded framing (2026-07-21).** The original four-figure soft-ramp plan +> below predates the focused matrix-mismatch-versus-optimization-cost review. +> New experiment results and the two replacement composite figures are in +> note 43. Keep this file for venue constraints and historical scheduling, +> not as the current narrative or figure plan. + The single planning document for the writing phase. Inputs: evidence ledger (note 36), reproduction manifest (note 40), corrections (note 37). Supersedes note 03 and the figure list in note 35. diff --git a/notes/42_master_technical_reference.md b/notes/42_master_technical_reference.md index 82c436e..65a2ae5 100644 --- a/notes/42_master_technical_reference.md +++ b/notes/42_master_technical_reference.md @@ -5,6 +5,12 @@ derivations, exact numbers, configs, and negative results. Built for writing the paper by hand: you should not need any other file open. Cross-checked against the raw CSVs on 2026-06-13. +> **2026-07-21 experiment addendum:** note 43 adds the FA/DFA depth-1-to-6 +> calibration, FA finite-time depth/width grid, learning-rate controls, and +> cross-entropy CNN validation. Its results supersede this file's earlier +> statement that DFA and non-MLP validation were absent; the theorem/proof +> material below remains the technical source. + Conventions in this file: - "no fit" = no post-hoc scalar tuned to the reported data; predictions come from initialization or two early kernel snapshots. diff --git a/notes/43_aaai_accept_bar_experiments.md b/notes/43_aaai_accept_bar_experiments.md new file mode 100644 index 0000000..11a981d --- /dev/null +++ b/notes/43_aaai_accept_bar_experiments.md @@ -0,0 +1,272 @@ +# AAAI Accept-Bar Experiments + +Status: **complete, paper-grade local outputs** (2026-07-21). This note +records the experiments added after the focused-paper review. It supplements +the pre-existing master reference (note 42); it does not rewrite the paper. + +## 1. What was added + +Four missing controls are now implemented and run: + +1. FA and DFA initialization-cost calibration at hidden depths `{1,2,3,4,6}`. +2. FA finite-time gap and operator-predictor errors over depth + `{1,2,3,4,6}` and width `{32,64,96}`. +3. Same-rate, exact-initialization-rate-compensated, and independently tuned + BP/FA learning-rate controls. +4. A non-MLP initialization validation: a two-convolution-layer CNN with + cross-entropy loss on an MNIST subset, for both FA and DFA. + +Two three-panel composite figures are generated from these results plus the +existing random-direction-deletion negative control. + +## 2. Shared implementation checks + +New shared implementation: `scripts/feedback_rules.py`. + +- Arbitrary-depth manual BP agrees with autograd to `5.55e-17` max error. +- FA and DFA output-layer gradients agree with BP exactly. +- A 4096-feedback-draw numerical moment test for each rule lands within the + Monte Carlo interval predicted by the exact initialization result. +- The CNN manual BP agrees with autograd to `5.55e-17`; FA and DFA preserve + the exact output gradient. + +Commands: + +```bash +python scripts/test_feedback_rules.py +python scripts/cnn_initialization_validation.py --self-test +``` + +## 3. FA/DFA initialization calibration across depth + +Configuration: + +- synthetic random regression, `N=64`, input dimension 16, output dimension 4; +- fixed width 64; +- hidden depth `{1,2,3,4,6}`; +- 6 fixed forward initializations per depth; +- 256 independent feedback draws per forward initialization and rule; +- real FA/DFA backward pass for every draw (15,360 feedback evaluations). + +The predicted quantity is the relative deficit in BP's first-order loss +decrease: + +```text +1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2. +``` + +Depth-aggregated results: + +| depth | predicted | FA measured | DFA measured | +|---:|---:|---:|---:| +| 1 | 0.050380 | 0.050200 | 0.050578 | +| 2 | 0.276966 | 0.275983 | 0.278078 | +| 3 | 0.402635 | 0.402218 | 0.402282 | +| 4 | 0.458542 | 0.457015 | 0.456779 | +| 6 | 0.623193 | 0.624234 | 0.623791 | + +Across all 60 rule/depth/init rows, max absolute calibration error is +`0.009164`; max standardized error is `2.412` Monte Carlo standard errors. +The same exact prediction covers FA and DFA even though their feedback paths +differ. + +Output: `outputs/aaai_depth_experiments/initialization_rows.csv` and +`initial_deficit_calibration.png`. + +## 4. Finite-time FA cost across depth and width + +Configuration: + +- same synthetic task family, `N=64`, output dimension 4; +- hidden depth `{1,2,3,4,6}`, width `{32,64,96}`; +- 4 forward initializations x 8 FA feedback initializations per cell; +- full-batch SGD, fixed learning rate `1e-3`, horizon `T=50`; +- early operator snapshot `s=20`; +- 480 FA trajectories plus cached matched BP trajectories. + +Curve uncertainty is computed across forward-initialization means (feedback +draws are nested within each initialization), rather than treating all 32 +feedback trajectories in a cell as independent. + +At width 64, the measured mean gap rises with depth: + +| depth | measured | frozen `K(0)` | linear velocity | early retangent | +|---:|---:|---:|---:|---:| +| 1 | 0.042081 | 0.046148 | 0.041747 | 0.043214 | +| 2 | 0.152697 | 0.176001 | 0.150869 | 0.159079 | +| 3 | 0.252548 | 0.301208 | 0.246641 | 0.263831 | +| 4 | 0.281210 | 0.360586 | 0.274093 | 0.302059 | +| 6 | 0.346053 | 0.518519 | 0.312131 | 0.371649 | + +For the same fixed-width architecture, the independent per-block alignment +proxy `prod_l 1/D_l` falls from `3.91e-3` at depth 1 to `3.39e-21` at depth 6, +an 18.06-order-of-magnitude change. Over that sweep, the measured initial +deficit changes from `0.0504` to `0.6240`, and the measured finite-time gap +from `0.0421` to `0.3461`. Thus depth makes the matrix-level mismatch +astronomically worse without producing a commensurate optimization-cost +explosion. The direct comparison is saved in +`mismatch_vs_cost_by_depth.{csv,png}`. + +Across the full 480-row grid: + +| predictor | MAE | relative MAE vs mean measured gap | correlation | +|---|---:|---:|---:| +| frozen `K(0)` | 0.06693 | 28.6% | 0.97166 | +| linear operator velocity | 0.01043 | 4.45% | 0.99492 | +| early retangent | 0.01394 | 5.95% | 0.99830 | + +The frozen operator increasingly underpredicts progress at larger depth. Two +snapshots recover most of the magnitude, supporting the scoped statement that +the later cost depends on operator evolution. The finite-time result remains +FA-only. + +Output: `outputs/aaai_depth_experiments/finite_time_rows.csv`, +`finite_time_metrics.csv`, and the corresponding plots. + +### Parameter-matched supplement + +The fixed-width result is accompanied by an equal-parameter-count control. +Using depth-2 width-64 as the target (`P=5376`, no biases), the closest +uniform widths are `{269,64,47,39,31}` for depths `{1,2,3,4,6}`, with all +parameter counts within 49 of the target. Each cell contains 4 forward +initializations x 8 feedback initializations; error bars again use the four +forward-initialization means. + +| depth | width | parameters | measured gap | +|---:|---:|---:|---:| +| 1 | 269 | 5380 | 0.02695 | +| 2 | 64 | 5376 | 0.15270 | +| 3 | 47 | 5358 | 0.18857 | +| 4 | 39 | 5343 | 0.37599 | +| 6 | 31 | 5425 | 0.21024 | + +The gap is not monotone in depth once parameter count is fixed: it peaks at +depth 4 and falls at depth 6. This is another reason not to infer optimization +cost directly from the number of independently mismatched matrices. Output: +`outputs/parameter_matched_depth_control/`. + +## 5. Learning-rate alternative explanation + +Configuration: + +- fixed width 64, depth `{1,2,3,4,6}`, `N=128`, 200 steps; +- baseline BP/FA learning rate `1e-3`; +- compensated FA rate `1e-3 / rho`, with `rho` read from one BP backward pass; +- independent BP and FA rates selected using 4 tuning initializations x 4 FA + feedback seeds from + `{0.00025,0.0005,0.001,0.002,0.004,0.008,0.016,0.032,0.064,0.128,0.256}`; +- stability-confirmed evaluation on 6 new forward initializations x 8 feedback + initializations; all 720 reported evaluation rows stable. + +Learning-rate curve uncertainty is likewise computed across the six forward +initializations, with eight feedback draws nested within each one. + +| depth | same-rate gap | compensated gap | reduction | independently tuned gap | +|---:|---:|---:|---:|---:| +| 1 | 0.05906 | 0.04053 | 31.4% | 0.56940 | +| 2 | 0.19916 | 0.09729 | 51.2% | 0.64210 | +| 3 | 0.27346 | 0.16263 | 40.5% | 1.13002 | +| 4 | 0.29340 | 0.18442 | 37.1% | 1.30812 | +| 6 | 0.40664 | 0.28850 | 29.1% | 1.75739 | + +Initialization compensation removes a material part of the same-rate gap but +does not remove it at any tested depth. Independent tuning does not rescue +FA: the selected BP rates are `0.064-0.128`, whereas the selected FA rates +fall from `0.064` at depth 1 to stability-confirmed `0.004` at depth 6. BP therefore makes much +more progress in the same 200-step budget. This result should be described as +a fixed-horizon optimization comparison, not a claim about asymptotic +convergence. + +Output: `outputs/learning_rate_compensation/`. + +## 6. Cross-entropy CNN initialization validation + +Configuration: + +- MNIST subset, 128 examples resized to `8x8`; +- `Conv(1,8,3x3) -> ReLU -> Conv(8,12,3x3) -> ReLU -> global average pool -> linear`; +- cross-entropy loss; +- 4 forward initializations; +- 2048 real-backward feedback draws per initialization and rule (FA and DFA). + +The predicted deficit spans `0.592-0.816`. Across the eight rows, max +absolute calibration error is `0.008309`, and max standardized error is only +`1.308` Monte Carlo standard errors. This validates the initialization +identity outside squared-loss MLPs without making a finite-time CNN claim. + +Output: `outputs/cnn_initialization_validation/`. + +## 7. Figures + +- `outputs/aaai_main_figures/figure1_mismatch_vs_initial_cost.{png,pdf}`: + BP/FA/DFA schematic; minimax mismatch plus the 250x null-model + overprediction; joint initialization calibration across rule, depth, MNIST + MLP, and cross-entropy CNN. +- `outputs/aaai_main_figures/figure2_training_dynamics.{png,pdf}`: + measured finite-time cost over depth/width; frozen-operator calibration; + two-snapshot calibration. +- Supplemental learning-rate figures: + `outputs/learning_rate_compensation/learning_rate_control_by_depth.png` and + `learning_rate_tuning_curves.png`. +- Direct depth contrast: + `outputs/aaai_depth_experiments/mismatch_vs_cost_by_depth.{csv,png}`. +- Parameter-matched supplement: + `outputs/parameter_matched_depth_control/parameter_matched_gap_by_depth.png`. + +## 8. Claim boundary after these experiments + +Supported: + +- The exact expected initialization result applies to both FA and DFA under + independent zero-mean feedback, and is validated through depth 6. +- It also holds for a cross-entropy CNN in the tested convolutional feedback + implementation. +- Simple initial-rate compensation reduces but does not eliminate the + finite-time FA/BP gap in the tested MLP setting. +- FA's later gap and frozen-model error grow with depth; two early operator + snapshots remain predictive. + +Not supported / not claimed: + +- DFA finite-time dynamics. +- CNN finite-time training equivalence or modern large-scale scaling. +- A universal absence of width/depth thresholds. +- A per-eigenmode erosion theorem. + +## 9. Reproduction commands + +```bash +python scripts/aaai_depth_experiments.py --part init \ + --depths 1 2 3 4 6 --width 64 --train-samples 64 \ + --init-seeds 6 --feedback-draws 256 --torch-threads 16 \ + --outdir outputs/aaai_depth_experiments + +python scripts/aaai_depth_experiments.py --part finite \ + --depths 1 2 3 4 6 --finite-widths 32 64 96 --train-samples 64 \ + --finite-init-seeds 4 --finite-feedback-seeds 8 \ + --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 \ + --outdir outputs/aaai_depth_experiments + +python scripts/aaai_depth_experiments.py --part summarize \ + --outdir outputs/aaai_depth_experiments + +python scripts/parameter_matched_depth_control.py \ + --depths 1 2 3 4 6 --reference-depth 2 --reference-width 64 \ + --train-samples 64 --init-seeds 4 --feedback-seeds 8 \ + --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 \ + --outdir outputs/parameter_matched_depth_control + +python scripts/learning_rate_compensation.py \ + --depths 1 2 3 4 6 --width 64 --train-samples 128 --steps 200 \ + --base-lr 0.001 --tune-init-seeds 4 --tune-feedback-seeds 4 \ + --eval-init-seeds 6 --eval-feedback-seeds 8 --torch-threads 16 \ + --outdir outputs/learning_rate_compensation + +python scripts/cnn_initialization_validation.py \ + --train-samples 128 --image-size 8 --channels 8 12 \ + --init-seeds 4 --feedback-draws 2048 --torch-threads 8 \ + --outdir outputs/cnn_initialization_validation + +python scripts/plot_aaai_main_figures.py \ + --outdir outputs/aaai_main_figures +``` diff --git a/notes/README.md b/notes/README.md index 853c122..021b966 100644 --- a/notes/README.md +++ b/notes/README.md @@ -7,6 +7,7 @@ capacity & operator-gap analysis)** — target AAAI-27. | doc | role | |---|---| +| [43_aaai_accept_bar_experiments.md](43_aaai_accept_bar_experiments.md) | **new focused-paper experiments**: FA/DFA depth calibration, finite-time depth grid, learning-rate controls, CE CNN validation, and two composite figures | | [42_master_technical_reference.md](42_master_technical_reference.md) | **write the paper from this**: every theorem with full statement + proof, every result with exact numbers/configs, derivations, negative results, honesty boundaries -- self-contained | | [41_paper_plan.md](41_paper_plan.md) | **the plan**: venue rules, narrative, section/page budget, theorem & figure numbering, related-work list, schedule, risks | | [36_evidence_ledger.md](36_evidence_ledger.md) | **the numbers, condensed**: every claim with its corrected headline figure, claim discipline, booby traps | @@ -73,6 +74,11 @@ Phase 6 — wrap-up for AAAI-27 (2026-06-09..13): test gap**; 40 reproduction manifest; 41 paper plan; 42 **master technical reference** (full proofs + numbers, the write-from doc). +Phase 7 — focused accept-bar controls (2026-07-21): +- 43 **FA/DFA depth calibration, FA finite-time depth/width grid, learning-rate + compensation and independent tuning, cross-entropy CNN initialization + validation, and focused Figures 1-2**. + ## Note conventions - Prefer formulas and precise claims over prose-only ideas. |