diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/36_evidence_ledger.md | 9 | ||||
| -rw-r--r-- | notes/39_teacher_test_gap.md | 53 | ||||
| -rw-r--r-- | notes/README.md | 4 |
3 files changed, 63 insertions, 3 deletions
diff --git a/notes/36_evidence_ledger.md b/notes/36_evidence_ledger.md index 14af7df..ed952bf 100644 --- a/notes/36_evidence_ledger.md +++ b/notes/36_evidence_ledger.md @@ -146,9 +146,12 @@ what they are; (31) and (25) deserve explicit mention in the paper. - MNIST e0 + estimator: `outputs/real_data_validation_mnist/` (script `scripts/real_data_validation.py`, self-test verifies kernels vs autograd at 1e-15). -- Teacher-task test gap (SGD, widths 8..96): `outputs/teacher_test_gap_sgd_T8000/` - — addresses "what about generalization": train-gap story plus a descriptive - test-MSE companion. Random-label sweeps measure optimization gap by design. +- Teacher-task test gap (note 39, SGD, widths 8..96, 24 FA runs/width): + train gap soft and monotone (0.249 -> 0.0069) while the **test gap changes + sign** — +0.177 (w=8), **-0.120 +/- 0.013 (w=32, FA generalizes better)**, + +0.045 (w=96). The optimization gap is the invariant cost; its test-side + consequence is task-dependent — exactly why the theory is built on the + former. `outputs/teacher_test_gap_sgd_T8000/teacher_train_vs_test_gap.png`. ## 6. Figure plan (sources verified on disk) diff --git a/notes/39_teacher_test_gap.md b/notes/39_teacher_test_gap.md new file mode 100644 index 0000000..80df10a --- /dev/null +++ b/notes/39_teacher_test_gap.md @@ -0,0 +1,53 @@ +# Teacher-Task Test Gap (Generalization Companion) + +Run: `outputs/teacher_test_gap_sgd_T8000/` via +`scripts/downstream_capacity_sweep.py --task mlp` (MLP teacher width 64, +2 hidden layers, normalized targets, N=256 train / 2048 test, SGD lr 0.01, +T=8000, widths {8..96}, 3 inits x 8 feedback seeds = 24 FA runs + 3 BP runs +per width, data_seed 7). Figure: +`teacher_train_vs_test_gap.png` (`scripts/plot_teacher_test_gap.py`). +Purpose: answer the inevitable "you only study train loss — what about +generalization?" with data instead of a deflection. + +## Result + +| width | train gap | test gap (mean +/- stderr, n=24) | +|---|---|---| +| 8 | 0.249 | **+0.177 +/- 0.020** | +| 12 | 0.158 | +0.038 +/- 0.016 | +| 16 | 0.112 | -0.038 +/- 0.015 | +| 24 | 0.080 | -0.044 +/- 0.018 | +| 32 | 0.085 | **-0.120 +/- 0.013** | +| 48 | 0.048 | -0.014 +/- 0.014 | +| 64 | 0.026 | +0.010 +/- 0.008 | +| 96 | 0.0069 | **+0.045 +/- 0.011** | + +- **Train (optimization) gap**: positive and monotone-decreasing, the same + soft ramp as the random-label sweeps — the irreducible operator cost. +- **Test gap**: task-dependent and **changes sign**. Under-capacity FA pays + on both sides (+0.177 at w=8); at intermediate width FA generalizes + *better* than BP (significantly: -0.120 +/- 0.013 at w=32) — random + feedback's slower, misaligned updates act as an implicit regularizer while + BP interpolates a mismatched narrow student; once both fit (w >= 64) the + test gap settles to small positive values. + +## Reading for the paper + +The paper's object — the optimization gap — is the *invariant* cost: always +positive, soft, exactly priced at init (T5), quantified over training (T7 + +estimator). The test-side consequence of that cost is genuinely +task-dependent and non-monotone, which is precisely why we do not build the +theory on it. One figure + two sentences in the experiments or limitations +section: report the sign flip honestly, cite the implicit-regularization +reading, and scope the claims to optimization. This also preempts "FA is +just worse" misreadings: at moderate width it can win on test while always +losing on train speed. + +## Caveats + +- Single task family (MLP teacher, noiseless, normalized targets), one + train-set size; the sign-flip location moves with task/teacher details. +- T=8000 SGD: BP train at w=96 is 1.1e-3 (essentially fit); FA 7.9e-3 — + the train-gap column is finite-time, consistent with the soft-ramp story. +- Adam-era teacher runs (notes 04-06 outputs) are superseded by this SGD run + for any paper use. diff --git a/notes/README.md b/notes/README.md index 158a8c5..76d8e46 100644 --- a/notes/README.md +++ b/notes/README.md @@ -10,6 +10,10 @@ Use this directory as the project memory. Keep rough ideas here before promoting - [36_evidence_ledger.md](36_evidence_ledger.md): **the writing base** — full evidence chain, corrected headline numbers, figure sources, claim discipline. +- [38_mnist_real_data_validation.md](38_mnist_real_data_validation.md): e0 + theorem + estimator replicated on MNIST (no fit). +- [39_teacher_test_gap.md](39_teacher_test_gap.md): teacher-task test gap — + optimization cost invariant, test effect task-dependent (sign flip). - [37_wrapup_corrections.md](37_wrapup_corrections.md): pre-writing audit fixes (e0 sampler now real FA backward; closed-form comparison now data/init-matched — corr 0.977 retired, 0.933 matched / 0.982 ensemble). |