Add teacher-task test-gap sweep and figure

SGD teacher sweep (widths 8-96, 24 FA runs/width): train gap soft and
monotone (0.249 -> 0.0069) while the test gap changes sign — FA
generalizes better than BP at intermediate width (-0.120 +/- 0.013 at
w=32) and settles slightly positive once both fit. Scopes the paper's
optimization-gap claims and preempts the generalization question.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

1 files changed, 53 insertions, 0 deletions

diff --git a/notes/39_teacher_test_gap.md b/notes/39_teacher_test_gap.md
new file mode 100644
index 0000000..80df10a
--- /dev/null
+++ b/notes/39_teacher_test_gap.md
@@ -0,0 +1,53 @@
+# Teacher-Task Test Gap (Generalization Companion)
+
+Run: `outputs/teacher_test_gap_sgd_T8000/` via
+`scripts/downstream_capacity_sweep.py --task mlp` (MLP teacher width 64,
+2 hidden layers, normalized targets, N=256 train / 2048 test, SGD lr 0.01,
+T=8000, widths {8..96}, 3 inits x 8 feedback seeds = 24 FA runs + 3 BP runs
+per width, data_seed 7). Figure:
+`teacher_train_vs_test_gap.png` (`scripts/plot_teacher_test_gap.py`).
+Purpose: answer the inevitable "you only study train loss — what about
+generalization?" with data instead of a deflection.
+
+## Result
+
+| width | train gap | test gap (mean +/- stderr, n=24) |
+|---|---|---|
+| 8 | 0.249 | **+0.177 +/- 0.020** |
+| 12 | 0.158 | +0.038 +/- 0.016 |
+| 16 | 0.112 | -0.038 +/- 0.015 |
+| 24 | 0.080 | -0.044 +/- 0.018 |
+| 32 | 0.085 | **-0.120 +/- 0.013** |
+| 48 | 0.048 | -0.014 +/- 0.014 |
+| 64 | 0.026 | +0.010 +/- 0.008 |
+| 96 | 0.0069 | **+0.045 +/- 0.011** |
+
+- **Train (optimization) gap**: positive and monotone-decreasing, the same
+  soft ramp as the random-label sweeps — the irreducible operator cost.
+- **Test gap**: task-dependent and **changes sign**. Under-capacity FA pays
+  on both sides (+0.177 at w=8); at intermediate width FA generalizes
+  *better* than BP (significantly: -0.120 +/- 0.013 at w=32) — random
+  feedback's slower, misaligned updates act as an implicit regularizer while
+  BP interpolates a mismatched narrow student; once both fit (w >= 64) the
+  test gap settles to small positive values.
+
+## Reading for the paper
+
+The paper's object — the optimization gap — is the *invariant* cost: always
+positive, soft, exactly priced at init (T5), quantified over training (T7 +
+estimator). The test-side consequence of that cost is genuinely
+task-dependent and non-monotone, which is precisely why we do not build the
+theory on it. One figure + two sentences in the experiments or limitations
+section: report the sign flip honestly, cite the implicit-regularization
+reading, and scope the claims to optimization. This also preempts "FA is
+just worse" misreadings: at moderate width it can win on test while always
+losing on train speed.
+
+## Caveats
+
+- Single task family (MLP teacher, noiseless, normalized targets), one
+  train-set size; the sign-flip location moves with task/teacher details.
+- T=8000 SGD: BP train at w=96 is 1.1e-3 (essentially fit); FA 7.9e-3 —
+  the train-gap column is finite-time, consistent with the soft-ramp story.
+- Adam-era teacher runs (notes 04-06 outputs) are superseded by this SGD run
+  for any paper use.