Add teacher-task test-gap sweep and figure

SGD teacher sweep (widths 8-96, 24 FA runs/width): train gap soft and
monotone (0.249 -> 0.0069) while the test gap changes sign — FA
generalizes better than BP at intermediate width (-0.120 +/- 0.013 at
w=32) and settles slightly positive once both fit. Scopes the paper's
optimization-gap claims and preempts the generalization question.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

4 files changed, 137 insertions, 3 deletions

diff --git a/notes/36_evidence_ledger.md b/notes/36_evidence_ledger.md
index 14af7df..ed952bf 100644
--- a/notes/36_evidence_ledger.md
+++ b/notes/36_evidence_ledger.md
@@ -146,9 +146,12 @@ what they are; (31) and (25) deserve explicit mention in the paper.
 - MNIST e0 + estimator: `outputs/real_data_validation_mnist/` (script
   `scripts/real_data_validation.py`, self-test verifies kernels vs autograd
   at 1e-15).
-- Teacher-task test gap (SGD, widths 8..96): `outputs/teacher_test_gap_sgd_T8000/`
-  — addresses "what about generalization": train-gap story plus a descriptive
-  test-MSE companion. Random-label sweeps measure optimization gap by design.
+- Teacher-task test gap (note 39, SGD, widths 8..96, 24 FA runs/width):
+  train gap soft and monotone (0.249 -> 0.0069) while the **test gap changes
+  sign** — +0.177 (w=8), **-0.120 +/- 0.013 (w=32, FA generalizes better)**,
+  +0.045 (w=96). The optimization gap is the invariant cost; its test-side
+  consequence is task-dependent — exactly why the theory is built on the
+  former. `outputs/teacher_test_gap_sgd_T8000/teacher_train_vs_test_gap.png`.
 
 ## 6. Figure plan (sources verified on disk)
 
diff --git a/notes/39_teacher_test_gap.md b/notes/39_teacher_test_gap.md
new file mode 100644
index 0000000..80df10a
--- /dev/null
+++ b/notes/39_teacher_test_gap.md
@@ -0,0 +1,53 @@
+# Teacher-Task Test Gap (Generalization Companion)
+
+Run: `outputs/teacher_test_gap_sgd_T8000/` via
+`scripts/downstream_capacity_sweep.py --task mlp` (MLP teacher width 64,
+2 hidden layers, normalized targets, N=256 train / 2048 test, SGD lr 0.01,
+T=8000, widths {8..96}, 3 inits x 8 feedback seeds = 24 FA runs + 3 BP runs
+per width, data_seed 7). Figure:
+`teacher_train_vs_test_gap.png` (`scripts/plot_teacher_test_gap.py`).
+Purpose: answer the inevitable "you only study train loss — what about
+generalization?" with data instead of a deflection.
+
+## Result
+
+| width | train gap | test gap (mean +/- stderr, n=24) |
+|---|---|---|
+| 8 | 0.249 | **+0.177 +/- 0.020** |
+| 12 | 0.158 | +0.038 +/- 0.016 |
+| 16 | 0.112 | -0.038 +/- 0.015 |
+| 24 | 0.080 | -0.044 +/- 0.018 |
+| 32 | 0.085 | **-0.120 +/- 0.013** |
+| 48 | 0.048 | -0.014 +/- 0.014 |
+| 64 | 0.026 | +0.010 +/- 0.008 |
+| 96 | 0.0069 | **+0.045 +/- 0.011** |
+
+- **Train (optimization) gap**: positive and monotone-decreasing, the same
+  soft ramp as the random-label sweeps — the irreducible operator cost.
+- **Test gap**: task-dependent and **changes sign**. Under-capacity FA pays
+  on both sides (+0.177 at w=8); at intermediate width FA generalizes
+  *better* than BP (significantly: -0.120 +/- 0.013 at w=32) — random
+  feedback's slower, misaligned updates act as an implicit regularizer while
+  BP interpolates a mismatched narrow student; once both fit (w >= 64) the
+  test gap settles to small positive values.
+
+## Reading for the paper
+
+The paper's object — the optimization gap — is the *invariant* cost: always
+positive, soft, exactly priced at init (T5), quantified over training (T7 +
+estimator). The test-side consequence of that cost is genuinely
+task-dependent and non-monotone, which is precisely why we do not build the
+theory on it. One figure + two sentences in the experiments or limitations
+section: report the sign flip honestly, cite the implicit-regularization
+reading, and scope the claims to optimization. This also preempts "FA is
+just worse" misreadings: at moderate width it can win on test while always
+losing on train speed.
+
+## Caveats
+
+- Single task family (MLP teacher, noiseless, normalized targets), one
+  train-set size; the sign-flip location moves with task/teacher details.
+- T=8000 SGD: BP train at w=96 is 1.1e-3 (essentially fit); FA 7.9e-3 —
+  the train-gap column is finite-time, consistent with the soft-ramp story.
+- Adam-era teacher runs (notes 04-06 outputs) are superseded by this SGD run
+  for any paper use.
diff --git a/notes/README.md b/notes/README.md
index 158a8c5..76d8e46 100644
--- a/notes/README.md
+++ b/notes/README.md
@@ -10,6 +10,10 @@ Use this directory as the project memory. Keep rough ideas here before promoting
 
 - [36_evidence_ledger.md](36_evidence_ledger.md): **the writing base** — full
   evidence chain, corrected headline numbers, figure sources, claim discipline.
+- [38_mnist_real_data_validation.md](38_mnist_real_data_validation.md): e0
+  theorem + estimator replicated on MNIST (no fit).
+- [39_teacher_test_gap.md](39_teacher_test_gap.md): teacher-task test gap —
+  optimization cost invariant, test effect task-dependent (sign flip).
 - [37_wrapup_corrections.md](37_wrapup_corrections.md): pre-writing audit
   fixes (e0 sampler now real FA backward; closed-form comparison now
   data/init-matched — corr 0.977 retired, 0.933 matched / 0.982 ensemble).
diff --git a/scripts/plot_teacher_test_gap.py b/scripts/plot_teacher_test_gap.py
new file mode 100644
index 0000000..8c4e0de
--- /dev/null
+++ b/scripts/plot_teacher_test_gap.py
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""Two-panel teacher-task figure: train gap (soft ramp) vs test gap (sign flip).
+
+Reads width_summary.csv from a teacher-task sweep produced by
+downstream_capacity_sweep.py and plots, side by side:
+  left  - FA/BP train gap vs width (log y): the optimization cost, soft ramp;
+  right - FA/BP test gap vs width (linear y, error bars): task-dependent,
+          crosses zero (FA can generalize better at intermediate width).
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+
+
+def parse_args() -> argparse.Namespace:
+    p = argparse.ArgumentParser(description="Teacher-task train vs test gap figure.")
+    p.add_argument("--run-dir", type=Path,
+                   default=Path("outputs/teacher_test_gap_sgd_T8000"))
+    return p.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    rows = []
+    with (args.run_dir / "width_summary.csv").open() as fh:
+        for row in csv.DictReader(fh):
+            rows.append({k: float(v) for k, v in row.items()})
+    rows.sort(key=lambda r: r["width"])
+
+    widths = np.array([r["width"] for r in rows])
+    n_fa = np.array([r["runs_fa"] for r in rows])
+    train_gap = np.array([r["train_gap_mean"] for r in rows])
+    train_se = np.array([r["train_gap_std"] for r in rows]) / np.sqrt(n_fa)
+    test_gap = np.array([r["gap_mean"] for r in rows])
+    test_se = np.array([r["gap_std"] for r in rows]) / np.sqrt(n_fa)
+
+    fig, axes = plt.subplots(1, 2, figsize=(11.2, 4.4), dpi=180)
+    axes[0].errorbar(widths, train_gap, yerr=train_se, fmt="o-", color="#c65f16",
+                     capsize=2.5)
+    axes[0].set_xscale("log", base=2)
+    axes[0].set_yscale("log")
+    axes[0].set_xlabel("hidden width")
+    axes[0].set_ylabel("FA train MSE - BP train MSE")
+    axes[0].set_title("Optimization gap: soft, monotone, never zero")
+    axes[0].grid(alpha=0.18, which="both")
+
+    axes[1].errorbar(widths, test_gap, yerr=test_se, fmt="o-", color="#174f91",
+                     capsize=2.5)
+    axes[1].axhline(0.0, color="black", linewidth=1)
+    axes[1].set_xscale("log", base=2)
+    axes[1].set_xlabel("hidden width")
+    axes[1].set_ylabel("FA test MSE - BP test MSE")
+    axes[1].set_title("Generalization gap: task-dependent, changes sign")
+    axes[1].grid(alpha=0.18)
+
+    fig.suptitle("Teacher task (MLP teacher, SGD, T=8000): optimization cost vs test effect",
+                 y=1.02)
+    fig.tight_layout()
+    out = args.run_dir / "teacher_train_vs_test_gap.png"
+    fig.savefig(out, bbox_inches="tight")
+    print(f"plot: {out}")
+
+
+if __name__ == "__main__":
+    main()