1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
# Teacher-Task Test Gap (Generalization Companion)
Run: `outputs/teacher_test_gap_sgd_T8000/` via
`scripts/downstream_capacity_sweep.py --task mlp` (MLP teacher width 64,
2 hidden layers, normalized targets, N=256 train / 2048 test, SGD lr 0.01,
T=8000, widths {8..96}, 3 inits x 8 feedback seeds = 24 FA runs + 3 BP runs
per width, data_seed 7). Figure:
`teacher_train_vs_test_gap.png` (`scripts/plot_teacher_test_gap.py`).
Purpose: answer the inevitable "you only study train loss — what about
generalization?" with data instead of a deflection.
## Result
| width | train gap | test gap (mean +/- stderr, n=24) |
|---|---|---|
| 8 | 0.249 | **+0.177 +/- 0.020** |
| 12 | 0.158 | +0.038 +/- 0.016 |
| 16 | 0.112 | -0.038 +/- 0.015 |
| 24 | 0.080 | -0.044 +/- 0.018 |
| 32 | 0.085 | **-0.120 +/- 0.013** |
| 48 | 0.048 | -0.014 +/- 0.014 |
| 64 | 0.026 | +0.010 +/- 0.008 |
| 96 | 0.0069 | **+0.045 +/- 0.011** |
- **Train (optimization) gap**: positive and monotone-decreasing, the same
soft ramp as the random-label sweeps — the irreducible operator cost.
- **Test gap**: task-dependent and **changes sign**. Under-capacity FA pays
on both sides (+0.177 at w=8); at intermediate width FA generalizes
*better* than BP (significantly: -0.120 +/- 0.013 at w=32) — random
feedback's slower, misaligned updates act as an implicit regularizer while
BP interpolates a mismatched narrow student; once both fit (w >= 64) the
test gap settles to small positive values.
## Reading for the paper
The paper's object — the optimization gap — is the *invariant* cost: always
positive, soft, exactly priced at init (T5), quantified over training (T7 +
estimator). The test-side consequence of that cost is genuinely
task-dependent and non-monotone, which is precisely why we do not build the
theory on it. One figure + two sentences in the experiments or limitations
section: report the sign flip honestly, cite the implicit-regularization
reading, and scope the claims to optimization. This also preempts "FA is
just worse" misreadings: at moderate width it can win on test while always
losing on train speed.
## Caveats
- Single task family (MLP teacher, noiseless, normalized targets), one
train-set size; the sign-flip location moves with task/teacher details.
- T=8000 SGD: BP train at w=96 is 1.1e-3 (essentially fit); FA 7.9e-3 —
the train-gap column is finite-time, consistent with the soft-ramp story.
- Adam-era teacher runs (notes 04-06 outputs) are superseded by this SGD run
for any paper use.
|
