Record fixed-width capacity probe

1 files changed, 113 insertions, 0 deletions

diff --git a/notes/09_fixed_width_probe.md b/notes/09_fixed_width_probe.md
new file mode 100644
index 0000000..9227843
--- /dev/null
+++ b/notes/09_fixed_width_probe.md
@@ -0,0 +1,113 @@
+# Fixed-Width N-Sweep Probe
+
+The previous width-sweep short-SGD plot showed that tangent-kernel predictions
+were systematically above empirical FA/BP train gaps. A decomposition showed
+the main error source was not FA:
+
+\[
+\mathbb E[L_{\mathrm{FA}}^{\mathrm{pred}}-L_{\mathrm{FA}}^{\mathrm{emp}}]
+\approx
+0.008,
+\]
+
+while fixed-kernel BP was too optimistic:
+
+\[
+\mathbb E[L_{\mathrm{BP}}^{\mathrm{pred}}-L_{\mathrm{BP}}^{\mathrm{emp}}]
+\approx
+-0.048.
+\]
+
+So the predicted gap was high mainly because the BP baseline was predicted too
+low.
+
+## Revised Design
+
+Do not shrink capacity by shrinking width first. That changes two things at
+once:
+
+1. the FA capacity margin;
+2. the finite-width quality of the BP tangent-kernel approximation.
+
+Instead, keep the MLP width fixed and vary the number of training samples:
+
+\[
+P-K_{\mathrm{FA}}-Nn_L.
+\]
+
+This changes the capacity margin while keeping finite-width BP bias roughly
+controlled.
+
+Probe settings:
+
+- width \(n=64\);
+- random-label regression;
+- \(N\in\{64,128,192,256,320\}\);
+- output dimension \(4\);
+- SGD, learning rate \(10^{-3}\), steps \(50\);
+- 2 init seeds and 4 feedback seeds per init.
+
+## Result
+
+Combined output:
+
+`outputs/fa_tangent_kernel_fixed_width64_sgd_short_combined/fixed_width64_N_sweep_theory_empirical_overlay.png`
+
+The fixed-width probe improves the calibration:
+
+\[
+\operatorname{corr}
+\left(
+\Delta L^{\mathrm{pred}},
+\Delta L^{\mathrm{emp}}
+\right)
+=
+0.983.
+\]
+
+\[
+\operatorname{MAE}=0.0244.
+\]
+
+The linear fit becomes
+
+\[
+\Delta L^{\mathrm{emp}}
+\approx
+0.890\,\Delta L^{\mathrm{pred}}-0.008.
+\]
+
+This is much cleaner than the width-sweep fit:
+
+\[
+\Delta L^{\mathrm{emp}}
+\approx
+0.546\,\Delta L^{\mathrm{pred}}+0.044.
+\]
+
+The remaining bias is again mostly BP fixed-kernel optimism:
+
+\[
+\mathbb E[L_{\mathrm{BP}}^{\mathrm{pred}}-L_{\mathrm{BP}}^{\mathrm{emp}}]
+\approx
+-0.020,
+\]
+
+with a small FA prediction bias:
+
+\[
+\mathbb E[L_{\mathrm{FA}}^{\mathrm{pred}}-L_{\mathrm{FA}}^{\mathrm{emp}}]
+\approx
+0.0045.
+\]
+
+## Interpretation
+
+The earlier plot was not the cleanest validation of the theory because the
+width axis mixed capacity shrinkage with finite-width NTK error. The fixed-width
+N-sweep is the cleaner short-horizon validation design.
+
+This does not yet solve the long-horizon memorization transition. Short-SGD
+validates the local tangent-kernel distribution; long-horizon capacity
+transition still needs either a time-varying kernel theory or a separate
+alignment-dynamics theorem.