Record T5 local tangent validation

1 files changed, 127 insertions, 0 deletions

diff --git a/notes/10_t5_local_validation.md b/notes/10_t5_local_validation.md
new file mode 100644
index 0000000..76713dd
--- /dev/null
+++ b/notes/10_t5_local_validation.md
@@ -0,0 +1,127 @@
+# T=5 Local Tangent-Kernel Validation
+
+The earlier short-SGD figure still had theory uniformly above empirical gaps.
+That was not fixed by only switching to a fixed-width N-sweep. The remaining
+issue was that \(T=50\) is too long for a fixed-initialization tangent-kernel
+prediction.
+
+## Horizon Diagnostic
+
+Using the same width/N/seeds and varying only the number of SGD steps:
+
+| steps | empirical gap | predicted gap | empirical / predicted | MAE |
+|---:|---:|---:|---:|---:|
+| 1 | 0.030734 | 0.029853 | 1.028 | 0.000881 |
+| 5 | 0.112912 | 0.113947 | 0.987 | 0.001712 |
+| 10 | 0.156304 | 0.164964 | 0.942 | 0.008660 |
+| 20 | 0.163335 | 0.183836 | 0.884 | 0.020500 |
+| 50 | 0.125547 | 0.154075 | 0.814 | 0.028528 |
+
+So the uniform-high theory issue is a finite-horizon error: fixed
+initialization kernels match \(T=1\) and \(T=5\), then drift out of regime.
+
+This is not a scaling bug:
+
+- discrete Euler vs continuous exponential has negligible effect;
+- the \(1/N\) factor is consistent with the implemented full-batch MSE
+  gradient;
+- the mismatch grows with training horizon.
+
+## Main Local Validation
+
+Use a fixed-width N-sweep:
+
+- architecture \(16\to64\to64\to4\);
+- random-label regression;
+- \(N\in\{64,96,128,160,192,224,256,320\}\);
+- SGD, learning rate \(10^{-3}\);
+- steps \(T=5\);
+- 4 init seeds and 8 feedback seeds per init;
+- total FA trajectories \(8\times4\times8=256\).
+
+Theory uses the fixed initialization kernels:
+
+\[
+K_{\mathrm{BP}}=J_{\mathrm{BP}}J_{\mathrm{BP}}^\top,
+\qquad
+K_{\mathrm{FA}}=J_{\mathrm{BP}}\tilde J_{\mathrm{FA}}^\top.
+\]
+
+Prediction:
+
+\[
+r_T^{\mathrm{lin}}
+=
+\exp\left(-\frac{\eta T}{N}K\right)r_0,
+\qquad
+L_T^{\mathrm{lin}}
+=
+\frac{1}{2N}\|r_T^{\mathrm{lin}}\|^2.
+\]
+
+No fitted multiplicative scale or offset is used.
+
+## Results
+
+Combined figure:
+
+`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_theory_empirical_overlay.png`
+
+Scatter:
+
+`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_predicted_vs_empirical.png`
+
+Across 256 FA trajectories:
+
+\[
+\operatorname{corr}
+\left(
+\Delta L_T^{\mathrm{pred}},
+\Delta L_T^{\mathrm{emp}}
+\right)
+=
+0.99947.
+\]
+
+\[
+\operatorname{MAE}=0.00083,
+\qquad
+\operatorname{RMSE}=0.00124.
+\]
+
+Mean gap:
+
+\[
+\mathbb E[\Delta L_T^{\mathrm{emp}}]=0.05953,
+\qquad
+\mathbb E[\Delta L_T^{\mathrm{pred}}]=0.06003.
+\]
+
+Ratio:
+
+\[
+\mathbb E[\Delta L_T^{\mathrm{emp}}/\Delta L_T^{\mathrm{pred}}]
+=
+0.99286.
+\]
+
+Linear fit:
+
+\[
+\Delta L_T^{\mathrm{emp}}
+\approx
+0.99496\,\Delta L_T^{\mathrm{pred}}
+-0.00019.
+\]
+
+This resolves the uniform-high issue for the local theory.
+
+## Interpretation
+
+The fixed-initialization tangent-kernel theory is a local-time theorem. It
+should be validated with \(T=O(1)\) small enough that the network remains in the
+initial tangent regime.
+
+The long-horizon memorization transition is a different contribution. It cannot
+be validated by the same fixed-kernel formula unless we add a time-varying
+kernel or explicit alignment-dynamics theory.