# T=5 Local Tangent-Kernel Validation

The earlier short-SGD figure still had theory uniformly above empirical gaps.
That was not fixed by only switching to a fixed-width N-sweep. The remaining
issue was that \(T=50\) is too long for a fixed-initialization tangent-kernel
prediction.

## Horizon Diagnostic

Using the same width/N/seeds and varying only the number of SGD steps:

| steps | empirical gap | predicted gap | empirical / predicted | MAE |
|---:|---:|---:|---:|---:|
| 1 | 0.030734 | 0.029853 | 1.028 | 0.000881 |
| 5 | 0.112912 | 0.113947 | 0.987 | 0.001712 |
| 10 | 0.156304 | 0.164964 | 0.942 | 0.008660 |
| 20 | 0.163335 | 0.183836 | 0.884 | 0.020500 |
| 50 | 0.125547 | 0.154075 | 0.814 | 0.028528 |

So the uniform-high theory issue is a finite-horizon error: fixed
initialization kernels match \(T=1\) and \(T=5\), then drift out of regime.

This is not a scaling bug:

- discrete Euler vs continuous exponential has negligible effect;
- the \(1/N\) factor is consistent with the implemented full-batch MSE
  gradient;
- the mismatch grows with training horizon.

## Main Local Validation

Use a fixed-width N-sweep:

- architecture \(16\to64\to64\to4\);
- random-label regression;
- \(N\in\{64,96,128,160,192,224,256,320\}\);
- SGD, learning rate \(10^{-3}\);
- steps \(T=5\);
- 4 init seeds and 8 feedback seeds per init;
- total FA trajectories \(8\times4\times8=256\).

Theory uses the fixed initialization kernels:

\[
K_{\mathrm{BP}}=J_{\mathrm{BP}}J_{\mathrm{BP}}^\top,
\qquad
K_{\mathrm{FA}}=J_{\mathrm{BP}}\tilde J_{\mathrm{FA}}^\top.
\]

Prediction:

\[
r_T^{\mathrm{lin}}
=
\exp\left(-\frac{\eta T}{N}K\right)r_0,
\qquad
L_T^{\mathrm{lin}}
=
\frac{1}{2N}\|r_T^{\mathrm{lin}}\|^2.
\]

No fitted multiplicative scale or offset is used.

## Results

Combined figure:

`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_theory_empirical_overlay.png`

Scatter:

`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_predicted_vs_empirical.png`

Across 256 FA trajectories:

\[
\operatorname{corr}
\left(
\Delta L_T^{\mathrm{pred}},
\Delta L_T^{\mathrm{emp}}
\right)
=
0.99947.
\]

\[
\operatorname{MAE}=0.00083,
\qquad
\operatorname{RMSE}=0.00124.
\]

Mean gap:

\[
\mathbb E[\Delta L_T^{\mathrm{emp}}]=0.05953,
\qquad
\mathbb E[\Delta L_T^{\mathrm{pred}}]=0.06003.
\]

Ratio:

\[
\mathbb E[\Delta L_T^{\mathrm{emp}}/\Delta L_T^{\mathrm{pred}}]
=
0.99286.
\]

Linear fit:

\[
\Delta L_T^{\mathrm{emp}}
\approx
0.99496\,\Delta L_T^{\mathrm{pred}}
-0.00019.
\]

This resolves the uniform-high issue for the local theory.

## Interpretation

The fixed-initialization tangent-kernel theory is a local-time theorem. It
should be validated with \(T=O(1)\) small enough that the network remains in the
initial tangent regime.

The long-horizon memorization transition is a different contribution. It cannot
be validated by the same fixed-kernel formula unless we add a time-varying
kernel or explicit alignment-dynamics theory.