# Short-SGD Tangent-Kernel Validation

This note records the first replacement for the rejected Adam downstream plot.
The goal is to validate a quantitative theory object, not to fit a post-hoc
transition curve.

## Setup

Task:

- random-label regression;
- input dimension \(16\);
- output dimension \(4\);
- \(N=128\), so \(Nn_L=512\);
- architecture \(16\to n\to n\to 4\);
- widths \(n\in\{8,12,16,24,32,48,64,96\}\).

Training:

- optimizer: SGD;
- learning rate: \(10^{-3}\);
- steps: \(50\);
- init seeds: \(4\);
- feedback seeds per init: \(8\);
- total FA trajectories: \(8\times4\times8=256\).

The short horizon is intentional. It keeps the experiment close to the fixed
tangent-kernel regime where the theory is actually supposed to apply.

## Theory Prediction

For each initialization and feedback seed, compute the full training-set BP
Jacobian

\[
J_{\mathrm{BP}}\in\mathbb R^{512\times P},
\]

and the FA pseudo-Jacobian

\[
\tilde J_{\mathrm{FA}}\in\mathbb R^{512\times P}.
\]

Then form

\[
K_{\mathrm{BP}}=J_{\mathrm{BP}}J_{\mathrm{BP}}^\top,
\qquad
K_{\mathrm{FA}}=J_{\mathrm{BP}}\tilde J_{\mathrm{FA}}^\top.
\]

For the initial residual \(r_0=f_{\theta_0}(X)-y\), predict finite-time loss by

\[
r_T^{\mathrm{lin}}
=
\exp\left(-\frac{\eta T}{N}K\right)r_0,
\]

\[
L_T^{\mathrm{lin}}
=
\frac{1}{2N}\|r_T^{\mathrm{lin}}\|^2.
\]

The predicted FA/BP train gap is

\[
\Delta L_T^{\mathrm{pred}}
=
L_T^{\mathrm{lin,FA}}-L_T^{\mathrm{lin,BP}}.
\]

No fitted scale or shifted task dimension is used.

## Results

Output directory:

`outputs/fa_tangent_kernel_capacity_sgd_short_256`

Main figures:

- `outputs/fa_tangent_kernel_capacity_sgd_short_256/sgd_short_theory_distribution_empirical_trajectories.png`
- `outputs/fa_tangent_kernel_capacity_sgd_short_256/sgd_short_theory_empirical_mean_by_margin.png`
- `outputs/fa_tangent_kernel_capacity_sgd_short_256/predicted_vs_empirical_train_gap.png`

Aggregate over 256 FA trajectories:

\[
\operatorname{corr}
\left(
\Delta L_T^{\mathrm{pred}},
\Delta L_T^{\mathrm{emp}}
\right)
=
0.9725.
\]

\[
\operatorname{MAE}
=
0.0560,
\qquad
\operatorname{RMSE}
=
0.1066.
\]

The theory overpredicts the magnitude slightly:

\[
\mathbb E[\Delta L_T^{\mathrm{pred}}]=0.2211,
\qquad
\mathbb E[\Delta L_T^{\mathrm{emp}}]=0.1651.
\]

The empirical-vs-predicted linear fit is approximately

\[
\Delta L_T^{\mathrm{emp}}
\approx
0.546\,\Delta L_T^{\mathrm{pred}}+0.044.
\]

This is a diagnostic bias, not a fitted correction for the theory.

## Interpretation

The short-SGD experiment validates the tangent-kernel prediction as a
quantitative local theory: the predicted distribution and the empirical
trajectory distribution have the same structure, and the per-trajectory
correlation is high.

The earlier 1000-step SGD probe does not match the fixed initialization kernel:
the correlation becomes negative and the prediction overestimates gap magnitude.
That failure is expected because long FA training changes the forward weights
and feedback alignment, so \(K_{\mathrm{FA}}\) is time-varying.

Therefore the experimental program should be split cleanly:

1. short/lazy SGD: fixed-kernel quantitative distribution validation;
2. long-horizon memorization transition: time-varying FA kernel or explicit
   alignment-dynamics theory.

Do not use Adam for the main quantitative validation.