# Early Predictor Probe

Measured time-varying \(K(t)\) is an oracle diagnostic. To get a predictive
finite-\(T\) theory, we need quantities computable from initialization or the
first few steps.

## Probe Setup

- architecture \(16\to64\to64\to4\);
- \(N=128\);
- target horizon \(T=50\);
- SGD, learning rate \(10^{-3}\);
- 4 init seeds and 8 feedback seeds;
- early probes \(s\in\{1,2,5\}\).

Candidate predictors:

- fixed \(K(0)\) gap prediction;
- FA kernel Frobenius drift:
  \[
  \|K_s^{FA}-K_0^{FA}\|_F/\|K_0^{FA}\|_F;
  \]
- BP kernel Frobenius drift:
  \[
  \|K_s^{BP}-K_0^{BP}\|_F/\|K_0^{BP}\|_F;
  \]
- FA/BP kernel overlap:
  \[
  \rho_s
  =
  \frac{\langle K_s^{FA},K_s^{BP}\rangle_F}
  {\|K_s^{FA}\|_F\|K_s^{BP}\|_F};
  \]
- residual-direction gains:
  \[
  \lambda_s^{FA}
  =
  \frac{r_s^\top K_s^{FA}r_s}{\|r_s\|^2},
  \qquad
  \lambda_s^{BP}
  =
  \frac{r_s^\top K_s^{BP}r_s}{\|r_s\|^2}.
  \]

## Result

Output:

`outputs/early_kernel_predictors_T50_N128_32runs/early_kernel_predictors.csv`

The strongest single predictor of the \(T=50\) empirical gap remains the fixed
\(K(0)\) predicted gap:

\[
\operatorname{corr}
\left(
\Delta L_{50}^{fixedK},
\Delta L_{50}^{emp}
\right)
\approx
0.951.
\]

The fixed prediction is biased high:

\[
\operatorname{MAE}
\left(
\Delta L_{50}^{fixedK},
\Delta L_{50}^{emp}
\right)
=
0.02063.
\]

If we fit only a linear calibration of fixed gap on this single setting, the
in-sample MAE drops to:

\[
0.00517.
\]

Adding an early drift predictor can reduce the in-sample residual further. The
best early predictors were:

- \(s=1\), fixed gap + FA directional drift:
  \[
  \operatorname{MAE}\approx0.00296;
  \]
- \(s=5\), fixed gap + BP Frobenius drift:
  \[
  \operatorname{MAE}\approx0.00307.
  \]

## Interpretation

This probe is useful but not yet a theorem-ready predictor.

The good news:

1. fixed \(K(0)\) already ranks trajectories well at \(T=50\);
2. early drift features explain part of the finite-\(T\) residual;
3. this supports the finite-\(T\) picture:
   \[
   \Delta L_T
   =
   F(K_0)
   +
   \text{early drift correction}
   +
   \text{higher-order residual}.
   \]

The caution:

The linear combinations above are in-sample on one architecture/task setting.
They are not yet valid paper predictors. The next step must test whether a
fixed formula learned or derived from one setting generalizes across \(N\),
width, and feedback seeds.

## Next Direction

The most promising non-oracle predictive object is not raw Frobenius drift. It
is the residual-direction kernel action:

\[
\lambda_t^{rule}
=
\frac{r_t^\top K_t^{rule}r_t}{\|r_t\|^2}.
\]

Reason: loss decrease is controlled by

\[
L_{t+1}-L_t
\approx
-
\frac{\eta}{N}
r_t^\top K_t r_t.
\]

So a finite-\(T\) predictor should approximate the cumulative directional gain:

\[
G_T^{rule}
=
\sum_{t<T}
\frac{\eta}{N}
\lambda_t^{rule}.
\]

For a predictive version, estimate the early slope:

\[
\lambda_t
\approx
\lambda_0+t\dot\lambda_0,
\qquad
\dot\lambda_0
\approx
\frac{\lambda_s-\lambda_0}{s},
\]

then predict

\[
G_T
\approx
\frac{\eta}{N}
\left(
T\lambda_0+\frac{T(T-1)}{2}\dot\lambda_0
\right).
\]

This is more directly tied to loss dynamics than global kernel overlap.

## Cross-N Directional Predictor Check

We also tested a no-fit directional-gain extrapolation across

\[
N\in\{64,96,128,160,192,224,256,320\}
\]

with width fixed at \(64\), target horizon \(T=50\), 2 init seeds, and 4
feedback seeds per init.

The attempted predictor used

\[
\lambda_t\approx\lambda_0+t\dot\lambda_0,
\qquad
\dot\lambda_0\approx\frac{\lambda_s-\lambda_0}{s},
\]

and

\[
L_T
\approx
L_0
\exp\left[
-\frac{2\eta}{N}
\left(
T\lambda_0+\frac{T(T-1)}{2}\dot\lambda_0
\right)
\right].
\]

This failed as a finite-\(T\) gap predictor. Across \(N\), the directional-gap
prediction had low correlation and large negative bias:

| early step | fixed \(K(0)\) corr | fixed MAE | directional corr | directional MAE |
|---:|---:|---:|---:|---:|
| 1 | 0.979 | 0.0241 | 0.124 | 0.2838 |
| 2 | 0.979 | 0.0241 | 0.138 | 0.2964 |
| 5 | 0.979 | 0.0241 | 0.110 | 0.3686 |

So the simple directional exponential formula should not be used.

The reason is likely that a scalar gain

\[
\lambda_t=\frac{r_t^\top K_t r_t}{\|r_t\|^2}
\]

does not preserve residual direction changes. BP and FA rotate the residual in
different ways, and the gap is a difference between two residual trajectories,
not just a difference between two scalar decay rates.

The more viable predictive object is therefore not a scalar exponential loss
model. It is a compressed operator model, for example:

\[
K_t^{FA}
\approx
K_0^{FA}
+
\alpha_t
\left(K_0^{BP}-K_0^{FA}\right),
\]

or a low-rank/residual-subspace version. The scalar order parameter can control
operator interpolation, but the residual update must remain vector-valued:

\[
r_{t+1}
=
\left(I-\frac{\eta}{N}K_t^{compressed}\right)r_t.
\]

This preserves direction changes and is closer to the measured \(K(t)\) product
that worked.