Probe early kernel predictors

1 files changed, 175 insertions, 0 deletions

diff --git a/notes/14_early_predictor_probe.md b/notes/14_early_predictor_probe.md
new file mode 100644
index 0000000..5cfe058
--- /dev/null
+++ b/notes/14_early_predictor_probe.md
@@ -0,0 +1,175 @@
+# Early Predictor Probe
+
+Measured time-varying \(K(t)\) is an oracle diagnostic. To get a predictive
+finite-\(T\) theory, we need quantities computable from initialization or the
+first few steps.
+
+## Probe Setup
+
+- architecture \(16\to64\to64\to4\);
+- \(N=128\);
+- target horizon \(T=50\);
+- SGD, learning rate \(10^{-3}\);
+- 4 init seeds and 8 feedback seeds;
+- early probes \(s\in\{1,2,5\}\).
+
+Candidate predictors:
+
+- fixed \(K(0)\) gap prediction;
+- FA kernel Frobenius drift:
+  \[
+  \|K_s^{FA}-K_0^{FA}\|_F/\|K_0^{FA}\|_F;
+  \]
+- BP kernel Frobenius drift:
+  \[
+  \|K_s^{BP}-K_0^{BP}\|_F/\|K_0^{BP}\|_F;
+  \]
+- FA/BP kernel overlap:
+  \[
+  \rho_s
+  =
+  \frac{\langle K_s^{FA},K_s^{BP}\rangle_F}
+  {\|K_s^{FA}\|_F\|K_s^{BP}\|_F};
+  \]
+- residual-direction gains:
+  \[
+  \lambda_s^{FA}
+  =
+  \frac{r_s^\top K_s^{FA}r_s}{\|r_s\|^2},
+  \qquad
+  \lambda_s^{BP}
+  =
+  \frac{r_s^\top K_s^{BP}r_s}{\|r_s\|^2}.
+  \]
+
+## Result
+
+Output:
+
+`outputs/early_kernel_predictors_T50_N128_32runs/early_kernel_predictors.csv`
+
+The strongest single predictor of the \(T=50\) empirical gap remains the fixed
+\(K(0)\) predicted gap:
+
+\[
+\operatorname{corr}
+\left(
+\Delta L_{50}^{fixedK},
+\Delta L_{50}^{emp}
+\right)
+\approx
+0.951.
+\]
+
+The fixed prediction is biased high:
+
+\[
+\operatorname{MAE}
+\left(
+\Delta L_{50}^{fixedK},
+\Delta L_{50}^{emp}
+\right)
+=
+0.02063.
+\]
+
+If we fit only a linear calibration of fixed gap on this single setting, the
+in-sample MAE drops to:
+
+\[
+0.00517.
+\]
+
+Adding an early drift predictor can reduce the in-sample residual further. The
+best early predictors were:
+
+- \(s=1\), fixed gap + FA directional drift:
+  \[
+  \operatorname{MAE}\approx0.00296;
+  \]
+- \(s=5\), fixed gap + BP Frobenius drift:
+  \[
+  \operatorname{MAE}\approx0.00307.
+  \]
+
+## Interpretation
+
+This probe is useful but not yet a theorem-ready predictor.
+
+The good news:
+
+1. fixed \(K(0)\) already ranks trajectories well at \(T=50\);
+2. early drift features explain part of the finite-\(T\) residual;
+3. this supports the finite-\(T\) picture:
+   \[
+   \Delta L_T
+   =
+   F(K_0)
+   +
+   \text{early drift correction}
+   +
+   \text{higher-order residual}.
+   \]
+
+The caution:
+
+The linear combinations above are in-sample on one architecture/task setting.
+They are not yet valid paper predictors. The next step must test whether a
+fixed formula learned or derived from one setting generalizes across \(N\),
+width, and feedback seeds.
+
+## Next Direction
+
+The most promising non-oracle predictive object is not raw Frobenius drift. It
+is the residual-direction kernel action:
+
+\[
+\lambda_t^{rule}
+=
+\frac{r_t^\top K_t^{rule}r_t}{\|r_t\|^2}.
+\]
+
+Reason: loss decrease is controlled by
+
+\[
+L_{t+1}-L_t
+\approx
+-
+\frac{\eta}{N}
+r_t^\top K_t r_t.
+\]
+
+So a finite-\(T\) predictor should approximate the cumulative directional gain:
+
+\[
+G_T^{rule}
+=
+\sum_{t<T}
+\frac{\eta}{N}
+\lambda_t^{rule}.
+\]
+
+For a predictive version, estimate the early slope:
+
+\[
+\lambda_t
+\approx
+\lambda_0+t\dot\lambda_0,
+\qquad
+\dot\lambda_0
+\approx
+\frac{\lambda_s-\lambda_0}{s},
+\]
+
+then predict
+
+\[
+G_T
+\approx
+\frac{\eta}{N}
+\left(
+T\lambda_0+\frac{T(T-1)}{2}\dot\lambda_0
+\right).
+\]
+
+This is more directly tied to loss dynamics than global kernel overlap.