From 11375078d783c4fa47a1901aac297c7c52e9c9d9 Mon Sep 17 00:00:00 2001
From: YurenHao0426 <Blackhao0426@gmail.com>
Date: Wed, 3 Jun 2026 10:02:51 -0500
Subject: Add early-kernel operator predictor

---
 notes/15_early_kernel_operator_predictor.md | 154 ++++++++++++++++++++++++++++
 1 file changed, 154 insertions(+)
 create mode 100644 notes/15_early_kernel_operator_predictor.md

(limited to 'notes')

diff --git a/notes/15_early_kernel_operator_predictor.md b/notes/15_early_kernel_operator_predictor.md
new file mode 100644
index 0000000..eacf852
--- /dev/null
+++ b/notes/15_early_kernel_operator_predictor.md
@@ -0,0 +1,154 @@
+# Early-Kernel Operator Predictor
+
+This note records the first predictive finite-\(T\) operator-level result.
+
+## Problem
+
+The fixed initialization tangent-kernel prediction
+
+\[
+r_{t+1}
+=
+\left(I-\frac{\eta}{N}K_0\right)r_t
+\]
+
+works at very short horizon, but at \(T=50\) it over-predicts the FA/BP train
+gap. The time-varying diagnostic in `notes/13_time_varying_kernel_diagnostic.md`
+showed that the error is caused mostly by kernel drift \(K_t-K_0\), not by a
+large second-order output residual.
+
+The goal here is a predictive, non-oracle correction: use only an early
+observation window, not the final loss gap.
+
+## Predictors
+
+Let \(s<T\) be an early probe step. For each BP or FA rule, compute
+
+\[
+K_0^{rule},
+\qquad
+K_s^{rule}.
+\]
+
+The strongest predictor in this probe is the early kernel velocity model:
+
+\[
+K_t^{rule}
+\approx
+K_0^{rule}
++
+t\frac{K_s^{rule}-K_0^{rule}}{s},
+\qquad
+0\le t<T.
+\]
+
+The residual prediction is then
+
+\[
+\hat r_{t+1}^{rule}
+=
+\left(
+I-\frac{\eta}{N}\hat K_t^{rule}
+\right)
+\hat r_t^{rule}.
+\]
+
+No fitted scale, offset, or final-gap calibration is used.
+
+We also tested two weaker alternatives:
+
+- FA-only compressed interpolation:
+  \[
+  K_t^{FA}
+  \approx
+  K_0^{FA}+\alpha_t(K_0^{BP}-K_0^{FA}),
+  \]
+  with \(\alpha_t\) inferred from the early projection of
+  \(K_s^{FA}-K_0^{FA}\).
+- early re-tangent:
+  run \(s\) empirical steps, then freeze \(K_s\) for the remaining \(T-s\)
+  steps.
+
+## Experiment
+
+- task: random-label regression;
+- architecture: \(16\to64\to64\to4\);
+- optimizer: full-batch SGD;
+- learning rate: \(10^{-3}\);
+- target horizon: \(T=50\);
+- early probe: \(s=20\);
+- train samples:
+  \[
+  N\in\{64,96,128,160,192,224,256,320\};
+  \]
+- 4 initialization seeds and 8 feedback seeds per \(N\);
+- 256 total trajectory points.
+
+Output directory:
+
+`outputs/compressed_operator_s20_256traj_T50_width64_plots`
+
+Key files:
+
+- `combined_rows.csv`
+- `predictor_metrics.csv`
+- `T50_s20_velocity_theory_band_empirical_trajectories.png`
+- `T50_s20_empirical_trajectories_only.png`
+- `T50_s20_prediction_scatter.png`
+- `T50_s20_prediction_error_by_margin.png`
+
+## Results
+
+Across all 256 trajectory points:
+
+| predictor | MAE | bias | corr |
+|---|---:|---:|---:|
+| fixed \(K(0)\) | 0.018487 | 0.018487 | 0.984460 |
+| FA compressed | 0.014915 | 0.014915 | 0.990472 |
+| early kernel velocity | 0.001893 | -0.001491 | 0.998848 |
+| early re-tangent | 0.004868 | 0.004868 | 0.999077 |
+
+Per-\(N\) summaries for the early kernel velocity model:
+
+| \(N\) | hard FA margin | fixed MAE | velocity MAE |
+|---:|---:|---:|---:|
+| 64 | 770 | 0.025444 | 0.001621 |
+| 96 | 642 | 0.021466 | 0.002578 |
+| 128 | 514 | 0.020630 | 0.001886 |
+| 160 | 386 | 0.017452 | 0.001879 |
+| 192 | 258 | 0.016190 | 0.001556 |
+| 224 | 130 | 0.015798 | 0.002354 |
+| 256 | 2 | 0.015586 | 0.001548 |
+| 320 | -254 | 0.015333 | 0.001725 |
+
+## Interpretation
+
+This is the first usable finite-\(T\) predictive version.
+
+The fixed \(K(0)\) theory gets the ordering and rough shape right, but its
+gap prediction is uniformly too high because it misses kernel drift. The early
+operator-velocity model removes almost all of that bias without using the final
+loss gap.
+
+The scalar directional-gain predictor in `notes/14_early_predictor_probe.md`
+failed because it compressed the dynamics too aggressively: it discarded
+changes in residual direction and in the full operator. The successful object
+is the operator \(K_t\), not a single scalar \(\lambda_t\).
+
+The early re-tangent model is conservative: it remains biased high. The linear
+velocity model is slightly biased low, so together they may define a clean
+no-fit prediction interval around the empirical gap.
+
+## Paper Use
+
+This should be folded into contribution 4:
+
+1. static distribution and local \(K(0)\) predictions;
+2. finite-\(T\) failure mode from kernel drift;
+3. predictive early-kernel correction;
+4. trajectory-level distribution validation.
+
+The current version is not yet a closed-form architecture-only theorem. It is a
+predictive finite-time theorem conditional on the early kernel pair
+\((K_0,K_s)\). That is acceptable if clearly separated from the prior-free
+initialization/capacity bounds.
-- 
cgit v1.2.3