diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/15_early_kernel_operator_predictor.md | 154 |
1 files changed, 154 insertions, 0 deletions
diff --git a/notes/15_early_kernel_operator_predictor.md b/notes/15_early_kernel_operator_predictor.md new file mode 100644 index 0000000..eacf852 --- /dev/null +++ b/notes/15_early_kernel_operator_predictor.md @@ -0,0 +1,154 @@ +# Early-Kernel Operator Predictor + +This note records the first predictive finite-\(T\) operator-level result. + +## Problem + +The fixed initialization tangent-kernel prediction + +\[ +r_{t+1} += +\left(I-\frac{\eta}{N}K_0\right)r_t +\] + +works at very short horizon, but at \(T=50\) it over-predicts the FA/BP train +gap. The time-varying diagnostic in `notes/13_time_varying_kernel_diagnostic.md` +showed that the error is caused mostly by kernel drift \(K_t-K_0\), not by a +large second-order output residual. + +The goal here is a predictive, non-oracle correction: use only an early +observation window, not the final loss gap. + +## Predictors + +Let \(s<T\) be an early probe step. For each BP or FA rule, compute + +\[ +K_0^{rule}, +\qquad +K_s^{rule}. +\] + +The strongest predictor in this probe is the early kernel velocity model: + +\[ +K_t^{rule} +\approx +K_0^{rule} ++ +t\frac{K_s^{rule}-K_0^{rule}}{s}, +\qquad +0\le t<T. +\] + +The residual prediction is then + +\[ +\hat r_{t+1}^{rule} += +\left( +I-\frac{\eta}{N}\hat K_t^{rule} +\right) +\hat r_t^{rule}. +\] + +No fitted scale, offset, or final-gap calibration is used. + +We also tested two weaker alternatives: + +- FA-only compressed interpolation: + \[ + K_t^{FA} + \approx + K_0^{FA}+\alpha_t(K_0^{BP}-K_0^{FA}), + \] + with \(\alpha_t\) inferred from the early projection of + \(K_s^{FA}-K_0^{FA}\). +- early re-tangent: + run \(s\) empirical steps, then freeze \(K_s\) for the remaining \(T-s\) + steps. + +## Experiment + +- task: random-label regression; +- architecture: \(16\to64\to64\to4\); +- optimizer: full-batch SGD; +- learning rate: \(10^{-3}\); +- target horizon: \(T=50\); +- early probe: \(s=20\); +- train samples: + \[ + N\in\{64,96,128,160,192,224,256,320\}; + \] +- 4 initialization seeds and 8 feedback seeds per \(N\); +- 256 total trajectory points. + +Output directory: + +`outputs/compressed_operator_s20_256traj_T50_width64_plots` + +Key files: + +- `combined_rows.csv` +- `predictor_metrics.csv` +- `T50_s20_velocity_theory_band_empirical_trajectories.png` +- `T50_s20_empirical_trajectories_only.png` +- `T50_s20_prediction_scatter.png` +- `T50_s20_prediction_error_by_margin.png` + +## Results + +Across all 256 trajectory points: + +| predictor | MAE | bias | corr | +|---|---:|---:|---:| +| fixed \(K(0)\) | 0.018487 | 0.018487 | 0.984460 | +| FA compressed | 0.014915 | 0.014915 | 0.990472 | +| early kernel velocity | 0.001893 | -0.001491 | 0.998848 | +| early re-tangent | 0.004868 | 0.004868 | 0.999077 | + +Per-\(N\) summaries for the early kernel velocity model: + +| \(N\) | hard FA margin | fixed MAE | velocity MAE | +|---:|---:|---:|---:| +| 64 | 770 | 0.025444 | 0.001621 | +| 96 | 642 | 0.021466 | 0.002578 | +| 128 | 514 | 0.020630 | 0.001886 | +| 160 | 386 | 0.017452 | 0.001879 | +| 192 | 258 | 0.016190 | 0.001556 | +| 224 | 130 | 0.015798 | 0.002354 | +| 256 | 2 | 0.015586 | 0.001548 | +| 320 | -254 | 0.015333 | 0.001725 | + +## Interpretation + +This is the first usable finite-\(T\) predictive version. + +The fixed \(K(0)\) theory gets the ordering and rough shape right, but its +gap prediction is uniformly too high because it misses kernel drift. The early +operator-velocity model removes almost all of that bias without using the final +loss gap. + +The scalar directional-gain predictor in `notes/14_early_predictor_probe.md` +failed because it compressed the dynamics too aggressively: it discarded +changes in residual direction and in the full operator. The successful object +is the operator \(K_t\), not a single scalar \(\lambda_t\). + +The early re-tangent model is conservative: it remains biased high. The linear +velocity model is slightly biased low, so together they may define a clean +no-fit prediction interval around the empirical gap. + +## Paper Use + +This should be folded into contribution 4: + +1. static distribution and local \(K(0)\) predictions; +2. finite-\(T\) failure mode from kernel drift; +3. predictive early-kernel correction; +4. trajectory-level distribution validation. + +The current version is not yet a closed-form architecture-only theorem. It is a +predictive finite-time theorem conditional on the early kernel pair +\((K_0,K_s)\). That is acceptable if clearly separated from the prior-free +initialization/capacity bounds. |