diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/14_early_predictor_probe.md | 78 |
1 files changed, 78 insertions, 0 deletions
diff --git a/notes/14_early_predictor_probe.md b/notes/14_early_predictor_probe.md index 5cfe058..43b9b6f 100644 --- a/notes/14_early_predictor_probe.md +++ b/notes/14_early_predictor_probe.md @@ -173,3 +173,81 @@ T\lambda_0+\frac{T(T-1)}{2}\dot\lambda_0 \] This is more directly tied to loss dynamics than global kernel overlap. + +## Cross-N Directional Predictor Check + +We also tested a no-fit directional-gain extrapolation across + +\[ +N\in\{64,96,128,160,192,224,256,320\} +\] + +with width fixed at \(64\), target horizon \(T=50\), 2 init seeds, and 4 +feedback seeds per init. + +The attempted predictor used + +\[ +\lambda_t\approx\lambda_0+t\dot\lambda_0, +\qquad +\dot\lambda_0\approx\frac{\lambda_s-\lambda_0}{s}, +\] + +and + +\[ +L_T +\approx +L_0 +\exp\left[ +-\frac{2\eta}{N} +\left( +T\lambda_0+\frac{T(T-1)}{2}\dot\lambda_0 +\right) +\right]. +\] + +This failed as a finite-\(T\) gap predictor. Across \(N\), the directional-gap +prediction had low correlation and large negative bias: + +| early step | fixed \(K(0)\) corr | fixed MAE | directional corr | directional MAE | +|---:|---:|---:|---:|---:| +| 1 | 0.979 | 0.0241 | 0.124 | 0.2838 | +| 2 | 0.979 | 0.0241 | 0.138 | 0.2964 | +| 5 | 0.979 | 0.0241 | 0.110 | 0.3686 | + +So the simple directional exponential formula should not be used. + +The reason is likely that a scalar gain + +\[ +\lambda_t=\frac{r_t^\top K_t r_t}{\|r_t\|^2} +\] + +does not preserve residual direction changes. BP and FA rotate the residual in +different ways, and the gap is a difference between two residual trajectories, +not just a difference between two scalar decay rates. + +The more viable predictive object is therefore not a scalar exponential loss +model. It is a compressed operator model, for example: + +\[ +K_t^{FA} +\approx +K_0^{FA} ++ +\alpha_t +\left(K_0^{BP}-K_0^{FA}\right), +\] + +or a low-rank/residual-subspace version. The scalar order parameter can control +operator interpolation, but the residual update must remain vector-valued: + +\[ +r_{t+1} += +\left(I-\frac{\eta}{N}K_t^{compressed}\right)r_t. +\] + +This preserves direction changes and is closer to the measured \(K(t)\) product +that worked. |
