1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
|
# Early-Kernel Operator Predictor
This note records the first predictive finite-\(T\) operator-level result.
## Problem
The fixed initialization tangent-kernel prediction
\[
r_{t+1}
=
\left(I-\frac{\eta}{N}K_0\right)r_t
\]
works at very short horizon, but at \(T=50\) it over-predicts the FA/BP train
gap. The time-varying diagnostic in `notes/13_time_varying_kernel_diagnostic.md`
showed that the error is caused mostly by kernel drift \(K_t-K_0\), not by a
large second-order output residual.
The goal here is a predictive, non-oracle correction: use only an early
observation window, not the final loss gap.
## Predictors
Let \(s<T\) be an early probe step. For each BP or FA rule, compute
\[
K_0^{rule},
\qquad
K_s^{rule}.
\]
The strongest predictor in this probe is the early kernel velocity model:
\[
K_t^{rule}
\approx
K_0^{rule}
+
t\frac{K_s^{rule}-K_0^{rule}}{s},
\qquad
0\le t<T.
\]
The residual prediction is then
\[
\hat r_{t+1}^{rule}
=
\left(
I-\frac{\eta}{N}\hat K_t^{rule}
\right)
\hat r_t^{rule}.
\]
No fitted scale, offset, or final-gap calibration is used.
We also tested two weaker alternatives:
- FA-only compressed interpolation:
\[
K_t^{FA}
\approx
K_0^{FA}+\alpha_t(K_0^{BP}-K_0^{FA}),
\]
with \(\alpha_t\) inferred from the early projection of
\(K_s^{FA}-K_0^{FA}\).
- early re-tangent:
run \(s\) empirical steps, then freeze \(K_s\) for the remaining \(T-s\)
steps.
## Experiment
- task: random-label regression;
- architecture: \(16\to64\to64\to4\);
- optimizer: full-batch SGD;
- learning rate: \(10^{-3}\);
- target horizon: \(T=50\);
- early probe: \(s=20\);
- train samples:
\[
N\in\{64,96,128,160,192,224,256,320\};
\]
- 4 initialization seeds and 8 feedback seeds per \(N\);
- 256 total trajectory points.
Output directory:
`outputs/compressed_operator_s20_256traj_T50_width64_plots`
Key files:
- `combined_rows.csv`
- `predictor_metrics.csv`
- `T50_s20_velocity_theory_band_empirical_trajectories.png`
- `T50_s20_empirical_trajectories_only.png`
- `T50_s20_prediction_scatter.png`
- `T50_s20_prediction_error_by_margin.png`
## Results
Across all 256 trajectory points:
| predictor | MAE | bias | corr |
|---|---:|---:|---:|
| fixed \(K(0)\) | 0.018487 | 0.018487 | 0.984460 |
| FA compressed | 0.014915 | 0.014915 | 0.990472 |
| early kernel velocity | 0.001893 | -0.001491 | 0.998848 |
| early re-tangent | 0.004868 | 0.004868 | 0.999077 |
Per-\(N\) summaries for the early kernel velocity model:
| \(N\) | hard FA margin | fixed MAE | velocity MAE |
|---:|---:|---:|---:|
| 64 | 770 | 0.025444 | 0.001621 |
| 96 | 642 | 0.021466 | 0.002578 |
| 128 | 514 | 0.020630 | 0.001886 |
| 160 | 386 | 0.017452 | 0.001879 |
| 192 | 258 | 0.016190 | 0.001556 |
| 224 | 130 | 0.015798 | 0.002354 |
| 256 | 2 | 0.015586 | 0.001548 |
| 320 | -254 | 0.015333 | 0.001725 |
## Interpretation
This is the first usable finite-\(T\) predictive version.
The fixed \(K(0)\) theory gets the ordering and rough shape right, but its
gap prediction is uniformly too high because it misses kernel drift. The early
operator-velocity model removes almost all of that bias without using the final
loss gap.
The scalar directional-gain predictor in `notes/14_early_predictor_probe.md`
failed because it compressed the dynamics too aggressively: it discarded
changes in residual direction and in the full operator. The successful object
is the operator \(K_t\), not a single scalar \(\lambda_t\).
The early re-tangent model is conservative: it remains biased high. The linear
velocity model is slightly biased low, so together they may define a clean
no-fit prediction interval around the empirical gap.
## Paper Use
This should be folded into contribution 4:
1. static distribution and local \(K(0)\) predictions;
2. finite-\(T\) failure mode from kernel drift;
3. predictive early-kernel correction;
4. trajectory-level distribution validation.
The current version is not yet a closed-form architecture-only theorem. It is a
predictive finite-time theorem conditional on the early kernel pair
\((K_0,K_s)\). That is acceptable if clearly separated from the prior-free
initialization/capacity bounds.
|
