Record systematic low bias analysis

1 files changed, 210 insertions, 0 deletions

diff --git a/notes/17_systematic_low_bias.md b/notes/17_systematic_low_bias.md
new file mode 100644
index 0000000..41fefb8
--- /dev/null
+++ b/notes/17_systematic_low_bias.md
@@ -0,0 +1,210 @@
+# Systematic Low Bias in the Operator Predictor
+
+We accept the current early-kernel operator predictor as the working version:
+
+\[
+\hat K_t^{rule}
+=
+K_0^{rule}
++
+t\frac{K_s^{rule}-K_0^{rule}}{s}.
+\]
+
+It gives accurate trajectory-level gap distributions across depth, width, and
+finite horizon stress tests. The remaining issue is systematic: the predicted
+FA/BP gap is usually slightly lower than the empirical gap.
+
+## Error Decomposition
+
+The gap prediction is
+
+\[
+\hat\Delta
+=
+\hat L^{FA}-\hat L^{BP}.
+\]
+
+So a negative gap bias can come from either:
+
+- \(\hat L^{FA}\) too low;
+- \(\hat L^{BP}\) too high;
+- both.
+
+Across the stress grid, the dominant source is \(\hat L^{BP}\) too high. That
+means the linear-velocity predictor makes BP slightly too weak, rather than
+making FA too strong.
+
+| setting | gap bias | BP loss bias | FA loss bias |
+|---|---:|---:|---:|
+| \(d=1,w=64,T=50,s=20\) | -0.000233 | 0.000207 | -0.000026 |
+| \(d=2,w=32,T=50,s=20\) | -0.005375 | 0.005715 | 0.000340 |
+| \(d=2,w=64,T=25,s=10\) | -0.002668 | 0.002973 | 0.000305 |
+| \(d=2,w=64,T=50,s=20\) | -0.001491 | 0.001946 | 0.000455 |
+| \(d=2,w=64,T=100,s=40\) | -0.002169 | 0.002949 | 0.000779 |
+| \(d=2,w=96,T=50,s=20\) | -0.001551 | 0.002465 | 0.000914 |
+| \(d=3,w=64,T=50,s=20\) | -0.004916 | 0.005885 | 0.000969 |
+
+Aggregated over all stress-grid rows:
+
+\[
+\operatorname{bias}(\hat\Delta^{linear})
+\approx
+-0.00213,
+\]
+
+with
+
+\[
+\operatorname{bias}(\hat L^{BP})
+\approx
+0.00263,
+\qquad
+\operatorname{bias}(\hat L^{FA})
+\approx
+0.00050.
+\]
+
+Therefore the systematic low gap comes mainly from over-predicting BP loss.
+
+## Not a Normalization Bug
+
+The normalization is unlikely to be the cause.
+
+Reasons:
+
+1. The same residual recursion
+   \[
+   r_{t+1}=(I-\eta K_t/N)r_t
+   \]
+   was validated in the \(T=5\) local experiment without fitted scaling.
+2. The measured time-varying \(K(t)\) diagnostic at \(T=50\) matched empirical
+   losses to \(10^{-4}\)-\(10^{-3}\) absolute error.
+3. A wrong \(1/N\), \(1/2\), or MSE normalization would not selectively make
+   the linear BP loss too high while leaving measured \(K(t)\) accurate.
+
+## Step Matching
+
+The step indexing also looks correct.
+
+The code computes \(K_s\) after \(s\) SGD updates and uses
+
+\[
+\hat K_t = K_0 + \frac{t}{s}(K_s-K_0)
+\]
+
+for update index \(t=0,\dots,T-1\). Thus \(\hat K_s=K_s\). An off-by-one change
+such as \(s-1\) would make the slope larger and would worsen the observed
+negative bias. A denominator \(s+1\) would reduce the bias numerically, but that
+would be an ad hoc correction rather than the correct indexing.
+
+## Kernel-Curvature Probe
+
+We probed actual kernel drift by measuring
+
+\[
+\alpha_t
+=
+\frac{
+\langle K_t-K_0,K_s-K_0\rangle_F
+}{
+\|K_s-K_0\|_F^2
+}.
+\]
+
+The linear-velocity model predicts
+
+\[
+\alpha_t^{linear}=t/s.
+\]
+
+In representative runs, after \(t>s\), the true \(\alpha_t\) is consistently
+below \(t/s\). Example: \(d=2,w=64,N=128,T=50,s=20\).
+
+| rule | \(t\) | actual \(\alpha_t\) | linear \(t/s\) |
+|---|---:|---:|---:|
+| BP | 30 | 1.199 | 1.500 |
+| BP | 40 | 1.347 | 2.000 |
+| BP | 50 | 1.459 | 2.500 |
+| FA | 30 | 1.073 | 1.500 |
+| FA | 40 | 1.123 | 2.000 |
+| FA | 50 | 1.166 | 2.500 |
+
+The same pattern appears in narrow and deeper networks.
+
+This means \(K_t-K_0\) is not linear in time. The early drift saturates. The
+linear predictor over-extrapolates the kernel drift after the early window.
+
+## Directional-Gain Effect
+
+Loss decrease depends on the residual-direction gain
+
+\[
+g_t
+=
+\frac{r_t^\top K_t r_t}{\|r_t\|^2}.
+\]
+
+For \(d=2,w=64,N=128,T=50,s=20\), at \(t=50\):
+
+| rule | actual gain | linear gain | linear minus actual |
+|---|---:|---:|---:|
+| BP | 161.8 | 150.5 | -11.3 |
+| FA | 179.3 | 176.6 | -2.7 |
+
+The linear predictor underestimates the BP directional gain more strongly than
+the FA directional gain. This directly explains the negative gap bias:
+
+\[
+\hat L^{BP} > L^{BP}_{emp}
+\quad\Rightarrow\quad
+\hat L^{FA}-\hat L^{BP}
+<
+L^{FA}_{emp}-L^{BP}_{emp}.
+\]
+
+## Interpretation
+
+The missed term is kernel-path curvature:
+
+\[
+K_t
+=
+K_0
++
+t\dot K_0
++
+\frac12t^2\ddot K
++
+\cdots.
+\]
+
+The early secant model captures \(\dot K\) well enough to predict the gap
+shape, but it ignores that the drift velocity decays. This decay is stronger
+for BP in the residual-relevant direction, so the predicted BP loss is too high.
+
+This is why:
+
+- fixed \(K(0)\) is biased high for the gap: it makes BP too strong;
+- early re-tangent is still biased high: freezing \(K_s\) still makes BP too
+  strong relative to the future path;
+- linear velocity is biased low: it makes BP too weak by over-extrapolating
+  drift saturation.
+
+The empirical gap is bracketed by the conservative frozen-\(K_s\) predictor and
+the aggressive linear-velocity predictor.
+
+## Working Position
+
+For the current paper version, use:
+
+1. linear-velocity predictor as the main distribution predictor;
+2. early re-tangent as a conservative upper bracket;
+3. report the systematic low bias as kernel-curvature residual, not as fitted
+   normalization error.
+
+Do not introduce a fitted scalar correction.
+
+If a tighter future version is needed, the clean next step is a no-fit
+curvature model using two early checkpoints, e.g. \(K_{s/2}\) and \(K_s\), to
+estimate a decaying velocity. That would remain predictive, but it is not
+needed for the current acceptable version.