Plan finite-time tangent extension

1 files changed, 226 insertions, 0 deletions

diff --git a/notes/11_finite_time_extension.md b/notes/11_finite_time_extension.md
new file mode 100644
index 0000000..1eee267
--- /dev/null
+++ b/notes/11_finite_time_extension.md
@@ -0,0 +1,226 @@
+# Finite-Time Extension Beyond Local Tangent Kernels
+
+The tagged checkpoint
+
+`t5-local-tangent-validation`
+
+contains the clean local result: fixed-initialization tangent kernels predict
+short-horizon FA/BP gap distributions at \(T=5\) without fitted scale.
+
+The remaining practical question is how to handle larger \(T\).
+
+## Core Issue
+
+The local theorem uses
+
+\[
+K_{\mathrm{FA}}(0)
+=
+J_{\mathrm{BP}}(\theta_0)
+\tilde J_{\mathrm{FA}}(\theta_0)^\top
+\]
+
+and predicts
+
+\[
+r_T
+\approx
+\exp\left(-\frac{\eta T}{N}K_{\mathrm{FA}}(0)\right)r_0.
+\]
+
+This is accurate for very small \(T\), but for larger \(T\):
+
+\[
+K_{\mathrm{FA}}(t)
+=
+J_{\mathrm{BP}}(\theta_t)
+\tilde J_{\mathrm{FA}}(\theta_t)^\top
+\]
+
+changes with the weights, gates, residual direction, and learned FA alignment.
+
+So the correct finite-time first-order model is
+
+\[
+r_{t+1}
+\approx
+\left(
+I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
+\right)r_t,
+\]
+
+and therefore
+
+\[
+r_T
+\approx
+\prod_{t=0}^{T-1}
+\left(
+I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
+\right)r_0.
+\]
+
+This is the natural finite-\(T\) extension. It has no fitted scalar, but it does
+require knowing or modeling \(K_{\mathrm{FA}}(t)\).
+
+## Level 1: Time-Varying Kernel Diagnostic
+
+First compute \(K_{\mathrm{FA}}(t)\) along actual FA trajectories at selected
+checkpoints:
+
+\[
+t\in\{0,1,2,5,10,20,50,\dots\}.
+\]
+
+Then roll the residual forward with the product formula above.
+
+This answers one immediate question:
+
+> Is the \(T=50\) error explained by kernel drift, or by higher-order output
+> Taylor error?
+
+If the time-varying product matches empirical \(T=50\), then the right theory is
+kernel evolution. If it does not, then second-order output Taylor terms are
+non-negligible.
+
+This is not yet a predictive theorem because it uses trajectory kernels, but it
+is the necessary diagnostic bridge.
+
+## Level 2: Perturbative Finite-Time Theory
+
+For moderate \(T\), expand the kernel:
+
+\[
+K_t
+=
+K_0+t\dot K_0+O(t^2\|\ddot K\|).
+\]
+
+The first derivative is
+
+\[
+\dot K_0
+=
+\nabla_\theta K(\theta_0)
+\left[
+-\frac{\eta}{N}
+\tilde J_{\mathrm{FA}}(\theta_0)^\top r_0
+\right].
+\]
+
+Then a Duhamel expansion gives
+
+\[
+r_T
+\approx
+\exp(-\tau K_0)r_0
+-
+\int_0^\tau
+\exp[-(\tau-s)K_0]
+\Delta K(s)
+\exp[-sK_0]r_0\,ds,
+\]
+
+where
+
+\[
+\tau=\frac{\eta T}{N},
+\qquad
+\Delta K(s)\approx s\,\dot K_0.
+\]
+
+This gives a finite-time correction with no empirical fitted scale. The cost is
+that it involves derivatives of the tangent kernel, i.e. a neural-tangent-
+hierarchy style object.
+
+This is theorem-friendly but may be heavy.
+
+## Level 3: Low-Dimensional Alignment Dynamics
+
+For a paper-friendly practical theory, approximate the changing FA kernel by a
+small number of order parameters:
+
+\[
+K_{\mathrm{FA}}(t)
+\approx
+a_t K_{\mathrm{FA}}(0)
++b_t K_{\mathrm{BP}}(0)
++E_t.
+\]
+
+Here \(b_t\) captures learned alignment toward the BP kernel. The order
+parameters can be defined from measurable, theory-motivated quantities:
+
+\[
+q_t
+=
+\cos^2(W_{l+1}(t)^\top,B_l),
+\]
+
+or from kernel overlap:
+
+\[
+\rho_t
+=
+\frac{
+\langle K_{\mathrm{FA}}(t),K_{\mathrm{BP}}(t)\rangle_F
+}{
+\|K_{\mathrm{FA}}(t)\|_F
+\|K_{\mathrm{BP}}(t)\|_F
+}.
+\]
+
+The target would be a closed finite-time prediction:
+
+\[
+r_T
+\approx
+\prod_t
+\left[
+I-\frac{\eta}{N}
+\left(
+a_tK_{\mathrm{FA}}(0)+b_tK_{\mathrm{BP}}(0)
+\right)
+\right]r_0.
+\]
+
+This becomes useful only if \(a_t,b_t\) obey a simple architecture-dependent
+trajectory law. Otherwise it becomes another post-hoc fit, which should not be
+used as a main theorem.
+
+## Recommended Next Experiment
+
+Do not jump directly to a new fitted finite-\(T\) curve.
+
+Run a diagnostic experiment with a small number of trajectories:
+
+1. train BP and FA with SGD to \(T=50\);
+2. save checkpoints at \(t=\{0,1,2,5,10,20,50\}\);
+3. compute \(K_{\mathrm{BP}}(t)\), \(K_{\mathrm{FA}}(t)\), and residuals;
+4. compare three predictions:
+   - fixed kernel: \(K(0)\);
+   - piecewise time-varying product using measured \(K(t)\);
+   - first-order drift correction using \(\dot K_0\), if feasible.
+
+If the measured time-varying product matches, then finite-\(T\) theory should
+focus on modeling kernel/alignment evolution. If it does not, then we need a
+second-order output expansion.
+
+## Paper Structure Implication
+
+Use the current \(T=5\) result for the rigorous local distribution theorem.
+
+Use larger \(T\) only after adding a separate finite-time theorem:
+
+\[
+\text{random feedback}
+\rightarrow
+K_{\mathrm{FA}}(0)
+\rightarrow
+K_{\mathrm{FA}}(t)
+\rightarrow
+\Delta L_T.
+\]
+
+This keeps the four-contribution structure intact without pretending the local
+theorem already explains long-horizon memorization.