# Finite-Time Extension Beyond Local Tangent Kernels

The tagged checkpoint

`t5-local-tangent-validation`

contains the clean local result: fixed-initialization tangent kernels predict
short-horizon FA/BP gap distributions at \(T=5\) without fitted scale.

The remaining practical question is how to handle larger \(T\).

## Core Issue

The local theorem uses

\[
K_{\mathrm{FA}}(0)
=
J_{\mathrm{BP}}(\theta_0)
\tilde J_{\mathrm{FA}}(\theta_0)^\top
\]

and predicts

\[
r_T
\approx
\exp\left(-\frac{\eta T}{N}K_{\mathrm{FA}}(0)\right)r_0.
\]

This is accurate for very small \(T\), but for larger \(T\):

\[
K_{\mathrm{FA}}(t)
=
J_{\mathrm{BP}}(\theta_t)
\tilde J_{\mathrm{FA}}(\theta_t)^\top
\]

changes with the weights, gates, residual direction, and learned FA alignment.

So the correct finite-time first-order model is

\[
r_{t+1}
\approx
\left(
I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
\right)r_t,
\]

and therefore

\[
r_T
\approx
\prod_{t=0}^{T-1}
\left(
I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
\right)r_0.
\]

This is the natural finite-\(T\) extension. It has no fitted scalar, but it does
require knowing or modeling \(K_{\mathrm{FA}}(t)\).

## Level 1: Time-Varying Kernel Diagnostic

First compute \(K_{\mathrm{FA}}(t)\) along actual FA trajectories at selected
checkpoints:

\[
t\in\{0,1,2,5,10,20,50,\dots\}.
\]

Then roll the residual forward with the product formula above.

This answers one immediate question:

> Is the \(T=50\) error explained by kernel drift, or by higher-order output
> Taylor error?

If the time-varying product matches empirical \(T=50\), then the right theory is
kernel evolution. If it does not, then second-order output Taylor terms are
non-negligible.

This is not yet a predictive theorem because it uses trajectory kernels, but it
is the necessary diagnostic bridge.

## Level 2: Perturbative Finite-Time Theory

For moderate \(T\), expand the kernel:

\[
K_t
=
K_0+t\dot K_0+O(t^2\|\ddot K\|).
\]

The first derivative is

\[
\dot K_0
=
\nabla_\theta K(\theta_0)
\left[
-\frac{\eta}{N}
\tilde J_{\mathrm{FA}}(\theta_0)^\top r_0
\right].
\]

Then a Duhamel expansion gives

\[
r_T
\approx
\exp(-\tau K_0)r_0
-
\int_0^\tau
\exp[-(\tau-s)K_0]
\Delta K(s)
\exp[-sK_0]r_0\,ds,
\]

where

\[
\tau=\frac{\eta T}{N},
\qquad
\Delta K(s)\approx s\,\dot K_0.
\]

This gives a finite-time correction with no empirical fitted scale. The cost is
that it involves derivatives of the tangent kernel, i.e. a neural-tangent-
hierarchy style object.

This is theorem-friendly but may be heavy.

## Level 3: Low-Dimensional Alignment Dynamics

For a paper-friendly practical theory, approximate the changing FA kernel by a
small number of order parameters:

\[
K_{\mathrm{FA}}(t)
\approx
a_t K_{\mathrm{FA}}(0)
+b_t K_{\mathrm{BP}}(0)
+E_t.
\]

Here \(b_t\) captures learned alignment toward the BP kernel. The order
parameters can be defined from measurable, theory-motivated quantities:

\[
q_t
=
\cos^2(W_{l+1}(t)^\top,B_l),
\]

or from kernel overlap:

\[
\rho_t
=
\frac{
\langle K_{\mathrm{FA}}(t),K_{\mathrm{BP}}(t)\rangle_F
}{
\|K_{\mathrm{FA}}(t)\|_F
\|K_{\mathrm{BP}}(t)\|_F
}.
\]

The target would be a closed finite-time prediction:

\[
r_T
\approx
\prod_t
\left[
I-\frac{\eta}{N}
\left(
a_tK_{\mathrm{FA}}(0)+b_tK_{\mathrm{BP}}(0)
\right)
\right]r_0.
\]

This becomes useful only if \(a_t,b_t\) obey a simple architecture-dependent
trajectory law. Otherwise it becomes another post-hoc fit, which should not be
used as a main theorem.

## Recommended Next Experiment

Do not jump directly to a new fitted finite-\(T\) curve.

Run a diagnostic experiment with a small number of trajectories:

1. train BP and FA with SGD to \(T=50\);
2. save checkpoints at \(t=\{0,1,2,5,10,20,50\}\);
3. compute \(K_{\mathrm{BP}}(t)\), \(K_{\mathrm{FA}}(t)\), and residuals;
4. compare three predictions:
   - fixed kernel: \(K(0)\);
   - piecewise time-varying product using measured \(K(t)\);
   - first-order drift correction using \(\dot K_0\), if feasible.

If the measured time-varying product matches, then finite-\(T\) theory should
focus on modeling kernel/alignment evolution. If it does not, then we need a
second-order output expansion.

## Paper Structure Implication

Use the current \(T=5\) result for the rigorous local distribution theorem.

Use larger \(T\) only after adding a separate finite-time theorem:

\[
\text{random feedback}
\rightarrow
K_{\mathrm{FA}}(0)
\rightarrow
K_{\mathrm{FA}}(t)
\rightarrow
\Delta L_T.
\]

This keeps the four-contribution structure intact without pretending the local
theorem already explains long-horizon memorization.