1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
|
# Finite-Time Extension Beyond Local Tangent Kernels
The tagged checkpoint
`t5-local-tangent-validation`
contains the clean local result: fixed-initialization tangent kernels predict
short-horizon FA/BP gap distributions at \(T=5\) without fitted scale.
The remaining practical question is how to handle larger \(T\).
## Core Issue
The local theorem uses
\[
K_{\mathrm{FA}}(0)
=
J_{\mathrm{BP}}(\theta_0)
\tilde J_{\mathrm{FA}}(\theta_0)^\top
\]
and predicts
\[
r_T
\approx
\exp\left(-\frac{\eta T}{N}K_{\mathrm{FA}}(0)\right)r_0.
\]
This is accurate for very small \(T\), but for larger \(T\):
\[
K_{\mathrm{FA}}(t)
=
J_{\mathrm{BP}}(\theta_t)
\tilde J_{\mathrm{FA}}(\theta_t)^\top
\]
changes with the weights, gates, residual direction, and learned FA alignment.
So the correct finite-time first-order model is
\[
r_{t+1}
\approx
\left(
I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
\right)r_t,
\]
and therefore
\[
r_T
\approx
\prod_{t=0}^{T-1}
\left(
I-\frac{\eta}{N}K_{\mathrm{FA}}(t)
\right)r_0.
\]
This is the natural finite-\(T\) extension. It has no fitted scalar, but it does
require knowing or modeling \(K_{\mathrm{FA}}(t)\).
## Level 1: Time-Varying Kernel Diagnostic
First compute \(K_{\mathrm{FA}}(t)\) along actual FA trajectories at selected
checkpoints:
\[
t\in\{0,1,2,5,10,20,50,\dots\}.
\]
Then roll the residual forward with the product formula above.
This answers one immediate question:
> Is the \(T=50\) error explained by kernel drift, or by higher-order output
> Taylor error?
If the time-varying product matches empirical \(T=50\), then the right theory is
kernel evolution. If it does not, then second-order output Taylor terms are
non-negligible.
This is not yet a predictive theorem because it uses trajectory kernels, but it
is the necessary diagnostic bridge.
## Level 2: Perturbative Finite-Time Theory
For moderate \(T\), expand the kernel:
\[
K_t
=
K_0+t\dot K_0+O(t^2\|\ddot K\|).
\]
The first derivative is
\[
\dot K_0
=
\nabla_\theta K(\theta_0)
\left[
-\frac{\eta}{N}
\tilde J_{\mathrm{FA}}(\theta_0)^\top r_0
\right].
\]
Then a Duhamel expansion gives
\[
r_T
\approx
\exp(-\tau K_0)r_0
-
\int_0^\tau
\exp[-(\tau-s)K_0]
\Delta K(s)
\exp[-sK_0]r_0\,ds,
\]
where
\[
\tau=\frac{\eta T}{N},
\qquad
\Delta K(s)\approx s\,\dot K_0.
\]
This gives a finite-time correction with no empirical fitted scale. The cost is
that it involves derivatives of the tangent kernel, i.e. a neural-tangent-
hierarchy style object.
This is theorem-friendly but may be heavy.
## Level 3: Low-Dimensional Alignment Dynamics
For a paper-friendly practical theory, approximate the changing FA kernel by a
small number of order parameters:
\[
K_{\mathrm{FA}}(t)
\approx
a_t K_{\mathrm{FA}}(0)
+b_t K_{\mathrm{BP}}(0)
+E_t.
\]
Here \(b_t\) captures learned alignment toward the BP kernel. The order
parameters can be defined from measurable, theory-motivated quantities:
\[
q_t
=
\cos^2(W_{l+1}(t)^\top,B_l),
\]
or from kernel overlap:
\[
\rho_t
=
\frac{
\langle K_{\mathrm{FA}}(t),K_{\mathrm{BP}}(t)\rangle_F
}{
\|K_{\mathrm{FA}}(t)\|_F
\|K_{\mathrm{BP}}(t)\|_F
}.
\]
The target would be a closed finite-time prediction:
\[
r_T
\approx
\prod_t
\left[
I-\frac{\eta}{N}
\left(
a_tK_{\mathrm{FA}}(0)+b_tK_{\mathrm{BP}}(0)
\right)
\right]r_0.
\]
This becomes useful only if \(a_t,b_t\) obey a simple architecture-dependent
trajectory law. Otherwise it becomes another post-hoc fit, which should not be
used as a main theorem.
## Recommended Next Experiment
Do not jump directly to a new fitted finite-\(T\) curve.
Run a diagnostic experiment with a small number of trajectories:
1. train BP and FA with SGD to \(T=50\);
2. save checkpoints at \(t=\{0,1,2,5,10,20,50\}\);
3. compute \(K_{\mathrm{BP}}(t)\), \(K_{\mathrm{FA}}(t)\), and residuals;
4. compare three predictions:
- fixed kernel: \(K(0)\);
- piecewise time-varying product using measured \(K(t)\);
- first-order drift correction using \(\dot K_0\), if feasible.
If the measured time-varying product matches, then finite-\(T\) theory should
focus on modeling kernel/alignment evolution. If it does not, then we need a
second-order output expansion.
## Paper Structure Implication
Use the current \(T=5\) result for the rigorous local distribution theorem.
Use larger \(T\) only after adding a separate finite-time theorem:
\[
\text{random feedback}
\rightarrow
K_{\mathrm{FA}}(0)
\rightarrow
K_{\mathrm{FA}}(t)
\rightarrow
\Delta L_T.
\]
This keeps the four-contribution structure intact without pretending the local
theorem already explains long-horizon memorization.
|
