1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
# T=5 Local Tangent-Kernel Validation
The earlier short-SGD figure still had theory uniformly above empirical gaps.
That was not fixed by only switching to a fixed-width N-sweep. The remaining
issue was that \(T=50\) is too long for a fixed-initialization tangent-kernel
prediction.
## Horizon Diagnostic
Using the same width/N/seeds and varying only the number of SGD steps:
| steps | empirical gap | predicted gap | empirical / predicted | MAE |
|---:|---:|---:|---:|---:|
| 1 | 0.030734 | 0.029853 | 1.028 | 0.000881 |
| 5 | 0.112912 | 0.113947 | 0.987 | 0.001712 |
| 10 | 0.156304 | 0.164964 | 0.942 | 0.008660 |
| 20 | 0.163335 | 0.183836 | 0.884 | 0.020500 |
| 50 | 0.125547 | 0.154075 | 0.814 | 0.028528 |
So the uniform-high theory issue is a finite-horizon error: fixed
initialization kernels match \(T=1\) and \(T=5\), then drift out of regime.
This is not a scaling bug:
- discrete Euler vs continuous exponential has negligible effect;
- the \(1/N\) factor is consistent with the implemented full-batch MSE
gradient;
- the mismatch grows with training horizon.
## Main Local Validation
Use a fixed-width N-sweep:
- architecture \(16\to64\to64\to4\);
- random-label regression;
- \(N\in\{64,96,128,160,192,224,256,320\}\);
- SGD, learning rate \(10^{-3}\);
- steps \(T=5\);
- 4 init seeds and 8 feedback seeds per init;
- total FA trajectories \(8\times4\times8=256\).
Theory uses the fixed initialization kernels:
\[
K_{\mathrm{BP}}=J_{\mathrm{BP}}J_{\mathrm{BP}}^\top,
\qquad
K_{\mathrm{FA}}=J_{\mathrm{BP}}\tilde J_{\mathrm{FA}}^\top.
\]
Prediction:
\[
r_T^{\mathrm{lin}}
=
\exp\left(-\frac{\eta T}{N}K\right)r_0,
\qquad
L_T^{\mathrm{lin}}
=
\frac{1}{2N}\|r_T^{\mathrm{lin}}\|^2.
\]
No fitted multiplicative scale or offset is used.
## Results
Combined figure:
`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_theory_empirical_overlay.png`
Scatter:
`outputs/fa_tangent_kernel_fixed_width64_sgd_T5_combined/T5_fixed_width64_N_sweep_predicted_vs_empirical.png`
Across 256 FA trajectories:
\[
\operatorname{corr}
\left(
\Delta L_T^{\mathrm{pred}},
\Delta L_T^{\mathrm{emp}}
\right)
=
0.99947.
\]
\[
\operatorname{MAE}=0.00083,
\qquad
\operatorname{RMSE}=0.00124.
\]
Mean gap:
\[
\mathbb E[\Delta L_T^{\mathrm{emp}}]=0.05953,
\qquad
\mathbb E[\Delta L_T^{\mathrm{pred}}]=0.06003.
\]
Ratio:
\[
\mathbb E[\Delta L_T^{\mathrm{emp}}/\Delta L_T^{\mathrm{pred}}]
=
0.99286.
\]
Linear fit:
\[
\Delta L_T^{\mathrm{emp}}
\approx
0.99496\,\Delta L_T^{\mathrm{pred}}
-0.00019.
\]
This resolves the uniform-high issue for the local theory.
## Interpretation
The fixed-initialization tangent-kernel theory is a local-time theorem. It
should be validated with \(T=O(1)\) small enough that the network remains in the
initial tangent regime.
The long-horizon memorization transition is a different contribution. It cannot
be validated by the same fixed-kernel formula unless we add a time-varying
kernel or explicit alignment-dynamics theory.
|
