1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
|
# Fixed-Width N-Sweep Probe
The previous width-sweep short-SGD plot showed that tangent-kernel predictions
were systematically above empirical FA/BP train gaps. A decomposition showed
the main error source was not FA:
\[
\mathbb E[L_{\mathrm{FA}}^{\mathrm{pred}}-L_{\mathrm{FA}}^{\mathrm{emp}}]
\approx
0.008,
\]
while fixed-kernel BP was too optimistic:
\[
\mathbb E[L_{\mathrm{BP}}^{\mathrm{pred}}-L_{\mathrm{BP}}^{\mathrm{emp}}]
\approx
-0.048.
\]
So the predicted gap was high mainly because the BP baseline was predicted too
low.
## Revised Design
Do not shrink capacity by shrinking width first. That changes two things at
once:
1. the FA capacity margin;
2. the finite-width quality of the BP tangent-kernel approximation.
Instead, keep the MLP width fixed and vary the number of training samples:
\[
P-K_{\mathrm{FA}}-Nn_L.
\]
This changes the capacity margin while keeping finite-width BP bias roughly
controlled.
Probe settings:
- width \(n=64\);
- random-label regression;
- \(N\in\{64,128,192,256,320\}\);
- output dimension \(4\);
- SGD, learning rate \(10^{-3}\), steps \(50\);
- 2 init seeds and 4 feedback seeds per init.
## Result
Combined output:
`outputs/fa_tangent_kernel_fixed_width64_sgd_short_combined/fixed_width64_N_sweep_theory_empirical_overlay.png`
The fixed-width probe improves the calibration:
\[
\operatorname{corr}
\left(
\Delta L^{\mathrm{pred}},
\Delta L^{\mathrm{emp}}
\right)
=
0.983.
\]
\[
\operatorname{MAE}=0.0244.
\]
The linear fit becomes
\[
\Delta L^{\mathrm{emp}}
\approx
0.890\,\Delta L^{\mathrm{pred}}-0.008.
\]
This is much cleaner than the width-sweep fit:
\[
\Delta L^{\mathrm{emp}}
\approx
0.546\,\Delta L^{\mathrm{pred}}+0.044.
\]
The remaining bias is again mostly BP fixed-kernel optimism:
\[
\mathbb E[L_{\mathrm{BP}}^{\mathrm{pred}}-L_{\mathrm{BP}}^{\mathrm{emp}}]
\approx
-0.020,
\]
with a small FA prediction bias:
\[
\mathbb E[L_{\mathrm{FA}}^{\mathrm{pred}}-L_{\mathrm{FA}}^{\mathrm{emp}}]
\approx
0.0045.
\]
## Interpretation
The earlier plot was not the cleanest validation of the theory because the
width axis mixed capacity shrinkage with finite-width NTK error. The fixed-width
N-sweep is the cleaner short-horizon validation design.
This does not yet solve the long-horizon memorization transition. Short-SGD
validates the local tangent-kernel distribution; long-horizon capacity
transition still needs either a time-varying kernel theory or a separate
alignment-dynamics theorem.
|
