1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
# Closed-Form Soft-Ramp Gap Law
> **Correction (2026-06-09, note 37):** the verification table below was
> computed on mismatched data/init seeds. Matched rerun: corr(log,log) =
> **0.933** (matched single init) / **0.982** (5-init geometric mean);
> `corr(lam_min_mean, -log gap) = 0.987`. Use these numbers, not 0.977.
This note gives a closed-form burden that quantifies the finite-time FA/BP train
gap and explains the empirical soft ramp (notes 22, 25) as a smooth spectral
phenomenon rather than a phase transition. It connects the exact initial-erosion
theorem (note 29) to the finite-time gap and sits one level below the early
operator-velocity estimator (notes 15-17): the closed form gives the *mechanism
and shape*; the estimator supplies the drift-corrected *magnitude*.
## Assumptions
**A1 (lazy / frozen tangent operators).** Over the horizon the tangent operators
are approximately frozen at initialization, so for squared loss
`L = ||r||^2/(2N)` the BP residual evolves as
```text
r_t^BP = (I - eta K_BP / N)^t r_0, K_BP = J J^T.
```
This is the standard lazy/NTK approximation. Operators actually drift; that
residual is exactly the `K_s - K_0` the estimator corrects (notes 13, 15).
**A2 (scalar erosion model).** The FA tangent operator `K_FA = J J_tilde^T` is
non-symmetric; only its symmetric part `S_FA = (K_FA + K_FA^T)/2` drives the
first-order loss change. We model
```text
S_FA ~= rho K_BP, rho = output_share = r^T K_out r / r^T K_BP r in (0,1).
```
This is *exact in the residual direction* by the initial-moment theorem
(note 29): `r^T E_B[S_FA] r = ||g_out^BP||^2 = r^T K_out r = rho * r^T K_BP r`,
i.e. `rho = 1 - E_B[e_0]`. Treating `S_FA` as a global scalar multiple of
`K_BP` extends this identity to all eigenmodes and is the one approximation
beyond A1.
## Theorem (closed-form FA/BP gap)
Under A1-A2, in the BP eigenbasis `K_BP v_i = lam_i v_i` with `c_i = v_i^T r_0`,
```text
gap_T = L_FA(T) - L_BP(T)
= (1/2N) sum_i c_i^2 [ (1 - eta rho lam_i / N)^{2T}
- (1 - eta lam_i / N)^{2T} ].
```
**Proof.** `L_rule(T) = ||r_T^rule||^2/(2N)`. Expanding `r_0 = sum_i c_i v_i`,
```text
r_T^BP = sum_i v_i (1 - eta lam_i/N)^T c_i
=> ||r_T^BP||^2 = sum_i c_i^2 (1 - eta lam_i/N)^{2T}.
```
Under A2, `S_FA = rho K_BP` shares eigenvectors `v_i` with eigenvalues
`rho lam_i`, so
```text
r_T^FA = sum_i v_i (1 - eta rho lam_i/N)^T c_i
=> ||r_T^FA||^2 = sum_i c_i^2 (1 - eta rho lam_i/N)^{2T}.
```
Subtract and divide by `2N`. ∎
## Corollary 1 (soft, not hard)
Because `rho < 1`, every mode decays no faster under FA than under BP, so each
bracket is `>= 0` and `gap_T >= 0`. `gap_T` is real-analytic in `(rho, {lam_i},
T)`. Therefore under any smooth deformation of the spectrum (e.g. width or
capacity margin) the gap moves smoothly: it is a **soft ramp**. A discontinuous
transition would require `rho -> 0` (FA fully stalls), which never happens since
`rho >= output_share > 0`. The hard-margin threshold
`Delta d_hard = max(0, k - (P - d))` is recovered only in that degenerate limit.
This is the operator-level version of the "no free capacity" statement
(note 26): nonzero `rho < 1` gives a positive contribution from every mode at
every margin.
## Corollary 2 (the ramp law: where it comes from)
Let `tau_i = N / (eta lam_i)`.
- **Underparameterized side** (many `tau_i >~ T`): each bracket
`~= 2 eta T (1 - rho) lam_i / N`, so
```text
gap_T ~= (1 - rho) (eta T / N^2) sum_{slow i} c_i^2 lam_i,
```
residual-mass limited; large at negative margin (ill-conditioned spectrum,
residual stuck in slow modes).
- **Converged side** (BP fit, `L_BP(T) ~= 0`): the gap is the slowest surviving
FA mode,
```text
gap_T ~= (c_min^2 / 2N) exp(-2 eta rho lam_min T / N),
i.e. log gap_T ~= const - (2 eta T / N) rho lam_min.
```
The soft ramp is therefore driven by `lam_min(w)` (smallest BP-NTK eigenvalue)
growing smoothly with capacity; the two regimes cross over smoothly.
## Verification (no fit)
Script: `scripts/closed_form_soft_ramp.py`.
Figure: `outputs/closed_form_soft_ramp/closed_form_vs_measured_ramp.png`.
Rows: `outputs/closed_form_soft_ramp/closed_form_vs_measured.csv`.
Frozen-initialization closed form (`rho`, `lam_i`, `c_i` all measured at init,
no fitted parameters) versus the dense `T=30000` ramp (note 22):
| quantity | value |
|---|---|
| `corr(log pred, log measured)` across `w=20..40` | **0.977** |
| measured `log(gap) ~ -0.021*margin`, fit `R^2` | **0.993** (log-linear ramp) |
| `lam_min(w)` over the sweep | 0.07 -> 0.99 |
| `corr(lam_min, -log measured)` | **0.963** (ramp driven by `lam_min`) |
## Scope / honesty
- Frozen-init captures the ramp **shape** (corr 0.977) but **compresses the
range**: it under-predicts the large-gap (negative-margin) end and
over-predicts the small-gap (positive-margin) tail. That residual is operator
drift `K_s - K_0`; the early-velocity estimator (notes 15-17) corrects it and
supplies the quantitative magnitude. Closed form = mechanism + shape;
estimator = number.
- A2 is exact only in the residual direction; a per-mode `rho_i` (spectral
erosion) refinement is the natural next step.
- Lazy regime, full-batch SGD, synthetic random-label MLP scope, as elsewhere.
## Role in the paper
This is the quantitative core of contribution 2 (soft erosion). It converts the
exact scalar burden `rho = 1 - E_B[e_0]` (note 29) into a finite-time gap and
proves the soft ramp is a spectral consequence of `rho < 1` plus a smoothly
growing `lam_min`, not a redundancy-exhaustion threshold. Pairs with: note 29
(supplies `rho`), notes 15-17 (drift-corrected magnitude), notes 22/25 (the
empirical ramp it explains).
|
