# Closed-Form Soft-Ramp Gap Law

> **Correction (2026-06-09, note 37):** the verification table below was
> computed on mismatched data/init seeds. Matched rerun: corr(log,log) =
> **0.933** (matched single init) / **0.982** (5-init geometric mean);
> `corr(lam_min_mean, -log gap) = 0.987`. Use these numbers, not 0.977.

This note gives a closed-form burden that quantifies the finite-time FA/BP train
gap and explains the empirical soft ramp (notes 22, 25) as a smooth spectral
phenomenon rather than a phase transition. It connects the exact initial-erosion
theorem (note 29) to the finite-time gap and sits one level below the early
operator-velocity estimator (notes 15-17): the closed form gives the *mechanism
and shape*; the estimator supplies the drift-corrected *magnitude*.

## Assumptions

**A1 (lazy / frozen tangent operators).** Over the horizon the tangent operators
are approximately frozen at initialization, so for squared loss
`L = ||r||^2/(2N)` the BP residual evolves as

```text
r_t^BP = (I - eta K_BP / N)^t r_0,    K_BP = J J^T.
```

This is the standard lazy/NTK approximation. Operators actually drift; that
residual is exactly the `K_s - K_0` the estimator corrects (notes 13, 15).

**A2 (scalar erosion model).** The FA tangent operator `K_FA = J J_tilde^T` is
non-symmetric; only its symmetric part `S_FA = (K_FA + K_FA^T)/2` drives the
first-order loss change. We model

```text
S_FA ~= rho K_BP,    rho = output_share = r^T K_out r / r^T K_BP r in (0,1).
```

This is *exact in the residual direction* by the initial-moment theorem
(note 29): `r^T E_B[S_FA] r = ||g_out^BP||^2 = r^T K_out r = rho * r^T K_BP r`,
i.e. `rho = 1 - E_B[e_0]`. Treating `S_FA` as a global scalar multiple of
`K_BP` extends this identity to all eigenmodes and is the one approximation
beyond A1.

## Theorem (closed-form FA/BP gap)

Under A1-A2, in the BP eigenbasis `K_BP v_i = lam_i v_i` with `c_i = v_i^T r_0`,

```text
gap_T = L_FA(T) - L_BP(T)
      = (1/2N) sum_i c_i^2 [ (1 - eta rho lam_i / N)^{2T}
                           - (1 - eta lam_i / N)^{2T} ].
```

**Proof.** `L_rule(T) = ||r_T^rule||^2/(2N)`. Expanding `r_0 = sum_i c_i v_i`,

```text
r_T^BP = sum_i v_i (1 - eta lam_i/N)^T c_i
=> ||r_T^BP||^2 = sum_i c_i^2 (1 - eta lam_i/N)^{2T}.
```

Under A2, `S_FA = rho K_BP` shares eigenvectors `v_i` with eigenvalues
`rho lam_i`, so

```text
r_T^FA = sum_i v_i (1 - eta rho lam_i/N)^T c_i
=> ||r_T^FA||^2 = sum_i c_i^2 (1 - eta rho lam_i/N)^{2T}.
```

Subtract and divide by `2N`. ∎

## Corollary 1 (soft, not hard)

Because `rho < 1`, every mode decays no faster under FA than under BP, so each
bracket is `>= 0` and `gap_T >= 0`. `gap_T` is real-analytic in `(rho, {lam_i},
T)`. Therefore under any smooth deformation of the spectrum (e.g. width or
capacity margin) the gap moves smoothly: it is a **soft ramp**. A discontinuous
transition would require `rho -> 0` (FA fully stalls), which never happens since
`rho >= output_share > 0`. The hard-margin threshold
`Delta d_hard = max(0, k - (P - d))` is recovered only in that degenerate limit.

This is the operator-level version of the "no free capacity" statement
(note 26): nonzero `rho < 1` gives a positive contribution from every mode at
every margin.

## Corollary 2 (the ramp law: where it comes from)

Let `tau_i = N / (eta lam_i)`.

- **Underparameterized side** (many `tau_i >~ T`): each bracket
  `~= 2 eta T (1 - rho) lam_i / N`, so

  ```text
  gap_T ~= (1 - rho) (eta T / N^2) sum_{slow i} c_i^2 lam_i,
  ```

  residual-mass limited; large at negative margin (ill-conditioned spectrum,
  residual stuck in slow modes).

- **Converged side** (BP fit, `L_BP(T) ~= 0`): the gap is the slowest surviving
  FA mode,

  ```text
  gap_T ~= (c_min^2 / 2N) exp(-2 eta rho lam_min T / N),
  i.e.  log gap_T ~= const - (2 eta T / N) rho lam_min.
  ```

The soft ramp is therefore driven by `lam_min(w)` (smallest BP-NTK eigenvalue)
growing smoothly with capacity; the two regimes cross over smoothly.

## Verification (no fit)

Script: `scripts/closed_form_soft_ramp.py`.
Figure: `outputs/closed_form_soft_ramp/closed_form_vs_measured_ramp.png`.
Rows: `outputs/closed_form_soft_ramp/closed_form_vs_measured.csv`.

Frozen-initialization closed form (`rho`, `lam_i`, `c_i` all measured at init,
no fitted parameters) versus the dense `T=30000` ramp (note 22):

| quantity | value |
|---|---|
| `corr(log pred, log measured)` across `w=20..40` | **0.977** |
| measured `log(gap) ~ -0.021*margin`, fit `R^2` | **0.993** (log-linear ramp) |
| `lam_min(w)` over the sweep | 0.07 -> 0.99 |
| `corr(lam_min, -log measured)` | **0.963** (ramp driven by `lam_min`) |

## Scope / honesty

- Frozen-init captures the ramp **shape** (corr 0.977) but **compresses the
  range**: it under-predicts the large-gap (negative-margin) end and
  over-predicts the small-gap (positive-margin) tail. That residual is operator
  drift `K_s - K_0`; the early-velocity estimator (notes 15-17) corrects it and
  supplies the quantitative magnitude. Closed form = mechanism + shape;
  estimator = number.
- A2 is exact only in the residual direction; a per-mode `rho_i` (spectral
  erosion) refinement is the natural next step.
- Lazy regime, full-batch SGD, synthetic random-label MLP scope, as elsewhere.

## Role in the paper

This is the quantitative core of contribution 2 (soft erosion). It converts the
exact scalar burden `rho = 1 - E_B[e_0]` (note 29) into a finite-time gap and
proves the soft ramp is a spectral consequence of `rho < 1` plus a smoothly
growing `lam_min`, not a redundancy-exhaustion threshold. Pairs with: note 29
(supplies `rho`), notes 15-17 (drift-corrected magnitude), notes 22/25 (the
empirical ramp it explains).