# C512 "Why slow + plateau" diagnostic campaign (2026-06-17)

Standalone record of the 7-experiment campaign the user proposed to diagnose why EP at C=512 BPE
plateaus at val CE 2.40 (orphan) and never reaches the BP capability band (1.0–1.5). Self-contained;
companion logs in `/tmp/lt_ep/`, general arc in `FINDINGS.md`.

---

## 0. Setup and the user's hypothesis

- **Models** (TinyStories BPE-4096, C=512 H=16 T=256 B=24; random ln4096 = 8.318):
  - EP "orphan" (lr 8e-4, warmup 800, resinit 0.1, holo N=2, t2sel 40, jr_floor 0.1, res_target
    1.5e-3, jr_max 32, pema 0.999): **best val CE 2.4037**, ran full 20k, zero excursions.
  - Standard BP transformer (same C/H/T, mlp=4): **best val CE 1.6953**.
  - Target: BP capability band **1.0–1.5** (BP-C512 itself is 1.70; BP needs C≥1024 to enter the band).

- **User's hypothesis (verbatim intent):** the "full-time slow + plateau" is **not** simple EP
  LR-inequivalence; it is a conflict between the **equilibrium architecture's contractive operating
  point** and **long-context mixing**, with LR merely expressing it as "8e-4 lives, 9e-4 explodes."
  Supporting prior: at a matched tight residual operating point EP ≥ BPTT; the apparent gap is
  EP-tight vs loose-BPTT (non-contractive), and the gap grows with T.

- **Three plateau types the user distinguished:** (1) invalid plateau (res ~1e-2, gate skipping,
  λ pinned); (2) valid-but-over-contracted plateau (res tiny, cos high, gate-skip low, still far
  from BP); (3) controller-fight / estimator-controller mismatch.

- **Priority order requested:** exp3 (BPTT+ctl) → exp2 (LR-sweep) → exp4 (warmup/gate decouple)
  → exp5 (λ grid) → exp6/exp7 (branch + mixing). exp1 (triangulation) foundational.

---

## 1. The seven experiments — spec, what ran, result, read

### exp3 — C512 BPTT+controller (the decisive matched cut)
- **Spec:** replicate the orphan EP recipe (resinit, qk-norm, warmup, jr_floor, res_target, gate,
  pema, lr 8e-4) but swap the EP task gradient for exact BPTT. If it plateaus ~2.2–2.4 → tax is
  architecture/controller; if ~1.9 → EP estimator/noise.
- **Ran:** `--mode bptt`, all else matched, 20k steps, fuse on.
- **Result:** descended cleanly to **best 3.85 @step 1400**, then **destabilized at step 1600**
  (jr→32, res→0.07) and lodged in a broken basin (val 6.27, res 0.068 — just under the 0.1 fuse,
  no abort) for 8400+ steps. Killed at step ~12.8k. **Worse than the EP orphan (3.85 vs 2.40).**
- **Read:** the answer came back neither 2.2–2.4 nor 1.9 — BPTT **broke**. At C512 the contraction
  controller does **not** keep the exact gradient on the manifold; EP's implicit contraction-bias is
  what kept the orphan alive. Inverts the S0 result (BPTT+ctl 1.635 < EP 1.676). *Caveat: single
  lr/seed; BPTT might be stable at another lr.*

### exp2 — one-step loss-decrease LR-sweep at the plateau ckpt
- **Spec:** fixed ckpt + batch, gradients {BPTT, EP-task, EP-task+jac}, one-step update over an lr
  grid, same-graph eval ΔCE. Answers "is it LR-inequivalence."
- **Result** (ΔCE, negative = better): BPTT best **−0.161** (lr 1e-2); EP best **−0.042** (lr 1e-4),
  **diverges for lr ≥ 3e-4**. cos(g_EP,g_BPTT) 0.107, k=|g_EP|/|g_BPTT| ≈ 486.
- **Read:** **not LR.** EP's best one-step descent over *all* lr is 4× below BPTT's; a magnitude
  rescale can't fix a 0.10-aligned direction.

### exp1 — gradient triangulation, multi-checkpoint (foundational)
- **Spec:** at step 0/200/800/2000/plateau, per-group cos of {g_EP, g_BPTT-150, g_BPTT-400},
  norm-ratio k, batch-to-batch variance, + res.
- **Result:**

  | ckpt | val | res | EP·BPTT150 | BPTT150·400 | k | EP self-cos | BPTT self-cos |
  |---|---|---|---|---|---|---|---|
  | step0 (random) | 9.90 | 2.1e-2 | 0.261 | 0.459 | 0.319 | +0.654 | +0.617 |
  | step200 | 6.07 | 4.2e-9 | **0.989** | 1.000 | 1.000 | +0.371 | +0.376 |
  | step800 | 4.11 | 3.9e-4 | 0.964 | 0.935 | 0.844 | +0.755 | +0.721 |
  | step2000 | 3.56 | 2.9e-4 | 0.885 | 0.845 | 0.675 | +0.445 | +0.358 |
  | **plateau (2.40)** | 2.40 | 2.6e-5 | **−0.045** | **1.000** | **4223** | **−0.273** | **+0.957** |

- **Read (decisive):** the EP estimator is **faithful while descending** (cos 0.99→0.89, steps
  200–2000) and undergoes an **SNR/coherence collapse at the optimum** — cos→0, k→4000, and crucially
  the **batch self-cos goes −0.27** (EP gradients on different batches anti-correlate) while BPTT is
  +0.96. BPTT-150≡BPTT-400 (cos 1.00) the entire time ⇒ the true gradient is always well-defined.
  **The 2.40 plateau is an EP estimator bias-floor / batch-incoherence near the optimum** — not
  horizon ambiguity (ruled out here), not LR (exp2).

### exp4 — warmup / validity-gate decouple (3 arms)
- **Spec:** armA current warmup; armB full-strength contraction early + task LR still warmup;
  armC no-warmup + smaller resinit / lower branch LR. Metric: gate-skip / res-under-gate / early val.
- **Result:** armA (warmup): res stuck ~1.5e-2 for ~50 steps → gate skips the nudge → **no task
  learning for ~50–60 steps** (cos≈0, the reported grad is pure jacreg); cos jumps to 0.99 once
  res<gate by step ~60; best 4.11@800. armB (`cprewarm 200`): res 1.7e-2 after the pre-phase, dead-
  zone only **partially** mitigated (val 5.97 vs 6.07 @100, best 4.13@600). armC (no-warmup +
  resinit 0.03): **no dead-zone** (cos 0.99 from step 20, best 4.03@300) but a violent excursion
  (res→0.14 @400, recovered), ended 4.71.
- **Read:** the warmup×gate dead-zone is real; establishing contraction once (armB) isn't enough —
  warmup's tiny lr can't deepen it. Smallest-resinit + no-warmup (armC) learns fastest early but is
  excursion-prone.

### exp5 — λ floor / target / jr_max grid (warm-start from 2.40)
- **Spec:** floor {0.1,0.05,0.03} × target {1.5e-3,3e-3,5e-3} × jr_max {4,8,16,32}, 1–2k steps, fuse on.
- **Result (best val CE, 1200 warm steps):** jr_max 4 → 2.376, **jr_max 8 → 2.3648**, jr_max 16 →
  2.416, jr_max 32 (base) → 2.463; floor 0.05/target1.5e-3 → 2.477; floor 0.03/target5e-3 → 2.482.
  (orphan = 2.404.)
- **Read:** **jr_max ≤ 8 beats 32 by ~0.04** and dips just below the orphan — the jr_max ceiling was
  a small tax; floor/target relaxation does not help. Still far from 1–1.5: the estimator floor
  dominates.

### exp6 — branch-growth + attention entropy + qk-RMS (incl. at plateau)
- **Spec:** log |attn|/|xin|, |ffn|/|xin|, |WO|, |pj|, |fc|, attention entropy, qk RMS, logit std;
  if branch contribution is small at the plateau, add trainable residual gates α with larger LR.
- **Result:** training (resinit 0.1): |WO| 1.95→4.91, |pj|→6.57, attn/xin 56→3.5, attn entropy
  4.56→3.88, qk_rms pinned 1.0. **At the plateau:** |pj|=44, |fc|=58, attn/xin≈1.0, attn entropy
  3.62, qk_rms 1.0.
- **Read:** branches grow fine and are **large** at the plateau ⇒ **no "stuck-small branch"** — the
  trainable-α-gate intervention is **not indicated**. qk-norm is bounding qk RMS (1.0) as designed.

### exp7 — mixing / influence length + per-position CE (EP vs BP)
- **Spec:** finite-diff |∂z*_t/∂x_{t−d}| vs distance + per-position CE, comparing tight-EP,
  BPTT-ctl-tight, BPTT-ctl-loose, standard BP.
- **Bug caught & fixed:** the EP probe first embedded the input with random-init weights (before
  loading the ckpt) → garbage (per-pos CE ~7); fixed to embed post-load and re-ran.
- **Result — influence ||Δ·_q||/δ vs distance d:**

  | d | 1 | 8 | 16 | 64 | 200 |
  |---|---|---|---|---|---|
  | BP | 5.34 | 1.81 | 0.88 | 0.15 | 0.003 |
  | EP-orphan | 0.238 | 0.102 | 0.071 | 0.019 | 0.006 |
  | BP ÷d1 | 1.0 | 0.34 | 0.16 | 0.029 | 0.0006 |
  | EP ÷d1 | 1.0 | 0.43 | 0.30 | 0.080 | 0.025 |

  Per-position CE: BP uniformly ~0.7–1.0 lower than EP at every position (no long-range-specific gap).
- **Read:** both decay with distance; EP ~20× weaker in **absolute** coupling (scale-caveated — z*
  vs hidden norm), but **normalized, EP reaches farther than BP**. ⇒ **does NOT support — arguably
  refutes — "contraction cuts long-range mixing."** EP's deficit is **uniform across position**,
  consistent with the estimator floor (exp1), not a mixing-length deficit.
- **Honest gap:** a stable BPTT-ctl-tight ckpt does not exist (BPTT broke, exp3), so the cross-method
  comparison is **EP-vs-BP only**, not the full tight/loose/BP set requested.

---

## 2. Consolidated verdict (data-level)

**The C512 2.40 plateau is an EP estimator SNR/coherence collapse near the optimum** (exp1: cos→0,
batch self-cos −0.27, k→4000, true gradient clean). Ruled out as causes: LR (exp2), horizon
ambiguity (exp1), λ floor/target (exp5), mixing-length / contraction-cut (exp7). The architecture
has a **separate** wall: exact-gradient BPTT can't stay stable at C512 (exp3 broke at 1600). EP's
contraction-bias makes it the more robust of the two here.

Of the user's three plateau types, the orphan is type (2) **valid-but-over-contracted** — but the
sharper statement is that "over-contracted" is not the mechanism; the binding constraint is the
**EP estimator bias floor**, which becomes batch-incoherent once the true gradient shrinks below it.

**Two levers the data points at (decision pending):**
- (a) **Reduce the EP estimator bias floor near the optimum** — N=4 holo / tracking-AEP / lock-in /
  navg averaging; jr_max≤8 also buys ~0.04 (exp5). The lever for EP's own plateau.
- (b) **The architecture's stability margin for exact gradients** — the separate exp3 wall.

---

## 3. Code / logs

- Trainer flags added this round: `--snap` (exp1 ckpts), `--cprewarm` (exp4 armB), `--probe_branch`
  (exp6, now with attn entropy + qk RMS), `--probe_bptt` (cos/k along trajectory).
- Scripts: `lr_sweep.py` (exp2), `triangulation.py` (exp1, +batch self-cos), `mixing_probe.py`
  (exp7 EP, bug-fixed), `bp_mixing.py` (exp7 BP), `bp_charlm.py --ckpt` (BP ckpt save).
- Orchestration: `master.sh` (first pass), `master2.sh` (full to-spec versions).
- Logs: `triangulation_multi.log`, `exp4_armB.log`/`exp4_armC.log`, `lam_jr{4,8,16}.log`,
  `exp6_train.log`/`exp6_plateau.log`, `bp_mixing.log`, `mixing2.log` (+ the corrected re-run).
- Checkpoints: `ep_C512_bpe.best.pt` (orphan 2.40), `bp_C512.pt` (BP 1.70), `snap_{0,200,800,2000}.pt`,
  `lam_jr8.pt` (2.365), `lam_t5.pt`.