path: root/docs/campaign/CASCADE_ABLATION_PLAN.md

# Cascade-EP ablation program — standard multi-layer LLM, EP only in training

**Date opened:** 2026-07-09 · **Trigger:** user directive — product form = standard L-layer
transformer (plain-forward inference); the looped/weight-tied block is demoted to physics testbed.
**Bridge:** layered energy E = Σ_l ½‖z_l − f_l(z_{l−1})‖² over DISTINCT standard blocks.
Free equilibrium == the standard forward pass (E=0) ⟹ inference is a normal LLM forward.
Training = two-phase (±β·CE at the top), relax states to nudged equilibria, ∇θ = (1/2β)[∂E/∂θ|₊ − ∂E/∂θ|₋].
Lineage: predictive-coding≈BP theorem family (Whittington-Bogacz 17; Song+ 20 / Z-IL), EP two-phase readout.
**First gate (2026-07-09):** `cascade_probe.py` L3 C128 random init → cos(cascEP, BP) **0.9968**
(blocks 0.9975/0.9980/0.9992, |EP|/|BP| 0.80–0.91).

## The five claims we are buying evidence for

- **K1 exactness-on-trajectory** — the two-phase gradient matches BP not just at init but along a
  real training trajectory (weights with grown Jacobians stiffen the relaxation).
- **K2 cost** — the nudged relaxation can be engineered to a small multiple of a BP step
  (scheme × K frontier), and the *physical* (Jacobi/parallel) scheme is not hopeless (analog story).
- **K3 training parity** — full training closes to BP final CE at equal arch/steps (the money claim).
- **K4 depth scaling** — no depth penalty vs BP at matched params (signal attenuation under control).
- **K5 analog price** — per-block Jᵀ feedback, dynamic noise, quantization: the tolerance ledger
  ports from the looped-block program; PAR wall applies per block.

Honest cost framing: on GPU cascade-EP is strictly MORE expensive per step than BP (K relax sweeps,
each ≈ one fwd+state-vjp). The value is: standard-form deployment + local rules + analog trainability.
The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether cascade beats that.

---

## STATUS 2026-07-11: K1+K2+K3 SEALED; D-tier in flight
- K1 exactness: cos 0.9998-1.0000 on-trajectory + BP-free formally audited (test_bp_free.py in repo).
- K2 cost: exact mode ~3.6x BP (v7); Sol audit says remaining eager headroom 5-10% (v8 queued).
- K3 quality: **matched-tuning PARITY n=3** (EP-exact 2.0500±0.015 vs BP 2.0530±0.004 @ C256 L6,
  lr 1e-3 both). Arc: fake-win (lr artifact) -> fake-tax (v7 dedups) -> parity. Fast mode = documented
  -4%CE/+20%speed dial. A0.4: TF32 free, bf16 production-only (cos 0.9427).
- D1a (L12xC512 45M): BP s1/s2 SEALED 1.9169/1.9194 (H8, lr1e-3, tok_init0.02, 4000 steps, adamw).
- E-tier: next in queue (softmax pathology / error-channel SNR / write pricing) -> Demo-0 spec sheet.

## D1a AUTOPSY + K-LADDER DIAGNOSTIC (2026-07-09 night)
**>>> CORRECTION (2026-07-10 02:xx): the "parent-death" below was a MISDIAGNOSIS. <<<**
The original D1a arms did NOT die -- they completed normally. When I checked at ~23:44 they were ALIVE
at step 3200 on GPUs 0/3 (both at 100%); my /proc scan was mangled by a zsh eval wrapper so I misread
"no casc alive", and GPU1 being free (11 MiB) fooled me (the runs were on 0/3, not 1). d1_ep_s1.log is
continuous 0->4000 at steady 0.679 it/s (3200->4000 = 19.6 min, matches its 00:06 mtime). **Original
D1a finals: d1_ep_s1 1.9745 / s2 2.0013 / s3 2.1188 (fixed K3; s2 blew at step 4000 skips=9, s3 blew
hard skips=23 governor ramped K->7); d1_ep_muon 2.7515 (Muon-on-EP, cos collapsed 0.82).** The d1b
experiments I launched (thinking the originals died) ran on the GENUINELY-FREE GPU1, so no competition
-- and they independently isolated the real mechanism + fix (below), which is the bigger prize. Net:
no harm, wrong death-story, and we now have BOTH the original un-floored 3-seed AND the beta-floor fix.
KEY read of the original 3-seed: un-floored K3 is HIGH-VARIANCE near the SNR cliff -- s1 got lucky and
stayed stable (1.9745, closest to BP), s2/s3 blew up late. Same-seed non-determinism (fb+autograd
reductions) means the un-floored estimator is not even reproducible near the cliff. That is the
strongest argument FOR the beta-floor (which pins cos=1.0000, stable, reproducible).

**What I ORIGINALLY (wrongly) concluded:** the 4 D1a arms all died at wall-clock 23:36, mid-run,
at a step boundary with NO traceback and NO DONE marker -> classic PARENT-DEATH (launched inline, not
nohup'd; the launching shell/session terminated and took them down). No OOM in journalctl/dmesg. NOT a
training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** (The nohup lesson still
stands as good practice, but it was not the cause here -- there was no death.)
**Interim signal BEFORE they died (the science):** at L12 the EP estimator degrades with training in a
way it did NOT at L6:
  - EP s1: best val 2.0444 @ step 2800, then val BOUNCED to 2.0951 @3200 (last line); cos(EP,BP)
    eroded 1.0000 -> 0.9942 (@2800) -> 0.9897 (@3200) as beta_t adapted DOWN 3e-3 -> 1.9e-5.
  - EP s3: cos fell to 0.9834 AND the quality gate started SKIPPING steps (skips=4).
  - vs BP s1/s2 which finished clean at 1.917. So at step ~3200 EP is ~0.10-0.13 CE above BP and the
    curve is stalling while cos degrades -- the DEPTH-ATTENUATION / estimator-SNR prediction (B6/K4).
**Mechanism hypothesis:** K=3 fb message-passing rounds were tuned at L6xC256; the deeper L12 nudged
equilibrium under-converges, and as beta_t shrinks (nudge -> tiny) the two-phase difference becomes a
small signal against fixed relaxation error -> cos erodes -> gradient quality drops late in training.
**Diagnostic launched (local GPU1, nohup, seed 1, full 4000 steps, H8 lr1e-3 tok_init0.02 beta3e-3):**
  - `d1b_ep_K3_s1` (K=3 control, honest 4000-step reproduction)
  - `d1b_ep_K8_s1` (K=8 = kmax, strongest relaxation -- does more convergence hold cos~1 and close CE?)
  - `d1_bp_s3` relaunch (completes the 3-seed BP reference).
**Decision rule:** if K8 holds cos>=0.999 through step 4000 and reaches ~BP CE -> gap was
under-convergence, fix = scale K with depth, then relaunch full 3-seed at min-sufficient K for the K4
verdict. If K8 does NOT close it -> genuine estimator depth-tax; next arm = beta-floor (needs a code
flag) and/or lambda_l per-layer energy weighting (B4). Follow-on (not yet launched): Muon-on-EP arm.

### RESULT 1 (2026-07-10 00:40): K REFUTED as the lever; BP 3-seed sealed.
- BP 3-seed reference SEALED: 1.9169 / 1.9194 / 1.9214 = **1.9192 +/- 0.0019** (L12 C512 H8).
- **cos is K-INVARIANT.** K3 and K8 track to 4 decimals through step 1200 (both 1.0->0.9997->0.9991)
  AND give identical val CE at every matched step (900: 2.545 vs 2.548; 1100: 2.399 vs 2.404).
  More relaxation rounds do NOTHING -> the cos erosion is NOT fb under-convergence. K8 killed (redundant).
- **Real mechanism = finite-beta SNR collapse.** beta_t = beta0*bscale*(SIG0/sig)^2 collapses ~120x
  (3e-3 -> 2.5e-5) as sig_tok grows 1.6->17.8. The estimator computes E/(NBT*beta_t) from residuals
  (z-o) that are O(beta_t*sig) ~ 4e-4 obtained by subtracting two O(17) states -> catastrophic
  cancellation as beta shrinks AND sig grows. Both worsen with depth. cos erodes 1.0 -> 0.997 (@2000)
  -> 0.98 (@2800 in the dead run). This is a beta-SCHEDULE problem, not a relaxation-depth problem.
- **Fix under test:** added `--beta_floor` / `--beta_fixed` flags. Launched paired arms seed 1
  (control = K3 floor=0, still running): `d1b_ep_bf1e4_s1` (floor 1e-4), `d1b_ep_bf3e4_s1` (floor 3e-4).
  Decision rule: if floored cos stays high through step 2000-2800 and CE drops toward BP 1.919 ->
  beta-floor is the depth fix; pick min-sufficient floor, run 3-seed K4 verdict. Watch drift guard at
  the higher floor (larger nudge). If floors DON'T help -> escalate to double-sided estimator (cancels
  O(beta) Taylor bias, allows large beta, 2x cost) or lambda_l energy weighting.

### RESULT 2 (2026-07-10 01:26): beta-floor CONFIRMED as the depth fix.
Paired seed-1 sweep, cos in the erosion zone (where control collapses):
| arm | cos @2000..4000 | best CE | skips |
|---|---|---|---|
| K3 control (floor 0) | 0.977 -> 0.944 -> **0.896@4000** | 2.0009 | **17** |
| bf1e4 (floor 1e-4) | 0.9996 (nearly flat) | 2.174@2000 (desc) | 0 |
| bf3e4 (floor 3e-4) | **1.0000 flat** | 2.161@2000 (desc) | 0 |
- Flooring beta_t ELIMINATES the erosion: bf3e4 holds cos=1.0000 exactly where the un-floored control
  collapses to 0.896 w/ 17 skips. Higher floor monotonically better CE at matched steps (3e-4 < 1e-4 <
  control). 3e-4 already achieves perfect cos + zero drift/skips -> the operating point (higher can only
  add Taylor bias). The un-floored control still banked best 2.0009 (from ~step 3200 before the late
  collapse), so beta-floor's CE win over 2.0009 is the depth-tax recovery.
- **K4 verdict LAUNCHED:** d1b_ep_bf3e4_s1/s2/s3 (floor 3e-4) 4000 steps vs BP 1.9169/1.9194/1.9214
  (1.9192). If EP 3-seed ~ 1.919 -> **K4 depth-parity SEALED at L12xC512 (real GPT-small shape)** ->
  green-light D1b long-run demo (the "neng kan" gate) + hardware outreach. Poller baqcm84j4 armed.
- FIX SHIPPED to trainer: `--beta_floor` is the depth knob. Recommend it becomes default-on (e.g. 3e-4)
  for L>=12; harmless at L6 (schedule never drops that low there). NOTE for the paper: this is a clean
  "EP as configuration microscope" second instance -- depth exposes a finite-beta SNR floor that BP
  (exact grad, scale-robust) never sees; the floor is the physical-relaxation analog of gradient
  precision. Muon-on-EP arm still pending after the verdict.

## STATUS 2026-07-09 (same day): Tier 0 CLOSED GREEN via the zil scheme; C1 running

- **Naive relaxation FAILS at depth** (the B1-lite sweep): jacobi K=40·L → cos 0.82 (L6) / 0.67 (L12)
  / 0.53 (L24), shrink dying 0.41→0.28; gsf/gsr with small-η+momentum no better; β-insensitive
  (0.01/0.03/0.1 identical) ⟹ binding error = RELAXATION INCOMPLETENESS, not Taylor bias.
  warp2.0 catastrophic (cos 0.11) under naive descent.
- **Two implementation traps found:** (1) NBT-normalized energy made γ=1 actually γ=1/128;
  (2) plain γ=1 reverse sweep WITHOUT interleaved reads contaminates e_l with J_l·δ_{l−1}
  (same β-order as the signal) — final-state readout is directionally ruined (cos 0.30@L6).
- **The fix = zil scheme (interleaved reverse sweep):** update z_l (γ=1, SUM units) then read
  θ_l IMMEDIATELY (e_l = −β·δ_l exact at the feedforward point; δ-recursion has NO linearization
  error). Single phase, β cancels exactly. **Results: cos = 1.0000 at L=6/12/24; io gate 0.9999;
  warp2.0 → 1.0000; real-trajectory ckpts (casc_bp6 s0→s4000) → 0.9998–1.0000. A0.1/A0.2/A0.3 all
  green.** Honest framing: zil is numerically BP restructured as per-layer local two-factor energy
  reads (no global backward graph); the EQUILIBRIUM mode (jacobi/CG to convergence) remains the
  physically-meaningful EP column — priced expensive by the sweep, CG/preconditioning is the B2 job,
  and it is the analog-hardware rung (E-tier).
- **C1 (zil) ran and is RETIRED with zil itself:** casc_ep6 best 3.3236 vs BP twin 2.9746 (gap 0.35
  — single-sided zil top-read carries an O(β) shift on the readout term; moot now).
  **USER DIRECTIVE (2026-07-09 night): zil is NOT the route — it is BP in disguise; the project
  stays on TRUE EP = equilibrium-mode two-phase relaxation.** zil survives only as (a) a diagnostic
  upper bound, (b) optionally a numerical STATE-INIT trick for GPU simulation (`--init_sweep`:
  readout still taken at the relaxed equilibrium = clean EP semantics; hardware needs no init trick
  — physics settles). **Critical path = B2: make the equilibrium solver cheap** (Adam-on-states /
  init-sweep warm start / GS-multi-sweep / λ_l preconditioning), then rerun C1 in equilibrium mode.

## Tier 0 — gate hardening (probe-scale, hours, no training)  → K1

| ID | question | design | decision rule |
|---|---|---|---|
| A0.1 | does cos survive depth? | cos vs L ∈ {3,6,12,24}, C128, Jacobi K auto-scaled; ≥4 batches | cos ≥ 0.98 at L12 or B1 must fix it |
| A0.2 | does cos survive training? | BP-train C256 L6 4k steps saving every 500 (`casc_bp_train.py`); gate at every ckpt; ALSO record required-K to reach res-tol | cos ≥ 0.97 at all ckpts; K growth ≤ 3× init→4k |
| A0.3 | full-θ gate | include emb/pos/readout(tied) grads in the gate | all groups ≥ 0.97 |
| A0.4 | precision | fp32 vs TF32 vs bf16 on the two-phase difference | pick cheapest safe mode (looped-EP lesson: TF32 killed relaxation — re-test here) |

## Tier 1 — relaxation engineering (the cost frontier)  → K2

| ID | axis | arms | metric |
|---|---|---|---|
| B1 | scheme × K | Jacobi (physical, parallel) vs Gauss-Seidel fwd vs GS reverse (algorithmic; Z-IL limit) × K ∈ {12,25,50,100,200,400} at L6 & L12 | K needed for cos ≥ 0.98; wall-clock multiple vs one BP step |
| B2 | state optimizer | GD vs +momentum vs Adam-on-states; η sweep | same |
| B3 | nudge β | {0.003,0.01,0.03,0.1,0.3} × one-sided vs two-sided | cos, shrinkage |EP|/|BP|, required K |
| B4 | energy weighting | raw ℓ₂ vs per-layer precision λ_l=1/RMS² vs LN-in-energy | per-block shrinkage PROFILE (fix the 0.80→0.91 depth attenuation) + relax conditioning |
| B5 | stopping | fixed-K vs relax-to-tol | natural K distribution |
| B6 | **depth attenuation / estimator SNR profile** | measure per-block error amplitude ‖e_l‖ and per-block cos vs depth, as f(L, β, K) | the estimator-precision law: how fast does the deep-layer signal die, and which knob (β, K, λ_l weighting) restores it |

B1 is the single most consequential experiment in the program: if GS-reverse needs K≈L (Z-IL limit)
we have a ~BP-cost algorithmic mode for GPU pretraining, and the Jacobi column is the honest
analog-hardware price. Report all three columns — they are different products.

**Dynamics-vs-estimator tradeoff (user insight, 2026-07-09):** the cascade is dynamically SIMPLER —
the free phase is EXACT (a plain forward; no res/T1/fixed-point error, no Hopf, no collapse), so
**C-tier default arms run with NO regularizers at all** (jr/resreg don't exist here; stability regs
return only if evidence demands). The difficulty MOVES to the estimator: the two-phase difference
must resolve per-layer error signals that ATTENUATE with depth (visible at L=3 already: shrink 0.80
bottom vs 0.91 top), finite-β Taylor bias and finite-K relaxation bias hit the deepest blocks first,
and the difference-of-O(1)-quantities structure makes precision (A0.4, fp32-vs-TF32) bind harder
than in looped-EP. B6 is the dedicated measurement; λ_l weighting (B4), β/K scheduling (B3/B1) and
per-block rebalance (C5) are the candidate antidotes.

## Tier 2 — small full-training ablations (C256 L6 T256 TinyStories, 8–16k steps)  → K3

| ID | arm | vs |
|---|---|---|
| C1 | **money run**: cascade-EP (B-tier winner) ×2–3 seeds | BP twin, same arch/data/AdamW/steps — target gap ≤ 0.05 CE |
| C2 | K budget: {K*, 2K*, 4K*} | CE-vs-cost curve (training may need less relax than the gate does — looped-EP precedent: t2sel 40 trains, 80 gates) |
| C3 | one-sided β (half cost) | two-sided |
| C4 | AdamW | SGDM (shrinkage sensitivity — does 0.8–0.9 amplitude matter under Adam's rescaling?) |
| C5 | shrinkage compensation: none | per-block grad-norm rebalance to BP profile (one-time calibration) |
| C6 | B4-winner energy weighting | raw |

Placement: 1080 farm **after a Pascal canary** (cascade-EP is a new workload class; the Pascal
pathology ban was derived on looped-EP+regs — do a 800-step canary + cross-env fingerprint first).
C256 L6 fits 8 GB (~19M params, ~2-3 GB act).

## Tier 3 — depth/scale rungs (Delta A40 chains)  → K4

| ID | design |
|---|---|
| D1 | **north-star demo re-target**: L12 C512 (≈45M, a real GPT-small shape) cascade-EP vs BP twin — replaces the single-block 33M rung as the flagship demo (task #15) |
| D2 | depth ladder at fixed params: L6/C724 vs L12/C512 vs L24/C362 — depth penalty vs BP? |
| D3 | T 256→512 sanity (relax cost tracks attention; expect no surprise) |

## Tier 4 — analog/hardware arms (port the tolerance machinery)  → K5

| ID | design |
|---|---|
| E1 | Jacobi + per-sweep dynamic noise: does the fnoise ≥1e-3 cliff reappear in cascade relaxation? |
| E2 | Jᵀ ablation: replace J_lᵀe with fixed random Bᵀ (feedback-alignment) / PAR projection — the per-block analog-feasibility tax; FA classically works on shallow stacks, test at L6 |
| E3 | static tolerance: wq8/wq6 weights inside relax |

## Sequencing & fleet

```
now:      A0.1 + A0.3 + B1-lite (shared local GPU, ~1h)  +  casc_bp_train ckpt producer (107 free 1080)
gate ok → B1 full / B2 / B3 / B4 (local A6000s as arms free; each = minutes-hours)
        → Pascal canary → C-tier fan-out on 1080 farm (6 arms × 1-2 days)
        → D1 chains on Delta A40 (queue behind current five lines)
E-tier:   after C1 lands (tolerance scripts port directly)
```

Naming: `casc_*` runs, wandb project **ept-cascade**. Gates report mean over ≥4 batches.
In-flight single-block arms (rescv2, govfloor, fastfull/fastpair, gov_s11-14) continue untouched —
they carry the dynamics paper + the two-stage-recipe science; D1 takes over the DEMO role only.