path: root/docs/campaign/CASCADE_ABLATION_PLAN.md

# Cascade-EP ablation program — standard multi-layer LLM, EP only in training

**Date opened:** 2026-07-09 · **Trigger:** user directive — product form = standard L-layer
transformer (plain-forward inference); the looped/weight-tied block is demoted to physics testbed.
**Bridge:** layered energy E = Σ_l ½‖z_l − f_l(z_{l−1})‖² over DISTINCT standard blocks.
Free equilibrium == the standard forward pass (E=0) ⟹ inference is a normal LLM forward.
Training = two-phase (±β·CE at the top), relax states to nudged equilibria, ∇θ = (1/2β)[∂E/∂θ|₊ − ∂E/∂θ|₋].
Lineage: predictive-coding≈BP theorem family (Whittington-Bogacz 17; Song+ 20 / Z-IL), EP two-phase readout.
**First gate (2026-07-09):** `cascade_probe.py` L3 C128 random init → cos(cascEP, BP) **0.9968**
(blocks 0.9975/0.9980/0.9992, |EP|/|BP| 0.80–0.91).

## The five claims we are buying evidence for

- **K1 exactness-on-trajectory** — the two-phase gradient matches BP not just at init but along a
  real training trajectory (weights with grown Jacobians stiffen the relaxation).
- **K2 cost** — the nudged relaxation can be engineered to a small multiple of a BP step
  (scheme × K frontier), and the *physical* (Jacobi/parallel) scheme is not hopeless (analog story).
- **K3 training parity** — full training closes to BP final CE at equal arch/steps (the money claim).
- **K4 depth scaling** — no depth penalty vs BP at matched params (signal attenuation under control).
- **K5 analog price** — per-block Jᵀ feedback, dynamic noise, quantization: the tolerance ledger
  ports from the looped-block program; PAR wall applies per block.

Honest cost framing: on GPU cascade-EP is strictly MORE expensive per step than BP (K relax sweeps,
each ≈ one fwd+state-vjp). The value is: standard-form deployment + local rules + analog trainability.
The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether cascade beats that.

---

## STATUS 2026-07-11: K1+K2+K3 SEALED; D-tier in flight
- K1 exactness: cos 0.9998-1.0000 on-trajectory + BP-free formally audited (test_bp_free.py in repo).
- K2 cost: exact mode ~3.6x BP (v7); Sol audit says remaining eager headroom 5-10% (v8 queued).
- K3 quality: **matched-tuning PARITY n=3** (EP-exact 2.0500±0.015 vs BP 2.0530±0.004 @ C256 L6,
  lr 1e-3 both). Arc: fake-win (lr artifact) -> fake-tax (v7 dedups) -> parity. Fast mode = documented
  -4%CE/+20%speed dial. A0.4: TF32 free, bf16 production-only (cos 0.9427).
- D1a (L12xC512 45M): BP s1/s2 SEALED 1.9169/1.9194 (H8, lr1e-3, tok_init0.02, 4000 steps, adamw).
- E-tier: next in queue (softmax pathology / error-channel SNR / write pricing) -> Demo-0 spec sheet.

## D1a AUTOPSY + K-LADDER DIAGNOSTIC (2026-07-09 night)
**>>> CORRECTION (2026-07-10 02:xx): the "parent-death" below was a MISDIAGNOSIS. <<<**
The original D1a arms did NOT die -- they completed normally. When I checked at ~23:44 they were ALIVE
at step 3200 on GPUs 0/3 (both at 100%); my /proc scan was mangled by a zsh eval wrapper so I misread
"no casc alive", and GPU1 being free (11 MiB) fooled me (the runs were on 0/3, not 1). d1_ep_s1.log is
continuous 0->4000 at steady 0.679 it/s (3200->4000 = 19.6 min, matches its 00:06 mtime). **Original
D1a finals: d1_ep_s1 1.9745 / s2 2.0013 / s3 2.1188 (fixed K3; s2 blew at step 4000 skips=9, s3 blew
hard skips=23 governor ramped K->7); d1_ep_muon 2.7515 (Muon-on-EP, cos collapsed 0.82).** The d1b
experiments I launched (thinking the originals died) ran on the GENUINELY-FREE GPU1, so no competition
-- and they independently isolated the real mechanism + fix (below), which is the bigger prize. Net:
no harm, wrong death-story, and we now have BOTH the original un-floored 3-seed AND the beta-floor fix.
KEY read of the original 3-seed: un-floored K3 is HIGH-VARIANCE near the SNR cliff -- s1 got lucky and
stayed stable (1.9745, closest to BP), s2/s3 blew up late. Same-seed non-determinism (fb+autograd
reductions) means the un-floored estimator is not even reproducible near the cliff. That is the
strongest argument FOR the beta-floor (which pins cos=1.0000, stable, reproducible).

**What I ORIGINALLY (wrongly) concluded:** the 4 D1a arms all died at wall-clock 23:36, mid-run,
at a step boundary with NO traceback and NO DONE marker -> classic PARENT-DEATH (launched inline, not
nohup'd; the launching shell/session terminated and took them down). No OOM in journalctl/dmesg. NOT a
training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** (The nohup lesson still
stands as good practice, but it was not the cause here -- there was no death.)
**Interim signal BEFORE they died (the science):** at L12 the EP estimator degrades with training in a
way it did NOT at L6:
  - EP s1: best val 2.0444 @ step 2800, then val BOUNCED to 2.0951 @3200 (last line); cos(EP,BP)
    eroded 1.0000 -> 0.9942 (@2800) -> 0.9897 (@3200) as beta_t adapted DOWN 3e-3 -> 1.9e-5.
  - EP s3: cos fell to 0.9834 AND the quality gate started SKIPPING steps (skips=4).
  - vs BP s1/s2 which finished clean at 1.917. So at step ~3200 EP is ~0.10-0.13 CE above BP and the
    curve is stalling while cos degrades -- the DEPTH-ATTENUATION / estimator-SNR prediction (B6/K4).
**Mechanism hypothesis:** K=3 fb message-passing rounds were tuned at L6xC256; the deeper L12 nudged
equilibrium under-converges, and as beta_t shrinks (nudge -> tiny) the two-phase difference becomes a
small signal against fixed relaxation error -> cos erodes -> gradient quality drops late in training.
**Diagnostic launched (local GPU1, nohup, seed 1, full 4000 steps, H8 lr1e-3 tok_init0.02 beta3e-3):**
  - `d1b_ep_K3_s1` (K=3 control, honest 4000-step reproduction)
  - `d1b_ep_K8_s1` (K=8 = kmax, strongest relaxation -- does more convergence hold cos~1 and close CE?)
  - `d1_bp_s3` relaunch (completes the 3-seed BP reference).
**Decision rule:** if K8 holds cos>=0.999 through step 4000 and reaches ~BP CE -> gap was
under-convergence, fix = scale K with depth, then relaunch full 3-seed at min-sufficient K for the K4
verdict. If K8 does NOT close it -> genuine estimator depth-tax; next arm = beta-floor (needs a code
flag) and/or lambda_l per-layer energy weighting (B4). Follow-on (not yet launched): Muon-on-EP arm.

### RESULT 1 (2026-07-10 00:40): K REFUTED as the lever; BP 3-seed sealed.
- BP 3-seed reference SEALED: 1.9169 / 1.9194 / 1.9214 = **1.9192 +/- 0.0019** (L12 C512 H8).
- **cos is K-INVARIANT.** K3 and K8 track to 4 decimals through step 1200 (both 1.0->0.9997->0.9991)
  AND give identical val CE at every matched step (900: 2.545 vs 2.548; 1100: 2.399 vs 2.404).
  More relaxation rounds do NOTHING -> the cos erosion is NOT fb under-convergence. K8 killed (redundant).
- **Real mechanism = finite-beta SNR collapse.** beta_t = beta0*bscale*(SIG0/sig)^2 collapses ~120x
  (3e-3 -> 2.5e-5) as sig_tok grows 1.6->17.8. The estimator computes E/(NBT*beta_t) from residuals
  (z-o) that are O(beta_t*sig) ~ 4e-4 obtained by subtracting two O(17) states -> catastrophic
  cancellation as beta shrinks AND sig grows. Both worsen with depth. cos erodes 1.0 -> 0.997 (@2000)
  -> 0.98 (@2800 in the dead run). This is a beta-SCHEDULE problem, not a relaxation-depth problem.
- **Fix under test:** added `--beta_floor` / `--beta_fixed` flags. Launched paired arms seed 1
  (control = K3 floor=0, still running): `d1b_ep_bf1e4_s1` (floor 1e-4), `d1b_ep_bf3e4_s1` (floor 3e-4).
  Decision rule: if floored cos stays high through step 2000-2800 and CE drops toward BP 1.919 ->
  beta-floor is the depth fix; pick min-sufficient floor, run 3-seed K4 verdict. Watch drift guard at
  the higher floor (larger nudge). If floors DON'T help -> escalate to double-sided estimator (cancels
  O(beta) Taylor bias, allows large beta, 2x cost) or lambda_l energy weighting.

### RESULT 2 (2026-07-10 01:26): beta-floor CONFIRMED as the depth fix.
Paired seed-1 sweep, cos in the erosion zone (where control collapses):
| arm | cos @2000..4000 | best CE | skips |
|---|---|---|---|
| K3 control (floor 0) | 0.977 -> 0.944 -> **0.896@4000** | 2.0009 | **17** |
| bf1e4 (floor 1e-4) | 0.9996 (nearly flat) | 2.174@2000 (desc) | 0 |
| bf3e4 (floor 3e-4) | **1.0000 flat** | 2.161@2000 (desc) | 0 |
- Flooring beta_t ELIMINATES the erosion: bf3e4 holds cos=1.0000 exactly where the un-floored control
  collapses to 0.896 w/ 17 skips. Higher floor monotonically better CE at matched steps (3e-4 < 1e-4 <
  control). 3e-4 already achieves perfect cos + zero drift/skips -> the operating point (higher can only
  add Taylor bias). The un-floored control still banked best 2.0009 (from ~step 3200 before the late
  collapse), so beta-floor's CE win over 2.0009 is the depth-tax recovery.
- **K4 verdict LAUNCHED:** d1b_ep_bf3e4_s1/s2/s3 (floor 3e-4) 4000 steps vs BP 1.9169/1.9194/1.9214
  (1.9192). If EP 3-seed ~ 1.919 -> **K4 depth-parity SEALED at L12xC512 (real GPT-small shape)** ->
  green-light D1b long-run demo (the "neng kan" gate) + hardware outreach. Poller baqcm84j4 armed.
- FIX SHIPPED to trainer: `--beta_floor` is the depth knob. Recommend it becomes default-on (e.g. 3e-4)
  for L>=12; harmless at L6 (schedule never drops that low there). NOTE for the paper: this is a clean
  "EP as configuration microscope" second instance -- depth exposes a finite-beta SNR floor that BP
  (exact grad, scale-robust) never sees; the floor is the physical-relaxation analog of gradient
  precision. Muon-on-EP arm still pending after the verdict.

## STATUS 2026-07-09 (same day): Tier 0 CLOSED GREEN via the zil scheme; C1 running

- **Naive relaxation FAILS at depth** (the B1-lite sweep): jacobi K=40·L → cos 0.82 (L6) / 0.67 (L12)
  / 0.53 (L24), shrink dying 0.41→0.28; gsf/gsr with small-η+momentum no better; β-insensitive
  (0.01/0.03/0.1 identical) ⟹ binding error = RELAXATION INCOMPLETENESS, not Taylor bias.
  warp2.0 catastrophic (cos 0.11) under naive descent.
- **Two implementation traps found:** (1) NBT-normalized energy made γ=1 actually γ=1/128;
  (2) plain γ=1 reverse sweep WITHOUT interleaved reads contaminates e_l with J_l·δ_{l−1}
  (same β-order as the signal) — final-state readout is directionally ruined (cos 0.30@L6).
- **The fix = zil scheme (interleaved reverse sweep):** update z_l (γ=1, SUM units) then read
  θ_l IMMEDIATELY (e_l = −β·δ_l exact at the feedforward point; δ-recursion has NO linearization
  error). Single phase, β cancels exactly. **Results: cos = 1.0000 at L=6/12/24; io gate 0.9999;
  warp2.0 → 1.0000; real-trajectory ckpts (casc_bp6 s0→s4000) → 0.9998–1.0000. A0.1/A0.2/A0.3 all
  green.** Honest framing: zil is numerically BP restructured as per-layer local two-factor energy
  reads (no global backward graph); the EQUILIBRIUM mode (jacobi/CG to convergence) remains the
  physically-meaningful EP column — priced expensive by the sweep, CG/preconditioning is the B2 job,
  and it is the analog-hardware rung (E-tier).
- **C1 (zil) ran and is RETIRED with zil itself:** casc_ep6 best 3.3236 vs BP twin 2.9746 (gap 0.35
  — single-sided zil top-read carries an O(β) shift on the readout term; moot now).
  **USER DIRECTIVE (2026-07-09 night): zil is NOT the route — it is BP in disguise; the project
  stays on TRUE EP = equilibrium-mode two-phase relaxation.** zil survives only as (a) a diagnostic
  upper bound, (b) optionally a numerical STATE-INIT trick for GPU simulation (`--init_sweep`:
  readout still taken at the relaxed equilibrium = clean EP semantics; hardware needs no init trick
  — physics settles). **Critical path = B2: make the equilibrium solver cheap** (Adam-on-states /
  init-sweep warm start / GS-multi-sweep / λ_l preconditioning), then rerun C1 in equilibrium mode.

## Tier 0 — gate hardening (probe-scale, hours, no training)  → K1

| ID | question | design | decision rule |
|---|---|---|---|
| A0.1 | does cos survive depth? | cos vs L ∈ {3,6,12,24}, C128, Jacobi K auto-scaled; ≥4 batches | cos ≥ 0.98 at L12 or B1 must fix it |
| A0.2 | does cos survive training? | BP-train C256 L6 4k steps saving every 500 (`casc_bp_train.py`); gate at every ckpt; ALSO record required-K to reach res-tol | cos ≥ 0.97 at all ckpts; K growth ≤ 3× init→4k |
| A0.3 | full-θ gate | include emb/pos/readout(tied) grads in the gate | all groups ≥ 0.97 |
| A0.4 | precision | fp32 vs TF32 vs bf16 on the two-phase difference | pick cheapest safe mode (looped-EP lesson: TF32 killed relaxation — re-test here) |

## Tier 1 — relaxation engineering (the cost frontier)  → K2

| ID | axis | arms | metric |
|---|---|---|---|
| B1 | scheme × K | Jacobi (physical, parallel) vs Gauss-Seidel fwd vs GS reverse (algorithmic; Z-IL limit) × K ∈ {12,25,50,100,200,400} at L6 & L12 | K needed for cos ≥ 0.98; wall-clock multiple vs one BP step |
| B2 | state optimizer | GD vs +momentum vs Adam-on-states; η sweep | same |
| B3 | nudge β | {0.003,0.01,0.03,0.1,0.3} × one-sided vs two-sided | cos, shrinkage |EP|/|BP|, required K |
| B4 | energy weighting | raw ℓ₂ vs per-layer precision λ_l=1/RMS² vs LN-in-energy | per-block shrinkage PROFILE (fix the 0.80→0.91 depth attenuation) + relax conditioning |
| B5 | stopping | fixed-K vs relax-to-tol | natural K distribution |
| B6 | **depth attenuation / estimator SNR profile** | measure per-block error amplitude ‖e_l‖ and per-block cos vs depth, as f(L, β, K) | the estimator-precision law: how fast does the deep-layer signal die, and which knob (β, K, λ_l weighting) restores it |

B1 is the single most consequential experiment in the program: if GS-reverse needs K≈L (Z-IL limit)
we have a ~BP-cost algorithmic mode for GPU pretraining, and the Jacobi column is the honest
analog-hardware price. Report all three columns — they are different products.

**Dynamics-vs-estimator tradeoff (user insight, 2026-07-09):** the cascade is dynamically SIMPLER —
the free phase is EXACT (a plain forward; no res/T1/fixed-point error, no Hopf, no collapse), so
**C-tier default arms run with NO regularizers at all** (jr/resreg don't exist here; stability regs
return only if evidence demands). The difficulty MOVES to the estimator: the two-phase difference
must resolve per-layer error signals that ATTENUATE with depth (visible at L=3 already: shrink 0.80
bottom vs 0.91 top), finite-β Taylor bias and finite-K relaxation bias hit the deepest blocks first,
and the difference-of-O(1)-quantities structure makes precision (A0.4, fp32-vs-TF32) bind harder
than in looped-EP. B6 is the dedicated measurement; λ_l weighting (B4), β/K scheduling (B3/B1) and
per-block rebalance (C5) are the candidate antidotes.

## Tier 2 — small full-training ablations (C256 L6 T256 TinyStories, 8–16k steps)  → K3

| ID | arm | vs |
|---|---|---|
| C1 | **money run**: cascade-EP (B-tier winner) ×2–3 seeds | BP twin, same arch/data/AdamW/steps — target gap ≤ 0.05 CE |
| C2 | K budget: {K*, 2K*, 4K*} | CE-vs-cost curve (training may need less relax than the gate does — looped-EP precedent: t2sel 40 trains, 80 gates) |
| C3 | one-sided β (half cost) | two-sided |
| C4 | AdamW | SGDM (shrinkage sensitivity — does 0.8–0.9 amplitude matter under Adam's rescaling?) |
| C5 | shrinkage compensation: none | per-block grad-norm rebalance to BP profile (one-time calibration) |
| C6 | B4-winner energy weighting | raw |

Placement: 1080 farm **after a Pascal canary** (cascade-EP is a new workload class; the Pascal
pathology ban was derived on looped-EP+regs — do a 800-step canary + cross-env fingerprint first).
C256 L6 fits 8 GB (~19M params, ~2-3 GB act).

## Tier 3 — depth/scale rungs (Delta A40 chains)  → K4

| ID | design |
|---|---|
| D1 | **north-star demo re-target**: L12 C512 (≈45M, a real GPT-small shape) cascade-EP vs BP twin — replaces the single-block 33M rung as the flagship demo (task #15) |
| D2 | depth ladder at fixed params: L6/C724 vs L12/C512 vs L24/C362 — depth penalty vs BP? |
| D3 | T 256→512 sanity (relax cost tracks attention; expect no surprise) |

## Tier 4 — analog/hardware arms (port the tolerance machinery)  → K5

| ID | design |
|---|---|
| E1 | Jacobi + per-sweep dynamic noise: does the fnoise ≥1e-3 cliff reappear in cascade relaxation? |
| E2 | Jᵀ ablation: replace J_lᵀe with fixed random Bᵀ (feedback-alignment) / PAR projection — the per-block analog-feasibility tax; FA classically works on shallow stacks, test at L6 |
| E3 | static tolerance: wq8/wq6 weights inside relax |

## Sequencing & fleet

```
now:      A0.1 + A0.3 + B1-lite (shared local GPU, ~1h)  +  casc_bp_train ckpt producer (107 free 1080)
gate ok → B1 full / B2 / B3 / B4 (local A6000s as arms free; each = minutes-hours)
        → Pascal canary → C-tier fan-out on 1080 farm (6 arms × 1-2 days)
        → D1 chains on Delta A40 (queue behind current five lines)
E-tier:   after C1 lands (tolerance scripts port directly)
```

Naming: `casc_*` runs, wandb project **ept-cascade**. Gates report mean over ≥4 batches.
In-flight single-block arms (rescv2, govfloor, fastfull/fastpair, gov_s11-14) continue untouched —
they carry the dynamics paper + the two-stage-recipe science; D1 takes over the DEMO role only.

### RESULT 3 (2026-07-10 03:03): K4 DEPTH-PARITY SEALED (EP-favorable) + full-epoch launched.
- **beta-floor 3e-4 EP 3-seed: 1.9005 / 1.9125 / 1.8591 = MEAN 1.8907** vs BP 1.9169/1.9194/1.9214
  (1.9192). **EP <= BP at L12xC512 (real GPT-small shape)** -- all 3 EP seeds below the best BP seed,
  cos pinned 1.0000 throughout, zero skips. The L12 depth-tax is FULLY removed by the beta-floor; K4
  closes EP-favorable. (Un-floored control was 2.00 + unstable/non-reproducible -- see RESULT 2.)
- Headline now: "standard L12 transformer, no backprop, equilibrium-EP with beta-floor = BP quality
  (slightly better) at matched tuning, real GPT-small shape."
- **FULL-EPOCH run LAUNCHED (user directive, auto-launched on verdict):** epoch_ep_bf3e4 -- 58,800
  steps = 1 epoch over TinyStories-BPE (361M tokens), beta_floor 3e-4 + --cosine (new flag), warmup
  500, save_every 5000. Running 2.376 it/s solo on GPU1 -> ~6.9 h. This is the "neng kan" generation
  demo (task #15). BP twin epoch DEFERRED (no free GPU; parity already sealed so it is nice-to-have).
- Next: generation samples at checkpoints; BP-twin epoch when a GPU frees; then scale-up corpus
  decision (FineWeb-Edu vs OLMo2/Dolma) for the larger model.

## ROADMAP PIVOT (2026-07-10 03:2x, user directive): QK-norm inserted; staged scale-up.
**User: cancel the full epoch (done — killed epoch_ep_bf3e4); insert a QK-norm version after the
current 3-seed; then stages TinyStories-full-epoch -> FineWeb-Edu -> OLMo2.**

**Why QK-norm:** RMS-normalize q,k per head before the scores (OLMo2/Llama-style). It BOUNDS the
attention logits, attacking the SAME root cause as the beta-floor (sig_tok growth -> logit blowup ->
finite-beta SNR collapse) but structurally. Analog-friendly (my analysis): it's divisive
normalization (mature analog/neuromorphic primitive), its Jacobian is symmetric (does NOT worsen the
PAR/non-reciprocity wall), it's feedforward (no digital root-finder / no adjoint), and it REUSES the
softmax current-normalization circuitry (reuse doctrine, no tapeout). Bonus analog wins: bounds the
input range of the analog softmax exp device; reduces sig-growth so relaxation is more robust.
Analog-preferred alternative to A/B in E-tier: tanh logit soft-cap (tanh is a native analog transfer
function -- possibly cheaper than the norm's square-sum+divide).

**Code:** nn.MultiheadAttention replaced by explicit CausalSelfAttn (SDPA-backed, fast) in BOTH
trainers; `--qk_norm` flag (RMS-norm over head_dim w/ learnable per-dim gain). Smoke: EP+qk_norm
cos=1.0000, 40.06M preserved, 2.49 it/s, SDPA works in the fb backward (fb is first-order, no
double-backward needed). Also added `--cosine` (warmup->cosine to 0.1x lr) for the long runs.

**QK-norm validation matrix (8 runs, L12 C512, 4000 steps, launched on GPU1):**
  - qk_bp_s1/s2/s3  = BP + qk_norm (new reference with the new block)
  - qk_ep_bf_s1/s2/s3 = EP + qk_norm + beta_floor 3e-4 (PARITY test vs qk_bp)
  - qk_ep_nf_s1/s2  = EP + qk_norm, NO beta_floor (ANALOG test: does qk_norm ALONE hold cos, letting
    us DROP the beta-floor? un-floored non-qk collapsed to cos 0.896 by step 4000 -- see RESULT 2).
Decision: (1) qk_ep_bf ~ qk_bp => parity preserved with qk_norm. (2) if qk_ep_nf ALSO holds cos~1 and
matches => qk_norm supersedes the beta-floor (fewer knobs, cleaner analog story). Watcher qk_watch.sh
fires at the early analog read (nf step 2500) or all-done.

**STAGED SCALE-UP (after qk_norm validates):**
  Stage 1: TinyStories FULL EPOCH (58,800 steps, 361M tok) with the validated qk_norm recipe + cosine
           -> the "neng kan" generation demo (task #15).
  Stage 2: FineWeb-Edu (real corpus, 32-50k tokenizer, ~150-300M params) -- best small-LM quality.
  Stage 3: OLMo2 / Dolma recipe -- fully-open reproducible baseline for the paper/collaborators.
  EP scaling knobs carried forward: beta_floor (or qk_norm if it supersedes), possibly double-sided
  nudge at larger scale (cancels O(beta) Taylor bias). $20k/run (Rain) ~ few-B tokens/run.

### RESULT 4 (2026-07-10 06:16): QK-norm validated — parity holds; beta-floor still needed; Stage 1 launched.
- **Parity with QK-norm (EP-favorable again):** BP+qknorm 1.9253/1.8753/1.9192 = 1.9066;
  EP+qknorm+beta_floor 1.8588/1.9176/1.8841 = **1.8868 <= BP**. QK-norm preserves EP=BP parity at L12.
- **ANALOG ANSWER: QK-norm does NOT replace the beta-floor** (they are complementary). EP+qknorm
  WITHOUT the floor still erodes cos (1.0 -> 0.946 by step 3200) and lands ~0.09 worse CE (2.02 vs
  1.89). Milder than the old non-QK collapse (0.896) but not fixed. **Why: sig_tok still grows to 21.5
  even with QK-norm** -- QK-norm normalizes q,k INSIDE attention (bounds the attention LOGITS) but does
  NOT bound the residual/embedding scale that drives beta_t = beta0*(sig0/sig)^2. So beta_t still
  collapses -> estimator SNR still needs the floor. QK-norm's payoff is (a) attention logit-bounding
  (analog softmax device range), (b) scale robustness (logit growth is worse in bigger/deeper models),
  (c) it is standard OLMo2/Llama -> good for the scale-up. Recipe = **qk_norm + beta_floor together**.
- **STAGE 1 LAUNCHED (user directive):** stage1_ep_qkbf -- TinyStories full epoch (58,800 steps, 361M
  tok), qk_norm + beta_floor 3e-4 + cosine, warmup 500, 2.4 it/s solo -> ~6.8 h. The "neng kan"
  generation demo. Watcher fires at step 10000 (first generation-worthy ckpt) / done / death.
  Then Stage 2 (FineWeb-Edu) -> Stage 3 (OLMo2).

### RESULT 5 (2026-07-10 07:3x): Stage-1 epoch BLEW UP @step 12100 — root-cause diagnosis (sig story REFUTED).
The qk_norm+beta_floor+cosine epoch was healthy to ~11400 (best val 1.6669) then blew up (val 1.67->7.6,
gn pre-clip 0.5->53) and oscillated in a degraded regime. **My first guess (sig_tok growth -> SNR
collapse -> add final_ln) was WRONG, refuted by its own telemetry:**
- sig rose only +8% (29.8@10000 -> 32.2@12000) then PLATEAUED; it was already ~30 at step 10000 when
  everything was healthy. An 8% change cannot cause a catastrophic transition.
- cos was FINE (0.9935) until step 11900; the cos drop is a CONSEQUENCE of the blowup, not the cause.
- grad-clip is ALREADY present (clip 1.0); gn=53 is pre-clip telemetry. Not a magnitude-spike issue.
**LEADING INDICATOR = skips (drift-guard rejections = nudged fb relaxation drift>0.5 = CONVERGENCE
FAILURE).** skips accelerate from ~step 11000 (4->13 by 11400) BEFORE gn (11700), cos (12000), val
(12100). **Diagnosis: a CONTRACTIVITY BIFURCATION in the nudged fb relaxation** -- as training sharpens
the operator (block Jacobians grow), an increasing fraction of batches have a non-contractive nudged
iteration -> skipped -> gradient bias -> a marginally-converged batch emits a bad step -> over the edge.
**This is the cascade analog of the looped-EP Hopf wall** (non-conservative attention loses
contractivity as CE drops -- documented in ep-c512-residual-defense-fix). 4000-step runs never saw it
(operator not sharp enough yet; edge ~step 11400). Right fix = CONTRACTIVITY control (resreg/jacreg or
geta<1 damping), NOT final_ln.
**CONFIRMATORY A/B/C (resume from ckpt-10000, pre-bifurcation, beta floored 3e-4 via --sig0 1.6):**
  A=control (K3,lr1e-3) -> should reproduce skip-climb+blowup; B=K8 (does more fb rounds hold skips?
  = marginal-contractivity test); C=lr3e-4 (slower sharpening -> delayed edge? = driver test).
  Code added: --resume, --sig0, --final_ln, --qk_norm(CausalSelfAttn/SDPA). Watcher diag_watch.sh armed.

## AUDIT (2026-07-10, model switch): re-review of the day's conclusions. Corrections + added controls.
**What SURVIVES audit:** RESULT 1 (K-invariance data is solid; K plumbed, paid wall-clock, identical
cos/CE); RESULT 2 (beta-floor effect is decisive and mechanistic: floored arms pin cos, unfloored
collapses); the 4k-horizon numbers themselves; the blowup telemetry read (skips lead gn lead cos lead
val); the D1a "no-death" correction; Delta cancellation scope.
**CORRECTIONS from audit:**
1. **Muon verdict RETRACTED as confounded.** d1_ep_muon (2.7515, cos 0.82) ran in the ORIGINAL D1a
   batch, i.e. WITHOUT beta_floor — its cos collapse mirrors the unfloored control (0.896). "Naive
   Muon-on-EP fails" is NOT established; needs a re-run with beta_floor before any conclusion.
2. **Parity claims toned down.** n=3 with best-of-noisy-val (6-batch val, min over ~500 evals ->
   selection bias ~0.02-0.03, applied to both arms) means "EP 1.8907 vs BP 1.9192" is PARITY with an
   EP-leaning point estimate, not "EP beats BP". (RESULT 3's all-3-EP-below-all-3-BP is p~=0.05 rank
   evidence — suggestive, not sealed.) Same for RESULT 4 (EP s2 1.9176 > BP best 1.8753).
3. **"Depth-tax FULLY removed" was premature** — true only at the 4k-step horizon; the epoch blowup at
   ~11.4k shows a second, longer-horizon wall. Claim scoped accordingly.
4. **"skips = relaxation non-convergence" is UNVERIFIED.** The skips counter conflates the drift-guard
   and the gn-EMA-guard; drift telemetry is stale-on-reject (GOV['drift'] not updated on drift-reject)
   while gn telemetry does update on gn-reject. Guard-split counters (skd/skg) now added to the log
   line for all future runs. The contractivity-bifurcation story remains the leading HYPOTHESIS, not
   a finding.
5. **A/B/C lacked the decisive control: a BP arm.** If BP-from-the-same-ckpt ALSO blows up, the blowup
   is a CONFIG instability (tied readout + NO final LayerNorm + sig~30 logits is genuinely nonstandard
   — every real GPT has final-LN; final_ln then likely IS the fix, via bounded logits/curvature, even
   though the sig->beta-SNR mechanism was refuted), and EP is exonerated. If BP sails through while A
   blows, the bifurcation is EP-specific -> jacreg/damped-fb. **diag_D_bp launched** (BP + --resume
   added to casc_bp_train, same ckpt-10000, qk_norm, lr 1e-3).
6. **Resume confounds now on record:** optimizer state is NOT in the ckpt (fresh Adam moments — sig
   jumped 29.8->35.6 within 300 steps of resume, visibly faster drift than the original run) and the
   data-order RNG restarts from the step-0 stream. So arm A can only reproduce the blowup
   STATISTICALLY, not at step 12100; if ALL arms blow immediately after resume, suspect the
   Adam-cold-start artifact rather than the original mechanism.
7. **Arm B (K8) is weakly informative by design:** for a genuinely divergent nudged iteration, MORE
   rounds = MORE drift, so both "K8 helps" and "K8 hurts" fit the story. The causal weight is on C
   (lr, sharpening-rate driver) and D (BP, EP-specificity).
8. Process fixes: watcher was not harness-tracked (user caught it — now all watchers via tracked bg
   tasks); zsh $VAR word-splitting cost two launch retries (all launches now via bash scripts).

### RESULT 6 (2026-07-10 09:35): WALL-2 DIAGNOSED — marginal under-convergence, EP-specific; kretry fix shipped; OLMo2 matrix launched.
A/B/C/D verdict (resume from pre-bifurcation ckpt-10000, beta floored):
| arm | skips @ window | note |
|---|---|---|
| A ctl (K3, lr1e-3) | **16, accelerating** (val wobble 2.00@12400) | leading indicator REPRODUCES |
| B K8 | **2** | rejections nearly eliminated |
| C lr3e-4 | **1**, best 1.5163 (best of all) | never touches the edge |
| D BP (same ckpt/config/lr) | clean through 12750 | **EP-specific confirmed** |
**Mechanism (two walls, two levers — revises "K refuted"):**
- Wall-1 (~2-4k): cos erosion = finite-beta SNR -> beta-floor (K genuinely irrelevant there).
- Wall-2 (~11k+): operator sharpens -> a growing fraction of batches sit at the CONTRACTIVITY EDGE of
  the nudged fb relaxation and under-converge at K3 -> drift-guard rejections climb -> gradient bias +
  occasional marginal escapes -> blowup. K8 CONVERGES those batches (16 -> 2 rejections) => marginal
  under-convergence, NOT hard divergence. lr modulates when the edge arrives (C: skips~1 and better CE).
  BP has no relaxation -> no wall-2 (D clean). Original 12100 didn't literally replay in A (fresh Adam
  + different data order — the recorded confounds) but the leading indicator did.
**FIX SHIPPED: `--kretry N`** — on drift-reject, RETRY the batch once with N fb rounds (B proved K8
converges them) instead of dropping it. Converts biased skips into converged gradients; costs extra
rounds ONLY on marginal batches (~0.1-1% of steps). Telemetry: skips=(d/g/r).
**OLMo2 4k matrix LAUNCHED** (ol_bp_s1-3 + ol_ep_s1-3, wd 0.1, EP: beta_floor 3e-4 + kretry 8; twin
step-0 losses bitwise-identical per seed). Watcher auto-computes parity and — if EP mean within 0.05
of BP — AUTO-LAUNCHES the Stage-1 OLMo2 TinyStories epoch (stage1_ol_ep, 58.8k steps, kretry armed).
OLMo2's bounded-per-branch signals may also shift wall-2 later; kretry is the belt-and-suspenders.

### RESULT 6-ADDENDUM (2026-07-10 10:5x): B(K8) ALSO BLEW at matched step — K delays, does NOT prevent.
diag_B_k8 @12600: train 3.37 / val 3.63 (best 1.6755 pre-blowup), gn 18.6, skips 2->14. So wall-2 is
NOT merely marginal under-convergence: the nudged fb iteration becomes GENUINELY DIVERGENT for a
growing batch fraction as the operator sharpens (true contractivity crossing — the cascade Hopf wall).
More rounds converge the marginal shell only; once past the edge no K helps. **kretry DEMOTED from fix
to mitigation** (still right for sporadic healthy-regime rejections). Note also: B blew with only 14
total rejections => most bad gradients passed UNDER the drift-0.5 threshold (loose guard + gn-EMA
poisoning during degradation).
Surviving facts: C (lr 3e-4) clean at 12600 (skips=1) -> sharpening RATE is the driver; D (BP) clean
-> EP-specific. **Defense ranking now: (1) OLMo2 arch (different operator: bounded branches + QK-norm;
diagnostics were all on the OLD arch) -> (2) lr channel (lower peak / faster decay through the
mid-training danger window) -> (3) true contractivity control (damped-fb gamma<1 / cascade-jacreg) if
OLMo2 still hits the wall.** Stage-1 OLMo2 epoch (parity-gated autolaunch) is the live test; watch
skips=(d/g/r) through the 10-14k window.

### RESULT 7 (2026-07-10 12:0x): OLMo2 4k PARITY — gate PASSED; arch worth ~0.07-0.10 CE to BOTH; epoch auto-launched.
- **BP+OLMo2: 1.8294/1.8333/1.8378 = 1.8335 (±0.004)** | **EP+OLMo2: 1.8294/1.8818/1.8817 = 1.8643** |
  gap +0.031 -> PASS (<=0.05) -> stage1_ol_ep AUTO-LAUNCHED (58.8k steps, qk+floor+kretry+cosine+wd).
- OLMo2 improved BOTH columns vs old arch at 4k (BP 1.9066->1.8335; EP 1.8868->1.8643) — the arch
  upgrade pays for itself immediately.
- HONEST READ: EP s1 == BP s1 to 4 decimals (1.8294, twin init); but EP s2/s3 trail their BP twins by
  ~0.045. Mean gap +0.031 is WITHIN the best-of-noisy-val metric band (~0.02-0.03, per audit), so:
  parity within noise, point estimate now slightly BP-leaning (was EP-leaning on old arch). Watch, not
  act: candidate causes = beta_t schedule now keyed to untied W_out sigma; norm-after changing fb
  conditioning (canary cos 0.9991 vs 1.0000). If the epoch shows a real gap, revisit.
- Pascal canaries GREEN (EP cos 0.9991 flat, 0 skips, 0.64 it/s; BP 2.0 it/s) -> farm UNBANNED for
  cascade: 2x BP-twin epochs (stage1_ol_bp_s1/s2, ~8h) + Muon-with-floor retest (ol_ep_muon_s1) now
  running on timan107 GPUs 6/2/7. NOTE farm-vs-local init differs (torch 2.3.1 vs 2.10 CUDA RNG) —
  config-matched anchors, not init-twins.
- Wall watch armed on the epoch: report at step 14000 (past the old 11.4k wall) with skips=(d/g/r).

### RESULT 8-PRELIM (2026-07-10 12:4x): Muon retraction CLOSED — with beta-floor, EP+Muon WINS big (n=1).
ol_ep_muon_s1 (OLMo2 + beta_floor + kretry + Muon, Pascal GPU7): **1.7316**, cos 0.9992, ZERO skips.
vs same-config-seed AdamW columns: BP 1.8294 / EP 1.8294 -> **-0.098 CE** (3-5x the metric noise band).
The original "naive Muon-on-EP fails (2.7515)" was ENTIRELY the missing beta-floor (audit correction
vindicated). Muon's known small/mid-scale advantage over AdamW TRANSFERS to EP gradients.
Controls launched: ol_bp_muon_s1 (the fair Muon-column comparison) + ol_ep_muon_s2 (seed robustness).
If BP+Muon lands ~1.73 too -> Muon helps both equally (parity preserved, recipe upgraded for BOTH
columns). If BP+Muon ~1.83 -> EP-specific synergy (bigger story, needs replication before claiming).
Interim: BP-twin epochs healthy at ~13.7k (best ~1.60 — already past the old-arch EP wall step);
local EP epoch at 4.8k, best 1.8148, zero skips, 1.97 it/s.

### RESULT 9 (2026-07-10 13:1x): WALL-2 ELIMINATED BY ARCHITECTURE — epoch cleared 10-14k with ZERO guard events.
stage1_ol_ep cleared the wall window (through step 14300): **skips=0 (d0/g0/r0) THE ENTIRE RUN** — not
one drift rejection, not one gn rejection, kretry never fired. (Old arch: 32 skips by 13000, blowup at
12100; K8 variant blew by 12600.) gn calm (~0.5), best val 1.6702 and descending at 1.97 it/s.
**OLMo2's bounded operator (norm-after-sublayer + QK-norm) stays contractive where the old block went
divergent — defense #1 closed the case; mitigations (kretry) unused.** The architecture change, made
for digital-standardness, is also the EP stability fix — "EP as configuration microscope" ends as
"modern standard config is EP-compatible out of the box."
WATCH ITEM: cos drifting slowly (0.9947@6k -> 0.9892@14k), beta already at floor. Watcher re-armed
with cos<0.985 trigger; if it keeps sliding by ~30k, try beta_floor 5e-4 or accept (grad quality still
fine at 0.989). Remaining epoch ETA ~6h.

### RESULT 8-FINAL (2026-07-10 13:3x): Muon attribution = GENERIC (helps both columns ~0.13 CE).
BP+Muon s1 **1.7020** vs BP+AdamW 1.8335; EP+Muon (1.7316/1.7191, n=2 mean 1.7254) vs EP+AdamW 1.8643.
Muon's advantage TRANSFERS to EP gradients at full magnitude — not an EP-specific synergy, the known
small/mid-scale Muon-beats-AdamW result, now demonstrated on backprop-free training. **Muon = default
optimizer for BOTH columns from Stage-2 (FineWeb-Edu) onward.** Muon-column EP-BP gap +0.023 ~ AdamW
column's +0.031 (consistent slight BP-lean on OLMo2, noise-band edge, on the watch list).
HW-narrative guard: Muon's Newton-Schulz is matrix-matrix (analog-dead) but the optimizer lives
DIGITAL-side per standing doctrine — GPU-pretraining Muon does NOT conflict with the factored-Adam
analog training story. Filling to 3v3 (BP+Muon s2/s3, EP+Muon s3) for the seal.

### WATCH (2026-07-10 14:0x): late-epoch cos erosion = intrinsic late-training SNR decline. Decision: let it run.
cos 0.9894@17k -> 0.9840@22k (-0.0011/1k), ZERO skips, gn calm, val still improving (1.6353). sigma
plateaued (~32) and beta at floor => ratio stable => NOT the sigma-growth wall-1. Mechanism: true
gradient magnitude shrinks as CE approaches optimum while the estimator noise floor stays constant ->
SNR falls with the signal. Extrapolates to cos~0.94 by 58.8k. DECISION: no mid-flight surgery (resume
reintroduces Adam/data confounds; cosine-LR shrinks late steps anyway). The BP epoch twins ARE the
measurement: EP final within ~0.03 of BP -> erosion harmless; 0.1 behind -> quantified problem with a
ready dial (late beta_floor schedule, e.g. 5e-4 past 20k). Pre-validation probe queued: when the farm
frees, run ckpt-25000 + floor 5e-4 x 2k steps. Watcher re-armed at cos<0.96.

### RESULT 8-SEALED + beta-floor dose-response (2026-07-10 16:1x).
**Muon 3v3 SEALED: BP+Muon 1.7020/1.7137/1.7136 = 1.7098 | EP+Muon 1.7316/1.7191/1.7435 = 1.7314.**
Muon default for both columns from Stage-2. The +0.02-0.03 BP-lean now CONSISTENT across two optimizer
columns (6v6) -> upgraded from noise to "probably real small effect"; primary suspect = late-training
SNR (see below), because it is beta-liftable:
**beta_floor dose-response @ckpt-25000 (same weights/batch): 3e-4 -> cos 0.984 | 5e-4 -> 0.9889 |
1e-3 -> 0.9940.** Raising the floor lifts cos exactly as the SNR mechanism predicts, with drift=0.000
and zero skips at 1e-3 (larger nudge does NOT destabilize the OLMo2 relaxation). 2k-step traces
harvesting (auto-kill at 27k). RECIPE UPDATE for Stage-2 (and the next epoch): late beta_floor
schedule — floor 3e-4 early, ramp to ~1e-3 in the back half (or floor ∝ 1/grad-norm). This likely
also closes the +0.02-0.03 gap.

### Dose-response SUSTAINED (25.6k-27k, 2k-step parallel traces): floor 3e-4 ~0.975 (accelerating
down, -0.0025/1k) | 5e-4 ~0.983 | 1e-3 ~0.990 flat, zero instability. Late-SNR mechanism + fix both
confirmed in-training. `--bf_late/--bf_late_at` flags shipped. NEXT-RUN RECIPE (post-epoch): OLMo2 +
Muon + beta_floor 3e-4 + bf_late 1e-3 @ ~20k + kretry 8 — expected to hold cos>=0.99 end-to-end and
likely close the +0.02-0.03 column gap.

### PLAN UPDATE (2026-07-10 17:2x): cos crossed 0.96 (0.9544@39.3k, accelerating) — flagship stays
UNTOUCHED (the control measuring erosion damage vs BP twins); PARALLEL bf1e3 continuation launched
from ckpt-40000 on the farm (floor 1e-3 for the remaining 18.8k steps). Endpoint comparison becomes a
clean quad: EP-control(3e-4) / EP-floor-lift(1e-3 from 40k) / BP-s1 / BP-s2 — quantifies BOTH the
erosion damage AND the fix's recovery in one shot.

### RESULT 10 (2026-07-10 19:0x): EPOCH ENDPOINTS + "NENG KAN" GATE PASSED + stage1b (improved recipe) launched.
**Epoch endpoints (58,800 steps / 361M tokens, OLMo2, AdamW):**
| arm | best val CE |
|---|---|
| BP s1 / s2 | **1.2750 / 1.2509** |
| EP (floor 3e-4 fixed) | **1.4802** (zero guard events end-to-end) |
| EP bf1e3-cont (floor->1e-3 @40k) | 1.4835@46k, running to 58.8k |
**EP-BP gap at epoch scale = +0.22** (was +0.03 at 4k): the late-SNR cos erosion (1.0 -> ~0.92-0.95)
is a REAL, horizon-growing CE cost with fixed floor 3e-4. Mechanism + dial both established
(dose-response); the improved recipe is designed to close this.
**GENERATION GATE ("neng kan") PASSED:** casc_gen.py (new; plain-forward standard-LLM inference) from
EP s55000: coherent multi-paragraph TinyStories — named characters, balanced-quote dialogue,
cause-effect, emotional arc (minor charm-defects vs BP's tighter coherence, consistent with +0.22).
**A 42.75M standard 12-layer transformer trained end-to-end WITHOUT backprop tells coherent stories;
inference is a plain forward pass.** Task #15 demo artifact exists.
**stage1b launched (the improved-recipe head-to-head):** stage1b_ep_muon (local GPU1: Muon + floor
3e-4 + bf_late 1e-3@15k + kretry + cosine[now also on Muon via build_hybrid total_steps]) vs
stage1b_bp_muon (farm GPU6: Muon + cosine). Expectation: EP ~1.25-1.35 (Muon -0.13 and erosion fix
~-0.1+), BP+Muon anchor moves too. ~8h both.

### QUEUE (user, 2026-07-10): double-sided nudge — implement AFTER stage1b endpoint.
The 0.22 diagnosis: EP's one extra constraint = the gradient is a DIFFERENTIAL MEASUREMENT
(SNR ∝ β|g|/(ε·σ)) vs BP's analytic adjoint. Escalation ladder: stage1b ramp (running) →
double-sided ±β (kills O(β) Taylor bias, unlocks ~10× β for SNR, 2× nudge cost; A/B at 25k-ckpt
2k-step probe when implemented) → fp64 E-accumulation / readout averaging. Analog note: this
constraint IS the hardware constraint (ε = device noise); β-scheduling learned here = chip ops
manual; hardware bonus = nudge amplitude free under multiplicative noise (r-indifference).

### RESULT 11 (2026-07-11): stage1b SEALED (gap 0.22->0.050); beta ceiling not reached; K exonerated on OLMo2; bf16 naive-cast dead.
- **stage1b endpoints: EP+Muon+floor-ramp 1.2808 | BP+Muon 1.2311 -> epoch gap 0.050** (fixed-floor was
  +0.22). EP now beats the old BP-AdamW epoch (1.2509/1.2750). Intervention-timing quad complete:
  fixed-floor 1.4802 / lift@40k 1.4479 / full ramp 1.2808 -- monotone earlier-is-better dose curve.
- **Gap probes @s45000 (2k-step sustained traces):** control cos 0.9947 | b2e3 0.9968 | **b3e3 0.9974
  (deficit halved, zero drift/skips)** | K5 0.9951 ~= control -> **K-invariance now proven on BOTH
  architectures; the residual deficit is beta-liftable, not relaxation-depth.** sigma(W_out)=80 by 45k:
  without the floor beta_t would be ~1e-6 -- the floor carries the entire late phase.
  CE-endpoint test launched: stage1b_f3e3cont (s45000 -> 58.8k at floor 3e-3, farm). If it closes to
  <=0.03, next-flagship recipe = ramp ...->3e-3@~35k; else double-sided (queued) takes the residual.
- **bf16 gate: naive full-cast FAILS at any beta.** floor 3e-4 -> cos 0.33; 3e-3 -> 0.67; 1e-2 -> 0.65
  (no longer SNR-limited: bf16 rounding distorts the nudged equilibrium itself; beta cannot compensate).
  Speed was 2.1x (5.2 it/s). VERDICT: cost baseline stays TF32 (validated); the x0.5 lever requires
  proper mixed precision (bf16 weights/matmuls + fp32 states/accumulation, autocast-style) -- queued as
  engineering upside, NOT in the Ben cost baseline. Wall-1 physics predicted all of this (SNR ∝ beta/eps;
  bf16 eps ~8000x fp32): the fp32/bf16/analog-noise beta-epsilon scaling story now has a second
  measured point.

## STANDING DIRECTIVE (user, 2026-07-11): LOOPED LINE ABANDONED.
The looped/weight-tied single-block line is retired as a research direction. Default everywhere:
cascade (tied, PCN-form energy over distinct standard blocks) is THE line. The looped record survives
ONLY as historical evidence inside the dynamics paper (Hopf phenomenology, dips, governor, eig
audits — valid data, past tense). Consequences: no new looped runs; looped-specific queue items
closed (adaptive-eps integrator, Pascal five-arm reg triage, S1-S3 looped ladder); report v3
sections 7.1/8 to be reframed past-tense on the Overleaf pass ("a companion system we studied", not
"our companion product"); AsymEP machinery = dynamics-paper subject matter, not the training recipe.

### RESULT 12 (2026-07-12): f3e3cont NEGATIVE — the residual 0.050 gap is NOT late-beta-SNR-limited.
stage1b_f3e3cont (floor 3e-3 from s45000): **1.2883** vs stage1b 1.2808 (floor 1e-3) — no gain (cos
0.994->0.997 bought nothing in CE). beta lever exhausted at 1e-3. Residual-gap suspects, in order:
(a) single-sided O(beta) Taylor bias sustained over 59k steps -> **next lever = RANDOM-SIGN beta**
(flip sign per batch; single-phase cost; averages away the systematic first-order bias; validated
competitive at full ImageNet by Kerjan-Hoier-Scellier) then centered (2x nudge) if needed;
(b) fb K=3 finite-relaxation bias; (c) Muon x gradient-noise interaction; (d) ~0.02-0.03 of the 0.05
is metric-noise band. Recipe note: random-sign is a one-line trainer change (sign of beta_t per step).

### RESULT 13 (2026-07-12): bsign (random-sign beta) NEUTRAL at 4k — 42M gap-chasing has hit the noise floor. THREAD CLOSED.
bsign 3-seed: 1.7031/1.7562/1.7552 (mean 1.7382) vs single-sided 1.7314 vs BP+Muon 1.7098. The bias
reduction is cancelled by injected update-direction variance at this horizon (seed spread now dominates:
s1 alone beat the BP mean). Ledger of the residual-0.05 epoch gap after three probes: NOT late-beta-SNR
(f3e3cont), NOT K (K5 probe), NOT first-order sign bias at short horizon (bsign). Remaining mass:
~0.02-0.03 metric-noise band + small unattributed accumulation. **Decision: stop polishing 42M.**
Carry `--bsign_rand` and a future centered mode as Stage-2 A/B flags; the gap question re-opens at
300M/real-corpus where it means something. Effort pivots to: (1) Stage-2 data pipeline (FineWeb-Edu +
32k tokenizer), (2) E-tier tolerance suite on the idle farm (hardware track / UIUC outreach feed).

### RESULT 14 (2026-07-12): E-TIER WAVE-1 — full analog-fault tolerance ledger at stage1b s55000.
`etier_probe.py`, farm GPUs 2/3/7 (shards A/B/C), stage1b_ep_muon_s55000.pt (clean valCE 1.2678),
B=8 eval batches; metrics = faulted valCE (Δ vs clean), cos(EP_faulted, BP_faulted) [self-consistency
of the learning signal under fault], cos(EP_faulted, BP_clean) [direction vs the ideal update].

| fault (component) | mild | medium | severe | verdict |
|---|---|---|---|---|
| wq — weight quant (crossbar #3) | 8b: +0.004 / 0.956 | 6b: +0.051 / 0.812 | 4b: +2.23 / 0.05 | **8b FREE, 6b marginal, 4b dead → ≥7b effective is the binding spec** |
| fnoise — fwd additive state noise (softmax/relax #5) | 1e-3: 0.000 / 0.975 | 3e-3: 0.000 / 0.974 | 1e-2: +0.001 / 0.970 | **FREE at 1% — looped-era 1e-3 cliff does NOT transfer to cascade** |
| divmis — divisive-norm mismatch (#4/#7) | 1%: 0.000 / 0.971 | 3%: +0.003 / 0.948 | 10%: +0.042 / 0.785 | 3% (routine matching) FREE; 10% marginal |
| rope — phase error rad (#2) | 0.01: 0.000 / 0.973 | 0.03: +0.001 / 0.967 | 0.1: +0.012 / 0.923 | 0.03 rad FREE; ~2° I/Q accuracy suffices |
| gilbert — gate gain error (#6) | 1%: 0.000 / 0.974 | 3%: 0.000 / 0.971 | 10%: +0.006 / 0.945 | **FREE at 10%** — translinear practice is comfortably inside |
| fbnoise — nudge/error-channel noise | 1e-2: 0.969 / 0.975 | **1e-1: 0.951 / 0.957** | 3e-1: 0.764 / 0.768 | **10% relative noise on the ERROR CHANNEL is FREE** (cos 0.95) — the r-indifference/large-nudge gift, now measured on cascade |

Reading: (a) the only hard constraint is crossbar weight precision (≥7b effective — inside standard
SRAM-CIM capability; 6b rescue = wave-2 quant-aware co-training); (b) everything dynamic — forward
noise 1%, error-channel noise 10%, gate/divider/phase mismatch at routine device tolerances — is
FREE at this scale. cos(EP,BP_faulted) stays ~0.97 under every non-fatal fault: the EP estimator
tracks whatever network the faults define, i.e. learning co-adapts to the fault (the analog-training
thesis in one number). CAVEAT: static probes at a trained checkpoint (eval CE + one-step gradient
direction), not training-under-fault; wave-2 = co-training with faults injected from step 0
(expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md
(per-row status updated) + UIUC outreach dossier.

### RESULT 16 (2026-07-13): STAGE-2 DATA PIPELINE LIVE + FINEWEB SMOKE PASSED.
`prepare_fineweb.py`: FineWeb-Edu sample-10BT -> 32k ByteLevel BPE (<|eot|> id 0) -> uint16 bins,
tinystories_bpe format, `--data` flag added to both trainers. SMOKE_READY in 11 min (download 5min
@80MB/s, tokenizer train 39s on 1.5GB, shard0 5min = 755M tokens; val = first 20M, disjoint).
Full 14 shards -> ~10.5B tokens (running, ~50 min ETA at 754M/5.4min per shard).
**fw_smoke (L12 C512 32k-vocab = 72.11M, --amp, 400 steps, GPU1): CE 10.51 -> 5.84, cos(EP,BP)
0.9999@0 / 0.9951@100 / 0.9995@400, ZERO skips, drift 0.002.** The estimator + amp + beta-governance
(sig grew 3.9->58, beta floored by step 100 — wall-1 machinery engaged correctly on the harder
corpus) all transfer to real web text at 4x vocab unchanged. Speed 0.89 it/s at this shape (bigger
head). Stage-2 recipe question OPEN for user: T=1024 (web-native context) vs T=256 (strict
TinyStories comparability) for the 300M run. NOTE: the "BP twin 2.9746" line in DONE prints is the
stale TinyStories reference (cosmetic); no fineweb BP twin exists yet.

### RESULT 15 (2026-07-12): bf16 MIXED PRECISION (--amp) VALIDATED — lossless at 4k, 1.56x wall-clock.
The /2-class cost lever, same-day pipeline: amp_gate.py static gate -> trainer flag -> 3-seed A/B.
SEMANTICS (why this lives while naive-cast --bf16 is dead): params/states/displacements/E-accum stay
fp32; ONLY block forwards run under autocast(bf16). RESULT 11's naive-cast death = pure STATE
quantization (wall-1: beta-displacement below bf16 resolution) — exactly as diagnosed.
- Gate (stage1b s55000, fp64 cosine): amp cos(EP,BP_fp32) 0.9682 vs fp32-EP 0.9687 (zero loss);
  beta=3e-3 -> 0.9878, 1e-2 -> 0.9966 (bigger beta ACTIVELY better — wall-1 SNR physics);
  bf16 fwd valCE -0.0002; BP_amp baseline 0.9993. amp_last (fp32 final rebuild) buys nothing ->
  amp_all everywhere; the E-subtraction term is not binding at production beta (fbnoise-tolerance
  prediction from RESULT 14 held: relative noise on forces is invisible).
- 3-seed 4k A/B (bsign flagset + --amp): 1.7086/1.7569/1.7562 mean 1.7406 vs fp32 3v3 mean 1.7314
  (Delta +0.009 inside the seed-noise band; amp_s1 BEAT the BP+Muon mean 1.7098). In-trainer bp_gate
  cos 0.9999 at step 0. Zero guard events.
- SPEED (solo GPU3/A6000, C512): amp 2.785 it/s vs fp32 1.789 it/s = 1.56x wall-clock; grows with
  width (tensor-core-bound share) -> treat 1.5x as the floor for 1-3B on H100.
- ~~CAVEAT + confirm step~~ **EPOCH CONFIRM SEALED (2026-07-12 late): stage1b_amp DONE best val
  CE 1.2868 vs fp32 1.2808 (Δ+0.006, inside the 0.02-0.03 best-of-noisy-val band); zero guard
  events over 58.8k; 2.80 vs ~1.79 it/s = the 1.56x held for the full epoch.** amp = Stage-2
  default, full confidence. EMAIL_BEN_DRAFT2 gate #6 CLEARED (the sent "validated this week"
  claim is now closed at epoch scale).
- Cost consequence: COST_MODEL.md v2.1 (sourced July-2026 prices: market H100 $1.87-2.99/GPU.h,
  AWS p5e blocks $4.97/GPU.h post-hike) — with amp measured, 3B-Chinchilla ~$40k / 7Bx20B ~$30k
  on AWS blocks: BOTH inside the $50k envelope individually. amp is the Stage-2 default.