AUDIT: retract confounded Muon verdict; tone down parity claims (n=3, best-of-noisy-val); scope depth-tax claim to 4k horizon; guard-split skip telemetry (skd/skg); add BP control arm diag_D_bp (+--resume in BP trainer); record resume confounds

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn

3 files changed, 49 insertions, 2 deletions

diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index 19f97bb..55df056 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -305,3 +305,40 @@ geta<1 damping), NOT final_ln.
   A=control (K3,lr1e-3) -> should reproduce skip-climb+blowup; B=K8 (does more fb rounds hold skips?
   = marginal-contractivity test); C=lr3e-4 (slower sharpening -> delayed edge? = driver test).
   Code added: --resume, --sig0, --final_ln, --qk_norm(CausalSelfAttn/SDPA). Watcher diag_watch.sh armed.
+
+## AUDIT (2026-07-10, model switch): re-review of the day's conclusions. Corrections + added controls.
+**What SURVIVES audit:** RESULT 1 (K-invariance data is solid; K plumbed, paid wall-clock, identical
+cos/CE); RESULT 2 (beta-floor effect is decisive and mechanistic: floored arms pin cos, unfloored
+collapses); the 4k-horizon numbers themselves; the blowup telemetry read (skips lead gn lead cos lead
+val); the D1a "no-death" correction; Delta cancellation scope.
+**CORRECTIONS from audit:**
+1. **Muon verdict RETRACTED as confounded.** d1_ep_muon (2.7515, cos 0.82) ran in the ORIGINAL D1a
+   batch, i.e. WITHOUT beta_floor — its cos collapse mirrors the unfloored control (0.896). "Naive
+   Muon-on-EP fails" is NOT established; needs a re-run with beta_floor before any conclusion.
+2. **Parity claims toned down.** n=3 with best-of-noisy-val (6-batch val, min over ~500 evals ->
+   selection bias ~0.02-0.03, applied to both arms) means "EP 1.8907 vs BP 1.9192" is PARITY with an
+   EP-leaning point estimate, not "EP beats BP". (RESULT 3's all-3-EP-below-all-3-BP is p~=0.05 rank
+   evidence — suggestive, not sealed.) Same for RESULT 4 (EP s2 1.9176 > BP best 1.8753).
+3. **"Depth-tax FULLY removed" was premature** — true only at the 4k-step horizon; the epoch blowup at
+   ~11.4k shows a second, longer-horizon wall. Claim scoped accordingly.
+4. **"skips = relaxation non-convergence" is UNVERIFIED.** The skips counter conflates the drift-guard
+   and the gn-EMA-guard; drift telemetry is stale-on-reject (GOV['drift'] not updated on drift-reject)
+   while gn telemetry does update on gn-reject. Guard-split counters (skd/skg) now added to the log
+   line for all future runs. The contractivity-bifurcation story remains the leading HYPOTHESIS, not
+   a finding.
+5. **A/B/C lacked the decisive control: a BP arm.** If BP-from-the-same-ckpt ALSO blows up, the blowup
+   is a CONFIG instability (tied readout + NO final LayerNorm + sig~30 logits is genuinely nonstandard
+   — every real GPT has final-LN; final_ln then likely IS the fix, via bounded logits/curvature, even
+   though the sig->beta-SNR mechanism was refuted), and EP is exonerated. If BP sails through while A
+   blows, the bifurcation is EP-specific -> jacreg/damped-fb. **diag_D_bp launched** (BP + --resume
+   added to casc_bp_train, same ckpt-10000, qk_norm, lr 1e-3).
+6. **Resume confounds now on record:** optimizer state is NOT in the ckpt (fresh Adam moments — sig
+   jumped 29.8->35.6 within 300 steps of resume, visibly faster drift than the original run) and the
+   data-order RNG restarts from the step-0 stream. So arm A can only reproduce the blowup
+   STATISTICALLY, not at step 12100; if ALL arms blow immediately after resume, suspect the
+   Adam-cold-start artifact rather than the original mechanism.
+7. **Arm B (K8) is weakly informative by design:** for a genuinely divergent nudged iteration, MORE
+   rounds = MORE drift, so both "K8 helps" and "K8 hurts" fit the story. The causal weight is on C
+   (lr, sharpening-rate driver) and D (BP, EP-specificity).
+8. Process fixes: watcher was not harness-tracked (user caught it — now all watchers via tracked bg
+   tasks); zsh $VAR word-splitting cost two launch retries (all launches now via bash scripts).
diff --git a/ep_run/casc_bp_train.py b/ep_run/casc_bp_train.py
index 7647fae..069d858 100644
--- a/ep_run/casc_bp_train.py
+++ b/ep_run/casc_bp_train.py
@@ -21,6 +21,7 @@ ap.add_argument('--cosine', action='store_true')         # warmup then cosine de
 ap.add_argument('--lr_min_ratio', type=float, default=0.1)
 ap.add_argument('--qk_norm', action='store_true')        # RMS-norm q,k per head before scores (OLMo2-style; bounds logits, analog-friendly)
 ap.add_argument('--final_ln', action='store_true')       # final LayerNorm before readout (standard GPT; bounds sig_tok growth -> keeps beta/estimator healthy on long runs)
+ap.add_argument('--resume', default='')                  # path to a ckpt (tok/pos/blocks) to continue from; step taken from ckpt
 args = ap.parse_args()
 torch.manual_seed(args.seed)
 dev = 'cuda' if torch.cuda.is_available() else 'cpu'
@@ -76,6 +77,12 @@ blocks = nn.ModuleList([Block(args.C, args.H, args.qk_norm) for _ in range(args.
 mask = torch.triu(torch.full((args.T, args.T), float('-inf'), device=dev), 1)
 ln_f = nn.LayerNorm(args.C).to(dev) if args.final_ln else nn.Identity()
 params = list(tok.parameters()) + list(pos.parameters()) + list(blocks.parameters()) + list(ln_f.parameters())
+start_step = 0
+if args.resume:
+    _ck = torch.load(args.resume, map_location=dev, weights_only=False)
+    tok.load_state_dict(_ck['tok']); pos.load_state_dict(_ck['pos']); blocks.load_state_dict(_ck['blocks'])
+    start_step = int(_ck.get('step', 0))
+    print(f'[resume] loaded {args.resume} at step {start_step}', flush=True)
 if args.opt == 'muon':
     from muon import build_hybrid
     opt, sched = build_hybrid(blocks, params, args.lr, args.muon_lr, args.warmup)
@@ -116,7 +123,8 @@ n = sum(p.numel() for p in params)
 print(f'[{args.tag}] cascade-BP L{args.L} C{args.C} H{args.H} T{args.T} | {n/1e6:.2f}M params | {dev}', flush=True)
 best, t0 = 1e9, time.time()
 outdir = Path('runs'); outdir.mkdir(exist_ok=True)
-for step in range(args.steps + 1):
+for _ in range(start_step): sched.step()   # advance LR schedule to the resumed step
+for step in range(start_step, args.steps + 1):
     x, y = get_batch('train')
     loss = F.cross_entropy(fwd(x).reshape(-1, vocab), y.reshape(-1))
     opt.zero_grad(set_to_none=True); loss.backward()
diff --git a/ep_run/casc_eq_train.py b/ep_run/casc_eq_train.py
index 74d6a3b..b85ac96 100644
--- a/ep_run/casc_eq_train.py
+++ b/ep_run/casc_eq_train.py
@@ -211,6 +211,7 @@ def ep_step(x, y):
         drift = sum(float((a - b).norm()) for a, b in zip(zp, zs_free)) / max(
             sum(float(b.norm()) for b in zs_free), 1e-9)
     if (not math.isfinite(drift)) or (drift > 0.5 and not args.noguard):
+        GOV['skd'] = GOV.get('skd', 0) + 1   # drift-guard reject (relaxation non-convergence)
         for p in all_params: p.grad = None
         return free_ce, beta_t, GOV['K'], False
     GOV['drift'] = drift
@@ -226,6 +227,7 @@ def ep_step(x, y):
     GOV['gema'] = 0.99 * GOV['gema'] + 0.01 * gn        # EMA always updates (frozen-ref bugfix)
     GOV['gn'] = gn
     if not math.isfinite(gn) or (gn > 8 * GOV['gema'] and not args.noguard):
+        GOV['skg'] = GOV.get('skg', 0) + 1   # gn-EMA-guard reject (gradient-magnitude spike)
         for p in all_params: p.grad = None
         return free_ce, beta_t, GOV['K'], False
     for p, g in zip(all_params, gs):
@@ -287,7 +289,7 @@ for step in range(start_step, args.steps + 1):
         val = evaluate(); best = min(best, val)
         gtag = '' if math.isnan(gcos) else f' cos={gcos:.4f}'
         print(f'step {step:5d}/{args.steps} | train {ce:.4f} val {val:.4f} (best {best:.4f}) '
-              f'| beta={beta_t:.2e} K={rounds} skips={skips}{gtag} '
+              f'| beta={beta_t:.2e} K={rounds} skips={skips}(d{GOV.get("skd",0)}/g{GOV.get("skg",0)}){gtag} '
               f'drift={GOV["drift"]:.3f} gn={GOV["gn"]:.2e} sig={GOV["sig"]:.1f} | {step/max(time.time()-t0,1e-9):.3f} it/s', flush=True)
         if wb is not None:
             try: wb.log({'train_ce': ce, 'val_ce': val, 'best': best, 'beta_t': beta_t,