diff options
Diffstat (limited to 'docs/campaign/CASCADE_ABLATION_PLAN.md')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 26 |
1 files changed, 25 insertions, 1 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index b6d8927..34d16d5 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -32,9 +32,33 @@ The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether - K3 quality: **matched-tuning PARITY n=3** (EP-exact 2.0500±0.015 vs BP 2.0530±0.004 @ C256 L6, lr 1e-3 both). Arc: fake-win (lr artifact) -> fake-tax (v7 dedups) -> parity. Fast mode = documented -4%CE/+20%speed dial. A0.4: TF32 free, bf16 production-only (cos 0.9427). -- D1a (L12xC512 45M): BP 1.917 sealed; EP + Muon arms landing. D1b long-run demo gated on D1a parity. +- D1a (L12xC512 45M): BP s1/s2 SEALED 1.9169/1.9194 (H8, lr1e-3, tok_init0.02, 4000 steps, adamw). - E-tier: next in queue (softmax pathology / error-channel SNR / write pricing) -> Demo-0 spec sheet. +## D1a AUTOPSY + K-LADDER DIAGNOSTIC (2026-07-09 night) +**What happened:** the 4 D1a arms (d1_ep_s1/s2/s3 + d1_bp_s3) all died at wall-clock 23:36, mid-run, +at a step boundary with NO traceback and NO DONE marker -> classic PARENT-DEATH (launched inline, not +nohup'd; the launching shell/session terminated and took them down). No OOM in journalctl/dmesg. NOT a +training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** +**Interim signal BEFORE they died (the science):** at L12 the EP estimator degrades with training in a +way it did NOT at L6: + - EP s1: best val 2.0444 @ step 2800, then val BOUNCED to 2.0951 @3200 (last line); cos(EP,BP) + eroded 1.0000 -> 0.9942 (@2800) -> 0.9897 (@3200) as beta_t adapted DOWN 3e-3 -> 1.9e-5. + - EP s3: cos fell to 0.9834 AND the quality gate started SKIPPING steps (skips=4). + - vs BP s1/s2 which finished clean at 1.917. So at step ~3200 EP is ~0.10-0.13 CE above BP and the + curve is stalling while cos degrades -- the DEPTH-ATTENUATION / estimator-SNR prediction (B6/K4). +**Mechanism hypothesis:** K=3 fb message-passing rounds were tuned at L6xC256; the deeper L12 nudged +equilibrium under-converges, and as beta_t shrinks (nudge -> tiny) the two-phase difference becomes a +small signal against fixed relaxation error -> cos erodes -> gradient quality drops late in training. +**Diagnostic launched (local GPU1, nohup, seed 1, full 4000 steps, H8 lr1e-3 tok_init0.02 beta3e-3):** + - `d1b_ep_K3_s1` (K=3 control, honest 4000-step reproduction) + - `d1b_ep_K8_s1` (K=8 = kmax, strongest relaxation -- does more convergence hold cos~1 and close CE?) + - `d1_bp_s3` relaunch (completes the 3-seed BP reference). +**Decision rule:** if K8 holds cos>=0.999 through step 4000 and reaches ~BP CE -> gap was +under-convergence, fix = scale K with depth, then relaunch full 3-seed at min-sufficient K for the K4 +verdict. If K8 does NOT close it -> genuine estimator depth-tax; next arm = beta-floor (needs a code +flag) and/or lambda_l per-layer energy weighting (B4). Follow-on (not yet launched): Muon-on-EP arm. + ## STATUS 2026-07-09 (same day): Tier 0 CLOSED GREEN via the zil scheme; C1 running - **Naive relaxation FAILS at depth** (the B1-lite sweep): jacobi K=40·L → cos 0.82 (L6) / 0.67 (L12) |