diff options
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 20 |
1 files changed, 18 insertions, 2 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index ede9c39..a4fab80 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -36,10 +36,26 @@ The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether - E-tier: next in queue (softmax pathology / error-channel SNR / write pricing) -> Demo-0 spec sheet. ## D1a AUTOPSY + K-LADDER DIAGNOSTIC (2026-07-09 night) -**What happened:** the 4 D1a arms (d1_ep_s1/s2/s3 + d1_bp_s3) all died at wall-clock 23:36, mid-run, +**>>> CORRECTION (2026-07-10 02:xx): the "parent-death" below was a MISDIAGNOSIS. <<<** +The original D1a arms did NOT die -- they completed normally. When I checked at ~23:44 they were ALIVE +at step 3200 on GPUs 0/3 (both at 100%); my /proc scan was mangled by a zsh eval wrapper so I misread +"no casc alive", and GPU1 being free (11 MiB) fooled me (the runs were on 0/3, not 1). d1_ep_s1.log is +continuous 0->4000 at steady 0.679 it/s (3200->4000 = 19.6 min, matches its 00:06 mtime). **Original +D1a finals: d1_ep_s1 1.9745 / s2 2.0013 / s3 2.1188 (fixed K3; s2 blew at step 4000 skips=9, s3 blew +hard skips=23 governor ramped K->7); d1_ep_muon 2.7515 (Muon-on-EP, cos collapsed 0.82).** The d1b +experiments I launched (thinking the originals died) ran on the GENUINELY-FREE GPU1, so no competition +-- and they independently isolated the real mechanism + fix (below), which is the bigger prize. Net: +no harm, wrong death-story, and we now have BOTH the original un-floored 3-seed AND the beta-floor fix. +KEY read of the original 3-seed: un-floored K3 is HIGH-VARIANCE near the SNR cliff -- s1 got lucky and +stayed stable (1.9745, closest to BP), s2/s3 blew up late. Same-seed non-determinism (fb+autograd +reductions) means the un-floored estimator is not even reproducible near the cliff. That is the +strongest argument FOR the beta-floor (which pins cos=1.0000, stable, reproducible). + +**What I ORIGINALLY (wrongly) concluded:** the 4 D1a arms all died at wall-clock 23:36, mid-run, at a step boundary with NO traceback and NO DONE marker -> classic PARENT-DEATH (launched inline, not nohup'd; the launching shell/session terminated and took them down). No OOM in journalctl/dmesg. NOT a -training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** +training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** (The nohup lesson still +stands as good practice, but it was not the cause here -- there was no death.) **Interim signal BEFORE they died (the science):** at L12 the EP estimator degrades with training in a way it did NOT at L6: - EP s1: best val 2.0444 @ step 2800, then val BOUNCED to 2.0951 @3200 (last line); cos(EP,BP) |