summaryrefslogtreecommitdiff
path: root/docs/campaign
diff options
context:
space:
mode:
authorYuren Hao <yurenh2@illinois.edu>2026-07-10 02:02:42 -0500
committerYuren Hao <yurenh2@illinois.edu>2026-07-10 02:02:42 -0500
commitf5861055f40a45805bc1d34f36fd1c005ad818b6 (patch)
treeda9b32e87fc15af728beebc7a5d8c827dfca5138 /docs/campaign
parent28a1566da92285fb2d990cd172fafae5cc0aec33 (diff)
CORRECTION: D1a did not die (misread mangled proc scan); originals completed s1 1.9745/s2 2.0013/s3 2.1188 + muon 2.7515; beta-floor fix independently found on free GPU1
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'docs/campaign')
-rw-r--r--docs/campaign/CASCADE_ABLATION_PLAN.md20
1 files changed, 18 insertions, 2 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index ede9c39..a4fab80 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -36,10 +36,26 @@ The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether
- E-tier: next in queue (softmax pathology / error-channel SNR / write pricing) -> Demo-0 spec sheet.
## D1a AUTOPSY + K-LADDER DIAGNOSTIC (2026-07-09 night)
-**What happened:** the 4 D1a arms (d1_ep_s1/s2/s3 + d1_bp_s3) all died at wall-clock 23:36, mid-run,
+**>>> CORRECTION (2026-07-10 02:xx): the "parent-death" below was a MISDIAGNOSIS. <<<**
+The original D1a arms did NOT die -- they completed normally. When I checked at ~23:44 they were ALIVE
+at step 3200 on GPUs 0/3 (both at 100%); my /proc scan was mangled by a zsh eval wrapper so I misread
+"no casc alive", and GPU1 being free (11 MiB) fooled me (the runs were on 0/3, not 1). d1_ep_s1.log is
+continuous 0->4000 at steady 0.679 it/s (3200->4000 = 19.6 min, matches its 00:06 mtime). **Original
+D1a finals: d1_ep_s1 1.9745 / s2 2.0013 / s3 2.1188 (fixed K3; s2 blew at step 4000 skips=9, s3 blew
+hard skips=23 governor ramped K->7); d1_ep_muon 2.7515 (Muon-on-EP, cos collapsed 0.82).** The d1b
+experiments I launched (thinking the originals died) ran on the GENUINELY-FREE GPU1, so no competition
+-- and they independently isolated the real mechanism + fix (below), which is the bigger prize. Net:
+no harm, wrong death-story, and we now have BOTH the original un-floored 3-seed AND the beta-floor fix.
+KEY read of the original 3-seed: un-floored K3 is HIGH-VARIANCE near the SNR cliff -- s1 got lucky and
+stayed stable (1.9745, closest to BP), s2/s3 blew up late. Same-seed non-determinism (fb+autograd
+reductions) means the un-floored estimator is not even reproducible near the cliff. That is the
+strongest argument FOR the beta-floor (which pins cos=1.0000, stable, reproducible).
+
+**What I ORIGINALLY (wrongly) concluded:** the 4 D1a arms all died at wall-clock 23:36, mid-run,
at a step boundary with NO traceback and NO DONE marker -> classic PARENT-DEATH (launched inline, not
nohup'd; the launching shell/session terminated and took them down). No OOM in journalctl/dmesg. NOT a
-training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.**
+training failure. **Lesson (re)applied: every relaunch is nohup + </dev/null.** (The nohup lesson still
+stands as good practice, but it was not the cause here -- there was no death.)
**Interim signal BEFORE they died (the science):** at L12 the EP estimator degrades with training in a
way it did NOT at L6:
- EP s1: best val 2.0444 @ step 2800, then val BOUNCED to 2.0951 @3200 (last line); cos(EP,BP)