summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorYuren Hao <yurenh2@illinois.edu>2026-07-24 18:59:55 -0500
committerYuren Hao <yurenh2@illinois.edu>2026-07-24 18:59:55 -0500
commit86ef095e9619c3c6908cb2c95e1cd27532ff2904 (patch)
tree44321bd3621f15f8df5addaa90404ad92994829d
parent27560970e2d967e62d22b8693811d7301fbc290d (diff)
RESULT 60b: res表在amp下宽度盲(bf16地板把fp32四个量级压成1.2×,棘轮首步死亡螺旋); 带内自适应在C768+amp判死,仪器=离线fp32探针; fw135m_b15(定点1.5e-3+bsign)+ckpt哨戒探针在飞; res/rho入wandb
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
-rw-r--r--docs/campaign/CASCADE_ABLATION_PLAN.md15
-rw-r--r--ep_run/casc_eq_train.py1
2 files changed, 16 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index 5bb1d73..68d7bc2 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -568,6 +568,21 @@ direction), not training-under-fault; wave-2 = co-training with faults injected
+### RESULT 60b (2026-07-24): THE RES METER IS WIDTH-BLIND UNDER AMP — in-band ratcheting
+is impossible at C768; the instrument is the offline fp32 probe. Ratchet arm killed at launch
+(6 knocks in the first step, beta -> 4.7e-5 death spiral: gate 1e-3 sat BELOW the bf16 floor).
+Measured amp-ensemble in-run residual (resdiag, 60-step diagnostics): healthy s250000 median
+9.9e-3 vs dead-corridor s345000 median 1.2e-2 — the fp32 4-decade separation (1.3e-6 vs
+2.6e-4+) collapses to 1.2x under bf16 forward noise. The old 0.02 constant "worked" at C512
+by sitting above the amp floor; at C768 NO res threshold separates healthy from sick. The
+fp32-probe-vs-amp-trainer ensemble mismatch strikes again (BBP lesson, meter edition).
+CONTINUATION: fw135m_b15 — s250000, FIXED beta 1.5e-3 (3x margin at 250k ceiling ~5e-3;
+in-corridor at 300k ~3e-3; damaged-lineage endgame reading 1-1.5e-3 = lower bound), bsign,
+res_gate 0.02 as NaN backstop only, 3-GPU, ~27h. SENTINEL: every new ckpt gets an fp32
+rho-probe at {1.5, 2.25, 3}e-3 — alert if the operating row goes DIVERGE (the calibrated
+instrument, out-of-band). Beta policy doctrine for the ladder: measured fixed points +
+offline-probe surveillance; in-band adaptive control is dead at width under amp.
+
### RESULT 60 (2026-07-24): s345000 CORRIDOR IS EMPTY (end075 3.4805 < original, worse);
closure located at ~275-300k; CONTINUATION = bsign+ratchet+res_gate from s250000. Two code
fixes en route (sign-wipe combo bug + configurable gate), both fire-certified.
diff --git a/ep_run/casc_eq_train.py b/ep_run/casc_eq_train.py
index 60b6113..0cd5521 100644
--- a/ep_run/casc_eq_train.py
+++ b/ep_run/casc_eq_train.py
@@ -908,6 +908,7 @@ for step in range(start_step, args.steps + 1):
with torch.no_grad():
GOV['w_rms'] = float(sum(p.float().pow(2).mean().sqrt() for p in all_params) / len(all_params))
_aux = {'gn': GOV.get('gn'), 'drift': GOV.get('drift'), 'sig': GOV.get('sig'),
+ 'res': GOV.get('res'), 'rho': GOV.get('rho'),
'clip_norm': GOV.get('clip_norm'), 'clip_fired': (None if GOV.get('clip_norm') is None else float(GOV['clip_norm'] > 1.0))}
for k in ('act_rms', 'w_rms', 'mag_ep', 'mag_bp'):
if GOV.get(k) is not None: _aux[k] = GOV[k]