diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-10 08:00:21 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-10 08:00:21 -0500 |
| commit | 35a9228dde348705040e4149f4da2f59fd37b9a8 (patch) | |
| tree | ba914cb2985e05af9e191c37d70fec7c80612a62 /docs/campaign | |
| parent | be85b470845ad19669259f10324a1a8877c8970e (diff) | |
Stage-1 epoch blowup @12100: sig story REFUTED (only +8%, cos fine till after); leading indicator=drift-guard skips -> contractivity bifurcation in nudged relaxation (cascade Hopf wall); add resume/sig0/final_ln; A/B/C diagnostic launched
Diffstat (limited to 'docs/campaign')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 22 |
1 files changed, 22 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 8e0ac43..19f97bb 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -283,3 +283,25 @@ fires at the early analog read (nf step 2500) or all-done. tok), qk_norm + beta_floor 3e-4 + cosine, warmup 500, 2.4 it/s solo -> ~6.8 h. The "neng kan" generation demo. Watcher fires at step 10000 (first generation-worthy ckpt) / done / death. Then Stage 2 (FineWeb-Edu) -> Stage 3 (OLMo2). + +### RESULT 5 (2026-07-10 07:3x): Stage-1 epoch BLEW UP @step 12100 — root-cause diagnosis (sig story REFUTED). +The qk_norm+beta_floor+cosine epoch was healthy to ~11400 (best val 1.6669) then blew up (val 1.67->7.6, +gn pre-clip 0.5->53) and oscillated in a degraded regime. **My first guess (sig_tok growth -> SNR +collapse -> add final_ln) was WRONG, refuted by its own telemetry:** +- sig rose only +8% (29.8@10000 -> 32.2@12000) then PLATEAUED; it was already ~30 at step 10000 when + everything was healthy. An 8% change cannot cause a catastrophic transition. +- cos was FINE (0.9935) until step 11900; the cos drop is a CONSEQUENCE of the blowup, not the cause. +- grad-clip is ALREADY present (clip 1.0); gn=53 is pre-clip telemetry. Not a magnitude-spike issue. +**LEADING INDICATOR = skips (drift-guard rejections = nudged fb relaxation drift>0.5 = CONVERGENCE +FAILURE).** skips accelerate from ~step 11000 (4->13 by 11400) BEFORE gn (11700), cos (12000), val +(12100). **Diagnosis: a CONTRACTIVITY BIFURCATION in the nudged fb relaxation** -- as training sharpens +the operator (block Jacobians grow), an increasing fraction of batches have a non-contractive nudged +iteration -> skipped -> gradient bias -> a marginally-converged batch emits a bad step -> over the edge. +**This is the cascade analog of the looped-EP Hopf wall** (non-conservative attention loses +contractivity as CE drops -- documented in ep-c512-residual-defense-fix). 4000-step runs never saw it +(operator not sharp enough yet; edge ~step 11400). Right fix = CONTRACTIVITY control (resreg/jacreg or +geta<1 damping), NOT final_ln. +**CONFIRMATORY A/B/C (resume from ckpt-10000, pre-bifurcation, beta floored 3e-4 via --sig0 1.6):** + A=control (K3,lr1e-3) -> should reproduce skip-climb+blowup; B=K8 (does more fb rounds hold skips? + = marginal-contractivity test); C=lr3e-4 (slower sharpening -> delayed edge? = driver test). + Code added: --resume, --sig0, --final_ln, --qk_norm(CausalSelfAttn/SDPA). Watcher diag_watch.sh armed. |
