diff options
Diffstat (limited to 'docs/campaign/CASCADE_ABLATION_PLAN.md')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 22 |
1 files changed, 22 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 8e0ac43..19f97bb 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -283,3 +283,25 @@ fires at the early analog read (nf step 2500) or all-done. tok), qk_norm + beta_floor 3e-4 + cosine, warmup 500, 2.4 it/s solo -> ~6.8 h. The "neng kan" generation demo. Watcher fires at step 10000 (first generation-worthy ckpt) / done / death. Then Stage 2 (FineWeb-Edu) -> Stage 3 (OLMo2). + +### RESULT 5 (2026-07-10 07:3x): Stage-1 epoch BLEW UP @step 12100 — root-cause diagnosis (sig story REFUTED). +The qk_norm+beta_floor+cosine epoch was healthy to ~11400 (best val 1.6669) then blew up (val 1.67->7.6, +gn pre-clip 0.5->53) and oscillated in a degraded regime. **My first guess (sig_tok growth -> SNR +collapse -> add final_ln) was WRONG, refuted by its own telemetry:** +- sig rose only +8% (29.8@10000 -> 32.2@12000) then PLATEAUED; it was already ~30 at step 10000 when + everything was healthy. An 8% change cannot cause a catastrophic transition. +- cos was FINE (0.9935) until step 11900; the cos drop is a CONSEQUENCE of the blowup, not the cause. +- grad-clip is ALREADY present (clip 1.0); gn=53 is pre-clip telemetry. Not a magnitude-spike issue. +**LEADING INDICATOR = skips (drift-guard rejections = nudged fb relaxation drift>0.5 = CONVERGENCE +FAILURE).** skips accelerate from ~step 11000 (4->13 by 11400) BEFORE gn (11700), cos (12000), val +(12100). **Diagnosis: a CONTRACTIVITY BIFURCATION in the nudged fb relaxation** -- as training sharpens +the operator (block Jacobians grow), an increasing fraction of batches have a non-contractive nudged +iteration -> skipped -> gradient bias -> a marginally-converged batch emits a bad step -> over the edge. +**This is the cascade analog of the looped-EP Hopf wall** (non-conservative attention loses +contractivity as CE drops -- documented in ep-c512-residual-defense-fix). 4000-step runs never saw it +(operator not sharp enough yet; edge ~step 11400). Right fix = CONTRACTIVITY control (resreg/jacreg or +geta<1 damping), NOT final_ln. +**CONFIRMATORY A/B/C (resume from ckpt-10000, pre-bifurcation, beta floored 3e-4 via --sig0 1.6):** + A=control (K3,lr1e-3) -> should reproduce skip-climb+blowup; B=K8 (does more fb rounds hold skips? + = marginal-contractivity test); C=lr3e-4 (slower sharpening -> delayed edge? = driver test). + Code added: --resume, --sig0, --final_ln, --qk_norm(CausalSelfAttn/SDPA). Watcher diag_watch.sh armed. |