diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-10 11:13:40 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-10 11:13:40 -0500 |
| commit | 9c2d7f74d30c637d55b0b34fb5fe830318c0b47c (patch) | |
| tree | 12e2d535ffbf833a9f52e0afef8fab6711952ed1 /docs | |
| parent | d6e51d2f33e5aaaf4d016c7009b58cfd6c74e4d1 (diff) | |
RESULT 6 addendum: K8 also blew at 12600 -> wall-2 is true contractivity crossing (K delays only); kretry demoted to mitigation; defense = OLMo2 arch > lr channel > damped-fb/jacreg
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 15 |
1 files changed, 15 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 2cc9f05..ea3b40f 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -366,3 +366,18 @@ rounds ONLY on marginal batches (~0.1-1% of steps). Telemetry: skips=(d/g/r). step-0 losses bitwise-identical per seed). Watcher auto-computes parity and — if EP mean within 0.05 of BP — AUTO-LAUNCHES the Stage-1 OLMo2 TinyStories epoch (stage1_ol_ep, 58.8k steps, kretry armed). OLMo2's bounded-per-branch signals may also shift wall-2 later; kretry is the belt-and-suspenders. + +### RESULT 6-ADDENDUM (2026-07-10 10:5x): B(K8) ALSO BLEW at matched step — K delays, does NOT prevent. +diag_B_k8 @12600: train 3.37 / val 3.63 (best 1.6755 pre-blowup), gn 18.6, skips 2->14. So wall-2 is +NOT merely marginal under-convergence: the nudged fb iteration becomes GENUINELY DIVERGENT for a +growing batch fraction as the operator sharpens (true contractivity crossing — the cascade Hopf wall). +More rounds converge the marginal shell only; once past the edge no K helps. **kretry DEMOTED from fix +to mitigation** (still right for sporadic healthy-regime rejections). Note also: B blew with only 14 +total rejections => most bad gradients passed UNDER the drift-0.5 threshold (loose guard + gn-EMA +poisoning during degradation). +Surviving facts: C (lr 3e-4) clean at 12600 (skips=1) -> sharpening RATE is the driver; D (BP) clean +-> EP-specific. **Defense ranking now: (1) OLMo2 arch (different operator: bounded branches + QK-norm; +diagnostics were all on the OLD arch) -> (2) lr channel (lower peak / faster decay through the +mid-training danger window) -> (3) true contractivity control (damped-fb gamma<1 / cascade-jacreg) if +OLMo2 still hits the wall.** Stage-1 OLMo2 epoch (parity-gated autolaunch) is the live test; watch +skips=(d/g/r) through the 10-14k window. |