diff options
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 16 |
1 files changed, 16 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 585ecc1..8e0ac43 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -267,3 +267,19 @@ fires at the early analog read (nf step 2500) or all-done. Stage 3: OLMo2 / Dolma recipe -- fully-open reproducible baseline for the paper/collaborators. EP scaling knobs carried forward: beta_floor (or qk_norm if it supersedes), possibly double-sided nudge at larger scale (cancels O(beta) Taylor bias). $20k/run (Rain) ~ few-B tokens/run. + +### RESULT 4 (2026-07-10 06:16): QK-norm validated — parity holds; beta-floor still needed; Stage 1 launched. +- **Parity with QK-norm (EP-favorable again):** BP+qknorm 1.9253/1.8753/1.9192 = 1.9066; + EP+qknorm+beta_floor 1.8588/1.9176/1.8841 = **1.8868 <= BP**. QK-norm preserves EP=BP parity at L12. +- **ANALOG ANSWER: QK-norm does NOT replace the beta-floor** (they are complementary). EP+qknorm + WITHOUT the floor still erodes cos (1.0 -> 0.946 by step 3200) and lands ~0.09 worse CE (2.02 vs + 1.89). Milder than the old non-QK collapse (0.896) but not fixed. **Why: sig_tok still grows to 21.5 + even with QK-norm** -- QK-norm normalizes q,k INSIDE attention (bounds the attention LOGITS) but does + NOT bound the residual/embedding scale that drives beta_t = beta0*(sig0/sig)^2. So beta_t still + collapses -> estimator SNR still needs the floor. QK-norm's payoff is (a) attention logit-bounding + (analog softmax device range), (b) scale robustness (logit growth is worse in bigger/deeper models), + (c) it is standard OLMo2/Llama -> good for the scale-up. Recipe = **qk_norm + beta_floor together**. +- **STAGE 1 LAUNCHED (user directive):** stage1_ep_qkbf -- TinyStories full epoch (58,800 steps, 361M + tok), qk_norm + beta_floor 3e-4 + cosine, warmup 500, 2.4 it/s solo -> ~6.8 h. The "neng kan" + generation demo. Watcher fires at step 10000 (first generation-worthy ckpt) / done / death. + Then Stage 2 (FineWeb-Edu) -> Stage 3 (OLMo2). |