QK-norm: replace nn.MHA with explicit SDPA attn + --qk_norm (OLMo2-style, analog-friendly); cancel epoch, insert 8-run qk validation, stage roadmap TinyStories-epoch->FineWeb-Edu->OLMo2

1 files changed, 36 insertions, 0 deletions

diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index 279b2fe..585ecc1 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -231,3 +231,39 @@ they carry the dynamics paper + the two-stage-recipe science; D1 takes over the
   demo (task #15). BP twin epoch DEFERRED (no free GPU; parity already sealed so it is nice-to-have).
 - Next: generation samples at checkpoints; BP-twin epoch when a GPU frees; then scale-up corpus
   decision (FineWeb-Edu vs OLMo2/Dolma) for the larger model.
+
+## ROADMAP PIVOT (2026-07-10 03:2x, user directive): QK-norm inserted; staged scale-up.
+**User: cancel the full epoch (done — killed epoch_ep_bf3e4); insert a QK-norm version after the
+current 3-seed; then stages TinyStories-full-epoch -> FineWeb-Edu -> OLMo2.**
+
+**Why QK-norm:** RMS-normalize q,k per head before the scores (OLMo2/Llama-style). It BOUNDS the
+attention logits, attacking the SAME root cause as the beta-floor (sig_tok growth -> logit blowup ->
+finite-beta SNR collapse) but structurally. Analog-friendly (my analysis): it's divisive
+normalization (mature analog/neuromorphic primitive), its Jacobian is symmetric (does NOT worsen the
+PAR/non-reciprocity wall), it's feedforward (no digital root-finder / no adjoint), and it REUSES the
+softmax current-normalization circuitry (reuse doctrine, no tapeout). Bonus analog wins: bounds the
+input range of the analog softmax exp device; reduces sig-growth so relaxation is more robust.
+Analog-preferred alternative to A/B in E-tier: tanh logit soft-cap (tanh is a native analog transfer
+function -- possibly cheaper than the norm's square-sum+divide).
+
+**Code:** nn.MultiheadAttention replaced by explicit CausalSelfAttn (SDPA-backed, fast) in BOTH
+trainers; `--qk_norm` flag (RMS-norm over head_dim w/ learnable per-dim gain). Smoke: EP+qk_norm
+cos=1.0000, 40.06M preserved, 2.49 it/s, SDPA works in the fb backward (fb is first-order, no
+double-backward needed). Also added `--cosine` (warmup->cosine to 0.1x lr) for the long runs.
+
+**QK-norm validation matrix (8 runs, L12 C512, 4000 steps, launched on GPU1):**
+  - qk_bp_s1/s2/s3  = BP + qk_norm (new reference with the new block)
+  - qk_ep_bf_s1/s2/s3 = EP + qk_norm + beta_floor 3e-4 (PARITY test vs qk_bp)
+  - qk_ep_nf_s1/s2  = EP + qk_norm, NO beta_floor (ANALOG test: does qk_norm ALONE hold cos, letting
+    us DROP the beta-floor? un-floored non-qk collapsed to cos 0.896 by step 4000 -- see RESULT 2).
+Decision: (1) qk_ep_bf ~ qk_bp => parity preserved with qk_norm. (2) if qk_ep_nf ALSO holds cos~1 and
+matches => qk_norm supersedes the beta-floor (fewer knobs, cleaner analog story). Watcher qk_watch.sh
+fires at the early analog read (nf step 2500) or all-done.
+
+**STAGED SCALE-UP (after qk_norm validates):**
+  Stage 1: TinyStories FULL EPOCH (58,800 steps, 361M tok) with the validated qk_norm recipe + cosine
+           -> the "neng kan" generation demo (task #15).
+  Stage 2: FineWeb-Edu (real corpus, 32-50k tokenizer, ~150-300M params) -- best small-LM quality.
+  Stage 3: OLMo2 / Dolma recipe -- fully-open reproducible baseline for the paper/collaborators.
+  EP scaling knobs carried forward: beta_floor (or qk_norm if it supersedes), possibly double-sided
+  nudge at larger scale (cancels O(beta) Taylor bias). $20k/run (Rain) ~ few-B tokens/run.