diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/BASELINE_SPEC.md | 50 |
1 files changed, 50 insertions, 0 deletions
diff --git a/docs/BASELINE_SPEC.md b/docs/BASELINE_SPEC.md new file mode 100644 index 0000000..5c5abc5 --- /dev/null +++ b/docs/BASELINE_SPEC.md @@ -0,0 +1,50 @@ +# Baseline Spec (BP twins for the scaling ladder) + +The gap numbers are only meaningful against a *twin*, not "a BP run". This file is the +contract. Entry doc for the repo: `ONBOARDING.md` (clone + `python pull_assets.py` restores +data/ckpts). + +## The twin contract (everything identical except the training rule) + +- **Code path**: use `ep_run/casc_bp_train.py` as-is — it already shares model classes, + data loader, and eval with the EP trainer, and `--L --C --H --T` are flags. For the ladder + you normally write **no new code**; the work is the run matrix + bookkeeping. +- **Identical**: architecture (`--olmo2`: norm-after-sublayer RMSNorm, full-width QK-norm, + RoPE 5e5, SwiGLU, no biases, untied head, 0.02 init), tokenizer (FineWeb-Edu 32k, + `data/fineweb_edu/{train,val}.bin`), batch geometry (B, T=256), steps, optimizer family + (Muon hybrid + cosine, `--wd 0.1`), seed list, eval cadence (`val` every 100 steps on the + same val.bin), wandb logging (project `ept-fineweb-*`, run name = tag). +- **BP's one degree of freedom**: a 3-point lr sweep around the EP-matched point + (x0.7 / x1.0 / x1.4) at the SMALLEST new rung only; carry the winner up the ladder. + At 42M the plateau was flat across 2x (1.2119/1.2311/1.2136) — verify once per family, + don't re-sweep every rung. This is the agreed fairness protocol. +- **Seeds**: n>=2 before any number is quoted anywhere (three single-seed crowns have been + retired this campaign; dip-statistics bites both directions). +- **Report per run**: best val CE, final-step CE, and tail-median; save ckpts `--save_every + 5000` (probes need the ladder). Results go into `docs/campaign/CASCADE_ABLATION_PLAN.md` + as a RESULT entry (see existing entries for the format) + wandb. + +## Optional second baseline (sanity, not twin) + +A standard-recipe reference (AdamW β=(0.9,0.95), wd 0.1, cosine, GPT init) on the same +data/eval answers "is the Muon twin itself weird". New file in `ep_run/`, reuse the data +loader and eval verbatim. Lower priority than the twins. + +## Planned rungs (x2 total params per step; hd=64 everywhere, vocab 32k fixed) + +| rung | shape | params (blocks + emb/head) | tokens (20/param) | note | +|---|---|---|---|---| +| 72M | L12 C512 H8 | 38.5 + 33.6 = 72.1M | 1.4B | done (crowns sealed) | +| 135M | L12 C768 H12 | 85 + 50 = 135M | 2.7B | width-only | +| 270M | L12 C1152 H18 | 191 + 76 = 267M | 5.4B | width-only | +| 550M | L16 C1536 H24 | 453 + 101 = 553M | 11B | **depth event** (L12→16), gated | +| 1.1B | L20 C2048 H32 | 1004 + 134 = 1138M | 22B | **depth event** (L16→20), gated | + +Width-first while L=12 because the EP stability corpus (ceiling measurements, bias +magnitudes, K=3 saturation) was all measured at L12 and depth sits in the loop-gain +exponent; each **depth event** requires a cheap re-gate before the long run: rho-grid probe +at the new depth (ceiling at production beta) + K-sweep (is K=3 still saturated) — if K must +grow with L, the cost model changes and must be re-quoted BEFORE any external commitment. + +EP-side recipe per rung (for reference, not the baseline writer's job): current 1.0x +candidate = `--bsign_rand` + fixed beta 3e-3 + guards silent, pending seed-2 confirmation. |
