From ca209ee5df16174d2137b5d5922581fc2f6d7e88 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Tue, 21 Jul 2026 11:37:38 -0500 Subject: =?UTF-8?q?BASELINE=5FSPEC:=20BP=E5=AD=AA=E7=94=9F=E5=A5=91?= =?UTF-8?q?=E7=BA=A6(=E5=94=AF=E4=B8=80=E8=87=AA=E7=94=B1=E5=BA=A6=3D?= =?UTF-8?q?=E6=9C=80=E5=B0=8F=E6=A1=A3lr=E4=B8=89=E7=82=B9=E6=89=AB,n>=3D2?= =?UTF-8?q?)+=C3=972=E9=98=B6=E6=A2=AF=E5=BD=A2=E7=8A=B6=E8=A1=A8(135M/270?= =?UTF-8?q?M=E5=AE=BD=E5=BA=A6=E4=BC=98=E5=85=88@L12,550M/1.1B=E6=B7=B1?= =?UTF-8?q?=E5=BA=A6=E4=BA=8B=E4=BB=B6=E5=B8=A6re-gate)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/BASELINE_SPEC.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 docs/BASELINE_SPEC.md (limited to 'docs') diff --git a/docs/BASELINE_SPEC.md b/docs/BASELINE_SPEC.md new file mode 100644 index 0000000..5c5abc5 --- /dev/null +++ b/docs/BASELINE_SPEC.md @@ -0,0 +1,50 @@ +# Baseline Spec (BP twins for the scaling ladder) + +The gap numbers are only meaningful against a *twin*, not "a BP run". This file is the +contract. Entry doc for the repo: `ONBOARDING.md` (clone + `python pull_assets.py` restores +data/ckpts). + +## The twin contract (everything identical except the training rule) + +- **Code path**: use `ep_run/casc_bp_train.py` as-is — it already shares model classes, + data loader, and eval with the EP trainer, and `--L --C --H --T` are flags. For the ladder + you normally write **no new code**; the work is the run matrix + bookkeeping. +- **Identical**: architecture (`--olmo2`: norm-after-sublayer RMSNorm, full-width QK-norm, + RoPE 5e5, SwiGLU, no biases, untied head, 0.02 init), tokenizer (FineWeb-Edu 32k, + `data/fineweb_edu/{train,val}.bin`), batch geometry (B, T=256), steps, optimizer family + (Muon hybrid + cosine, `--wd 0.1`), seed list, eval cadence (`val` every 100 steps on the + same val.bin), wandb logging (project `ept-fineweb-*`, run name = tag). +- **BP's one degree of freedom**: a 3-point lr sweep around the EP-matched point + (x0.7 / x1.0 / x1.4) at the SMALLEST new rung only; carry the winner up the ladder. + At 42M the plateau was flat across 2x (1.2119/1.2311/1.2136) — verify once per family, + don't re-sweep every rung. This is the agreed fairness protocol. +- **Seeds**: n>=2 before any number is quoted anywhere (three single-seed crowns have been + retired this campaign; dip-statistics bites both directions). +- **Report per run**: best val CE, final-step CE, and tail-median; save ckpts `--save_every + 5000` (probes need the ladder). Results go into `docs/campaign/CASCADE_ABLATION_PLAN.md` + as a RESULT entry (see existing entries for the format) + wandb. + +## Optional second baseline (sanity, not twin) + +A standard-recipe reference (AdamW β=(0.9,0.95), wd 0.1, cosine, GPT init) on the same +data/eval answers "is the Muon twin itself weird". New file in `ep_run/`, reuse the data +loader and eval verbatim. Lower priority than the twins. + +## Planned rungs (x2 total params per step; hd=64 everywhere, vocab 32k fixed) + +| rung | shape | params (blocks + emb/head) | tokens (20/param) | note | +|---|---|---|---|---| +| 72M | L12 C512 H8 | 38.5 + 33.6 = 72.1M | 1.4B | done (crowns sealed) | +| 135M | L12 C768 H12 | 85 + 50 = 135M | 2.7B | width-only | +| 270M | L12 C1152 H18 | 191 + 76 = 267M | 5.4B | width-only | +| 550M | L16 C1536 H24 | 453 + 101 = 553M | 11B | **depth event** (L12→16), gated | +| 1.1B | L20 C2048 H32 | 1004 + 134 = 1138M | 22B | **depth event** (L16→20), gated | + +Width-first while L=12 because the EP stability corpus (ceiling measurements, bias +magnitudes, K=3 saturation) was all measured at L12 and depth sits in the loop-gain +exponent; each **depth event** requires a cheap re-gate before the long run: rho-grid probe +at the new depth (ceiling at production beta) + K-sweep (is K=3 still saturated) — if K must +grow with L, the cost model changes and must be re-quoted BEFORE any external commitment. + +EP-side recipe per rung (for reference, not the baseline writer's job): current 1.0x +candidate = `--bsign_rand` + fixed beta 3e-3 + guards silent, pending seed-2 confirmation. -- cgit v1.2.3