1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
|
# Baseline Spec (BP twins for the scaling ladder)
The gap numbers are only meaningful against a *twin*, not "a BP run". This file is the
contract. Entry doc for the repo: `ONBOARDING.md` (clone + `python pull_assets.py` restores
data/ckpts).
## The twin contract (everything identical except the training rule)
- **Code path**: use `ep_run/casc_bp_train.py` as-is — it already shares model classes,
data loader, and eval with the EP trainer, and `--L --C --H --T` are flags. For the ladder
you normally write **no new code**; the work is the run matrix + bookkeeping.
- **Identical**: architecture (`--olmo2`: norm-after-sublayer RMSNorm, full-width QK-norm,
RoPE 5e5, SwiGLU, no biases, untied head, 0.02 init), tokenizer (FineWeb-Edu 32k,
`data/fineweb_edu/{train,val}.bin`), batch geometry (B, T=256), steps, optimizer family
(Muon hybrid + cosine, `--wd 0.1`), seed list, eval cadence (`val` every 100 steps on the
same val.bin), wandb logging (project `ept-fineweb-*`, run name = tag).
- **BP's one degree of freedom**: a 3-point lr sweep around the EP-matched point
(x0.7 / x1.0 / x1.4) at the SMALLEST new rung only; carry the winner up the ladder.
At 42M the plateau was flat across 2x (1.2119/1.2311/1.2136) — verify once per family,
don't re-sweep every rung. This is the agreed fairness protocol.
- **Seeds**: n>=2 before any number is quoted anywhere (three single-seed crowns have been
retired this campaign; dip-statistics bites both directions).
- **Report per run**: best val CE, final-step CE, and tail-median; save ckpts `--save_every
5000` (probes need the ladder). Results go into `docs/campaign/CASCADE_ABLATION_PLAN.md`
as a RESULT entry (see existing entries for the format) + wandb.
## First task (warm-up, closes an open hole)
The 72M BP twin is n=1 (best 3.2884, seed 1). Run seeds 2 and 3 with the exact original
config (`--L 12 --C 512 --H 8 --T 256 --B 24 --lr 1e-3 --opt muon --cosine --steps 234000
--warmup 1000 --olmo2 --wd 0.1 --amp --save_every 5000`, seeds 2/3) — ~11-13h each on one
H100/A6000-class GPU. This sets the honest-gap denominator for every 72M claim.
## Optional second baseline (sanity, not twin)
A standard-recipe reference (AdamW β=(0.9,0.95), wd 0.1, cosine, GPT init) on the same
data/eval answers "is the Muon twin itself weird". New file in `ep_run/`, reuse the data
loader and eval verbatim. Lower priority than the twins.
## Planned rungs (x2 total params per step; hd=64 everywhere, vocab 32k fixed)
| rung | shape | params (blocks + emb/head) | tokens (20/param) | note |
|---|---|---|---|---|
| 72M | L12 C512 H8 | 38.5 + 33.6 = 72.1M | 1.4B | done (crowns sealed) |
| 135M | L12 C768 H12 | 85 + 50 = 135M | 2.7B | width-only |
| 270M | L12 C1152 H18 | 191 + 76 = 267M | 5.4B | width-only |
| 550M | L16 C1536 H24 | 453 + 101 = 553M | 11B | **depth event** (L12→16), gated |
| 1.1B | L20 C2048 H32 | 1004 + 134 = 1138M | 22B | **depth event** (L16→20), gated |
Width-first while L=12 because the EP stability corpus (ceiling measurements, bias
magnitudes, K=3 saturation) was all measured at L12 and depth sits in the loop-gain
exponent; each **depth event** requires a cheap re-gate before the long run: rho-grid probe
at the new depth (ceiling at production beta) + K-sweep (is K=3 still saturated) — if K must
grow with L, the cost model changes and must be re-quoted BEFORE any external commitment.
EP-side recipe per rung (for reference, not the baseline writer's job): current 1.0x
candidate = `--bsign_rand` + fixed beta 3e-3 + guards silent, pending seed-2 confirmation.
|
