1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
|
# FW135M BP Twin Run Sheet
Source of truth: `docs/BASELINE_SPEC.md`.
Status: commands configured on branch `xiang`; no training has started.
## First width-only rung
- Shape: `L12/C768/H12/T256`
- Head dimension: `64`
- Exact local parameter count: `135,303,936`
- Data: existing FineWeb-Edu 32k bins
- Batch: `B24`
- Target: `20N = 2,706,078,720` tokens
- EP-matched complete-batch exposure: `2,703,366,144` tokens
- Trainer argument: `--steps 440000` (440,001 inclusive updates, matching `fw135m_bsign`)
## What changes from 72M
- Width: `512 -> 768`
- Heads: `8 -> 12`
- Parameters: `72.11M -> 135.30M`
- Run length: increases to preserve approximately 20 tokens/parameter
## What stays identical
- L12 depth and 64-dimensional heads
- T256 and B24
- FineWeb-Edu data and 32k tokenizer
- OLMo2-style model implementation
- Muon hybrid optimizer and cosine schedule
- 1,000-step warmup, matching `fw135m_bsign`
- Weight decay 0.1 and BF16 autocast
- Seed list and validation cadence
- Existing `casc_bp_train.py` code path
## LR sweep
At this smallest new rung only:
- `7e-4`
- `1e-3`
- `1.4e-3`
Only Adam-side LR changes. Carry the winner to larger rungs.
No result may be quoted with fewer than two seeds. Record best validation CE, final-step CE, and tail median. Save every 5,000 steps and log validation every 100 steps.
## Dry-run launcher
From `ep_run/`:
```bash
python -m py_compile fw135m_baseline.py
python fw135m_baseline.py --mode smoke
python fw135m_baseline.py --mode sweep
python fw135m_baseline.py --mode full
```
Commands print by default. Add `--execute` only when ready.
Smoke:
```bash
python fw135m_baseline.py --mode smoke --execute
```
LR sweep:
```bash
python fw135m_baseline.py --mode sweep --execute
```
Selected full run:
```bash
python fw135m_baseline.py --mode full --lr <selected-lr> --seed 1 --execute
python fw135m_baseline.py --mode full --lr <selected-lr> --seed 2 --execute
```
W&B remains off until configured. Later:
```bash
python fw135m_baseline.py --mode full --lr <selected-lr> \
--wandb_project <project> --execute
```
## Final EP/BP comparison
The EP arm must use the same shape, data, B/T, steps, optimizer family, selected shared settings, seeds, and evaluation cadence. Only the training rule and EP-specific `β` controls differ.
|
