1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
# FW135M BP Sweep Handoff
The 135M BP smoke test completed on an RTX A6000:
- `fw135m_bp_smoke_s1`
- 400/400 steps completed; best validation CE `5.7126`
- W&B: `eqprop-llm-training/ept-fineweb-135M`
The BP sweep is matched to the active `fw135m_bsign` EP run in all shared
settings: `L12/C768/H12/T256/B24`, FineWeb-Edu 32k, OLMo2, Muon
(`--muon_lr 0.02`), BF16, weight decay `0.1`, cosine to `0.1×`, 440,000
trainer steps, 1,000 warmup steps, and NCCL data parallelism. `B24` is per
rank, so four A6000s use effective batch `96`. The intentional difference is
BP versus EP's `--bsign_rand --beta 0.003`.
## Run one candidate
From `ep_run/`, each candidate should use all four A6000s:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 7e-4 1
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1e-3 1
CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1.4e-3 1
```
Run the three commands sequentially when only four GPUs are available. Each
candidate gets a distinct W&B run name:
`fw135m_bp_lr7em4_s1`, `fw135m_bp_lr1em3_s1`, or `fw135m_bp_lr1p4em3_s1`.
The launcher defaults data to `ep_run/data/fineweb_edu`. Set
`EPT_DATA_ROOT=/path/to/data` only when FineWeb data is stored elsewhere.
Choose the LR by best validation CE, final validation CE, and tail-median CE.
Then run the selected BP setting for seeds 1 and 2 before reporting a BP/EP
comparison.
|
