diff options
Diffstat (limited to 'docs/campaign/FW135M_BP_HANDOFF.md')
| -rw-r--r-- | docs/campaign/FW135M_BP_HANDOFF.md | 35 |
1 files changed, 35 insertions, 0 deletions
diff --git a/docs/campaign/FW135M_BP_HANDOFF.md b/docs/campaign/FW135M_BP_HANDOFF.md new file mode 100644 index 0000000..1467ac0 --- /dev/null +++ b/docs/campaign/FW135M_BP_HANDOFF.md @@ -0,0 +1,35 @@ +# FW135M BP Sweep Handoff + +The 135M BP smoke test completed on an RTX A6000: + +- `fw135m_bp_smoke_s1` +- 400/400 steps completed; best validation CE `5.7126` +- W&B: `eqprop-llm-training/ept-fineweb-135M` + +The BP sweep is matched to the active `fw135m_bsign` EP run in all shared +settings: `L12/C768/H12/T256/B24`, FineWeb-Edu 32k, OLMo2, Muon +(`--muon_lr 0.02`), BF16, weight decay `0.1`, cosine to `0.1×`, 440,000 +trainer steps, 1,000 warmup steps, and NCCL data parallelism. `B24` is per +rank, so four A6000s use effective batch `96`. The intentional difference is +BP versus EP's `--bsign_rand --beta 0.003`. + +## Run one candidate + +From `ep_run/`, each candidate should use all four A6000s: + +```bash +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 7e-4 1 +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1e-3 1 +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1.4e-3 1 +``` + +Run the three commands sequentially when only four GPUs are available. Each +candidate gets a distinct W&B run name: +`fw135m_bp_lr7em4_s1`, `fw135m_bp_lr1em3_s1`, or `fw135m_bp_lr1p4em3_s1`. + +The launcher defaults data to `ep_run/data/fineweb_edu`. Set +`EPT_DATA_ROOT=/path/to/data` only when FineWeb data is stored elsewhere. + +Choose the LR by best validation CE, final validation CE, and tail-median CE. +Then run the selected BP setting for seeds 1 and 2 before reporting a BP/EP +comparison. |