diff options
Diffstat (limited to 'docs/campaign')
| -rw-r--r-- | docs/campaign/FW135M_BP_BASELINE.md | 5 | ||||
| -rw-r--r-- | docs/campaign/FW135M_BP_HANDOFF.md | 35 |
2 files changed, 38 insertions, 2 deletions
diff --git a/docs/campaign/FW135M_BP_BASELINE.md b/docs/campaign/FW135M_BP_BASELINE.md index 3f81ba8..2aade2c 100644 --- a/docs/campaign/FW135M_BP_BASELINE.md +++ b/docs/campaign/FW135M_BP_BASELINE.md @@ -12,8 +12,8 @@ Status: commands configured on branch `xiang`; no training has started. - Data: existing FineWeb-Edu 32k bins - Batch: `B24` - Target: `20N = 2,706,078,720` tokens -- Complete-batch exposure: `2,706,081,792` tokens -- Trainer argument: `--steps 440442` (440,443 inclusive updates) +- EP-matched complete-batch exposure: `2,703,366,144` tokens +- Trainer argument: `--steps 440000` (440,001 inclusive updates, matching `fw135m_bsign`) ## What changes from 72M @@ -29,6 +29,7 @@ Status: commands configured on branch `xiang`; no training has started. - FineWeb-Edu data and 32k tokenizer - OLMo2-style model implementation - Muon hybrid optimizer and cosine schedule +- 1,000-step warmup, matching `fw135m_bsign` - Weight decay 0.1 and BF16 autocast - Seed list and validation cadence - Existing `casc_bp_train.py` code path diff --git a/docs/campaign/FW135M_BP_HANDOFF.md b/docs/campaign/FW135M_BP_HANDOFF.md new file mode 100644 index 0000000..1467ac0 --- /dev/null +++ b/docs/campaign/FW135M_BP_HANDOFF.md @@ -0,0 +1,35 @@ +# FW135M BP Sweep Handoff + +The 135M BP smoke test completed on an RTX A6000: + +- `fw135m_bp_smoke_s1` +- 400/400 steps completed; best validation CE `5.7126` +- W&B: `eqprop-llm-training/ept-fineweb-135M` + +The BP sweep is matched to the active `fw135m_bsign` EP run in all shared +settings: `L12/C768/H12/T256/B24`, FineWeb-Edu 32k, OLMo2, Muon +(`--muon_lr 0.02`), BF16, weight decay `0.1`, cosine to `0.1×`, 440,000 +trainer steps, 1,000 warmup steps, and NCCL data parallelism. `B24` is per +rank, so four A6000s use effective batch `96`. The intentional difference is +BP versus EP's `--bsign_rand --beta 0.003`. + +## Run one candidate + +From `ep_run/`, each candidate should use all four A6000s: + +```bash +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 7e-4 1 +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1e-3 1 +CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 bash runs/fw135m_bp_sweep.sh 1.4e-3 1 +``` + +Run the three commands sequentially when only four GPUs are available. Each +candidate gets a distinct W&B run name: +`fw135m_bp_lr7em4_s1`, `fw135m_bp_lr1em3_s1`, or `fw135m_bp_lr1p4em3_s1`. + +The launcher defaults data to `ep_run/data/fineweb_edu`. Set +`EPT_DATA_ROOT=/path/to/data` only when FineWeb data is stored elsewhere. + +Choose the LR by best validation CE, final validation CE, and tail-median CE. +Then run the selected BP setting for seeds 1 and 2 before reporting a BP/EP +comparison. |