diff options
Diffstat (limited to 'docs/campaign/CASCADE_ABLATION_PLAN.md')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 17 |
1 files changed, 17 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 2710582..c044fb6 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -563,6 +563,23 @@ direction), not training-under-fault; wave-2 = co-training with faults injected (expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md (per-row status updated) + UIUC outreach dossier. + +### RESULT 17 (2026-07-13): fw72m LAUNCHED (the 62.7M-crown run) + size ladder complete + resume upgraded. +- **fw72m**: 72.11M (L12 C512 @32k vocab) x 1.44B FineWeb-Edu tokens (Chinchilla), --amp, first NCCL + 2-GPU DDP production run (GPU0+3, B12/rank = global B24 preserving recipe semantics). Startup: + world=2, cos 0.9999, zero skips, **2.92 it/s -> ETA ~22h**. Crown context: KHS VGG10 = 62.7M + (verified from their Table 6: convs 9.2M + dense 25088->2048 = 51.4M + head 2.05M = 62.65M). + BP twin queued for GPU1 (waiting on user's phasescan). Corpus: 9.99B tokens, doc-level shuffled + (9.67M docs, seed 1234, val re-drawn 20M disjoint) per user directive. +- **nsize ladder DONE** (TinyStories 4k-vocab, 3k steps, fp32, seed 1): c256 1.9265 / c512 1.7920 / + c768 1.7913 / c1024 1.7876 — at FIXED 3k steps quality saturates with width (data/steps-bound, + expected); purpose = noise-robustness probe ckpts (runs/nsize_*_s3000.pt x4). Probe script queued. +- **Resume upgraded to exact**: MultiOpt gains state_dict/load_state_dict; trainer saves 'opt' in + every ckpt and restores it on --resume (chunked HPC jobs no longer lose Adam/Muon state). +- **Delta storage recon**: /scratch 1.5T/1.5T FULL, /work/hdd/bfqt over quota -> 300M data transfer + BLOCKED until space found (own old-ept footprint = first cleanup candidate). A100x4 queue ~8.5d, + A40 same-day (48h cap -> needs exact resume, now DONE). + ### RESULT 16 (2026-07-13): STAGE-2 DATA PIPELINE LIVE + FINEWEB SMOKE PASSED. `prepare_fineweb.py`: FineWeb-Edu sample-10BT -> 32k ByteLevel BPE (<|eot|> id 0) -> uint16 bins, tinystories_bpe format, `--data` flag added to both trainers. SMOKE_READY in 11 min (download 5min |