diff options
| author | yurenh <blackhao0426@gmail.com> | 2026-08-31 18:16:31 -0500 |
|---|---|---|
| committer | yurenh <blackhao0426@gmail.com> | 2026-08-31 18:16:31 -0500 |
| commit | 17a81b9c86cfedd70812a0e83f33798b64c1678e (patch) | |
| tree | 49df72e00c1d8465b249a777dd4e7db13458e099 /README.md | |
| parent | 7db653a60d5125774d60da8d38ee3d49a787be91 (diff) | |
data prep (FineWeb-Edu->GPT2 BPE), rho diagnostics, tests, measured-cost notes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 6 |
1 files changed, 6 insertions, 0 deletions
@@ -21,3 +21,9 @@ python scripts/prepare_data.py --dataset fineweb-edu --tokens 3e9 --out data/fin torchrun --nproc_per_node=8 scripts/train.py --model configs/model/m124.yaml --train configs/train/zbp_n16.yaml ``` Global batch is fixed in the train config; per-rank micro-batch and accumulation adapt to world size. + +## Measured cost (48 GB Ampere-class, d=512 L=8 seq 1024, shared GPU) +BP 312 ms/step; ZBP n=16 **8.7x**, n=64 **29x** (FLOPs-bound: probe batching is already saturated at +probe_chunk 8; the score core uses its own chunk <= 4 since its memory goes as chunk*B*H*T^2). +Ladder arms: **bp / zbp_n16 / zbp_n64** per size (n=4 optional). Headroom if needed: torch.compile on the +query path; forward differences (n+1 instead of 2n queries) as a cheaper biased arm. |