summaryrefslogtreecommitdiff
path: root/scripts/prepare_data.py
AgeCommit message (Collapse)Author
29 hoursprepare_data: hard-exit after DONE — HF streaming prefetch threads hang ↵HEADmainYurenHao0426
interpreter shutdown after break Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe
30 hoursprepare_data: write .partial then rename, so interrupted prep can't leave a ↵YurenHao0426
half train.bin that gets mistaken for complete Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe
35 hoursdata prep (FineWeb-Edu->GPT2 BPE), rho diagnostics, tests, measured-cost notesyurenh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe