diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-13 06:16:21 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-13 06:16:21 -0500 |
| commit | 4404b945a7cccc52b8a69c0fbbd96776d4a967b0 (patch) | |
| tree | 57710baa340d3c6b0ddbd94c901c53b88e48a95b | |
| parent | 1d638edd83dc888cb392e902eb86ad57b22949a1 (diff) | |
RESULT 16: FineWeb-Edu pipeline live + smoke passed (CE 10.5->5.8, cos 0.9995, zero skips)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 13 |
1 files changed, 13 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 3377dab..2710582 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -563,6 +563,19 @@ direction), not training-under-fault; wave-2 = co-training with faults injected (expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md (per-row status updated) + UIUC outreach dossier. +### RESULT 16 (2026-07-13): STAGE-2 DATA PIPELINE LIVE + FINEWEB SMOKE PASSED. +`prepare_fineweb.py`: FineWeb-Edu sample-10BT -> 32k ByteLevel BPE (<|eot|> id 0) -> uint16 bins, +tinystories_bpe format, `--data` flag added to both trainers. SMOKE_READY in 11 min (download 5min +@80MB/s, tokenizer train 39s on 1.5GB, shard0 5min = 755M tokens; val = first 20M, disjoint). +Full 14 shards -> ~10.5B tokens (running, ~50 min ETA at 754M/5.4min per shard). +**fw_smoke (L12 C512 32k-vocab = 72.11M, --amp, 400 steps, GPU1): CE 10.51 -> 5.84, cos(EP,BP) +0.9999@0 / 0.9951@100 / 0.9995@400, ZERO skips, drift 0.002.** The estimator + amp + beta-governance +(sig grew 3.9->58, beta floored by step 100 — wall-1 machinery engaged correctly on the harder +corpus) all transfer to real web text at 4x vocab unchanged. Speed 0.89 it/s at this shape (bigger +head). Stage-2 recipe question OPEN for user: T=1024 (web-native context) vs T=256 (strict +TinyStories comparability) for the 300M run. NOTE: the "BP twin 2.9746" line in DONE prints is the +stale TinyStories reference (cosmetic); no fineweb BP twin exists yet. + ### RESULT 15 (2026-07-12): bf16 MIXED PRECISION (--amp) VALIDATED — lossless at 4k, 1.56x wall-clock. The /2-class cost lever, same-day pipeline: amp_gate.py static gate -> trainer flag -> 3-seed A/B. SEMANTICS (why this lives while naive-cast --bf16 is dead): params/states/displacements/E-accum stay |