diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 13 |
1 files changed, 13 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 3377dab..2710582 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -563,6 +563,19 @@ direction), not training-under-fault; wave-2 = co-training with faults injected (expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md (per-row status updated) + UIUC outreach dossier. +### RESULT 16 (2026-07-13): STAGE-2 DATA PIPELINE LIVE + FINEWEB SMOKE PASSED. +`prepare_fineweb.py`: FineWeb-Edu sample-10BT -> 32k ByteLevel BPE (<|eot|> id 0) -> uint16 bins, +tinystories_bpe format, `--data` flag added to both trainers. SMOKE_READY in 11 min (download 5min +@80MB/s, tokenizer train 39s on 1.5GB, shard0 5min = 755M tokens; val = first 20M, disjoint). +Full 14 shards -> ~10.5B tokens (running, ~50 min ETA at 754M/5.4min per shard). +**fw_smoke (L12 C512 32k-vocab = 72.11M, --amp, 400 steps, GPU1): CE 10.51 -> 5.84, cos(EP,BP) +0.9999@0 / 0.9951@100 / 0.9995@400, ZERO skips, drift 0.002.** The estimator + amp + beta-governance +(sig grew 3.9->58, beta floored by step 100 — wall-1 machinery engaged correctly on the harder +corpus) all transfer to real web text at 4x vocab unchanged. Speed 0.89 it/s at this shape (bigger +head). Stage-2 recipe question OPEN for user: T=1024 (web-native context) vs T=256 (strict +TinyStories comparability) for the 300M run. NOTE: the "BP twin 2.9746" line in DONE prints is the +stale TinyStories reference (cosmetic); no fineweb BP twin exists yet. + ### RESULT 15 (2026-07-12): bf16 MIXED PRECISION (--amp) VALIDATED — lossless at 4k, 1.56x wall-clock. The /2-class cost lever, same-day pipeline: amp_gate.py static gate -> trainer flag -> 3-seed A/B. SEMANTICS (why this lives while naive-cast --bf16 is dead): params/states/displacements/E-accum stay |