diff options
Diffstat (limited to 'docs/campaign/CASCADE_ABLATION_PLAN.md')
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 21 |
1 files changed, 21 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 8b52369..d604fc6 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -562,3 +562,24 @@ thesis in one number). CAVEAT: static probes at a trained checkpoint (eval CE + direction), not training-under-fault; wave-2 = co-training with faults injected from step 0 (expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md (per-row status updated) + UIUC outreach dossier. + +### RESULT 15 (2026-07-12): bf16 MIXED PRECISION (--amp) VALIDATED — lossless at 4k, 1.56x wall-clock. +The /2-class cost lever, same-day pipeline: amp_gate.py static gate -> trainer flag -> 3-seed A/B. +SEMANTICS (why this lives while naive-cast --bf16 is dead): params/states/displacements/E-accum stay +fp32; ONLY block forwards run under autocast(bf16). RESULT 11's naive-cast death = pure STATE +quantization (wall-1: beta-displacement below bf16 resolution) — exactly as diagnosed. +- Gate (stage1b s55000, fp64 cosine): amp cos(EP,BP_fp32) 0.9682 vs fp32-EP 0.9687 (zero loss); + beta=3e-3 -> 0.9878, 1e-2 -> 0.9966 (bigger beta ACTIVELY better — wall-1 SNR physics); + bf16 fwd valCE -0.0002; BP_amp baseline 0.9993. amp_last (fp32 final rebuild) buys nothing -> + amp_all everywhere; the E-subtraction term is not binding at production beta (fbnoise-tolerance + prediction from RESULT 14 held: relative noise on forces is invisible). +- 3-seed 4k A/B (bsign flagset + --amp): 1.7086/1.7569/1.7562 mean 1.7406 vs fp32 3v3 mean 1.7314 + (Delta +0.009 inside the seed-noise band; amp_s1 BEAT the BP+Muon mean 1.7098). In-trainer bp_gate + cos 0.9999 at step 0. Zero guard events. +- SPEED (solo GPU3/A6000, C512): amp 2.785 it/s vs fp32 1.789 it/s = 1.56x wall-clock; grows with + width (tensor-core-bound share) -> treat 1.5x as the floor for 1-3B on H100. +- CAVEAT + confirm step: 4k is short-horizon; stage1b_amp (58.8k full epoch, identical recipe+seed, + GPU1) running as the epoch-scale confirm vs 1.2808. EMAIL_BEN_DRAFT2 send-gate #6 waits on it. +- Cost consequence: COST_MODEL.md v2.1 (sourced July-2026 prices: market H100 $1.87-2.99/GPU.h, + AWS p5e blocks $4.97/GPU.h post-hike) — with amp measured, 3B-Chinchilla ~$40k / 7Bx20B ~$30k + on AWS blocks: BOTH inside the $50k envelope individually. amp is the Stage-2 default. |