diff options
| -rw-r--r-- | EMAIL_BEN_DRAFT2.md | 24 | ||||
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 21 | ||||
| -rw-r--r-- | docs/campaign/COST_MODEL.md | 83 |
3 files changed, 78 insertions, 50 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md index 0b6fc4a..a8f20ce 100644 --- a/EMAIL_BEN_DRAFT2.md +++ b/EMAIL_BEN_DRAFT2.md @@ -16,17 +16,17 @@ The results promised yesterday, concretely: - **The gap ledger.** At matched hyperparameters, 4k steps: EP statistically indistinguishable from its tuned backprop control (three seeds each). Full epoch: 0.050 nats behind, with the mechanism identified — the late-training decline of estimator signal-to-noise as the true gradient shrinks against a fixed precision floor (the quantified version of your β remark) — and a β-schedule that removes most of it. Muon's improvement over AdamW transfers to EP gradients at full magnitude. - **Solver cost.** Our nudged phase runs the full EP step at 3.2× the FLOPs of a backprop step (measured; single-sided, K=3 full-depth sweeps per nudge). For calibration against your Appendix B: that ratio is what turns an 18-day run into a ~4-day one at equal hardware. -On costs — calibrated against a measured datapoint rather than list pricing (a 1B × 10B-token backprop run we executed on rented H200s cost ≈$1k all-in), combined with the measured 3.2× EP/BP FLOP ratio: +On costs, at current (July 2026) rental prices — market H100 at $2–2.7/GPU-hour; AWS H200 capacity blocks at ≈$5/GPU-hour after this month's increase — with our measured utilization, the measured 3.2× EP/BP FLOP ratio, mixed-precision EP validated against the fp32 reference this week (gradient cosine unchanged; 1.56× measured wall-clock), and a 1.3× failure allowance: -| run | cost at market H100/H200 rates, incl. 1.3× failure allowance | -|---|---| -| 300M × 6B tokens — recipe validation on a standard corpus | ~$400 | -| **1B × 20B tokens, Chinchilla-optimal** | **~$4k** (≤$8k even at the uncalibrated worst case) | -| 1B backprop control, same tokens | ~$1.5k | +| run | AWS capacity blocks | market rental | +|---|---|---| +| 300M × 6B tokens — recipe validation on a standard corpus | ~$500 | ~$300 | +| **1B × 20B tokens, Chinchilla-optimal** | **~$5k** | ~$2.5k | +| 1B backprop control, same tokens | ~$1.5k | ~$700 | -Three qualifications. The numbers assume the TF32 path we have verified bit-exact against fp32; a standard mixed-precision variant (bf16 matrix products, fp32 states and energy accumulation) is not yet validated for EP and would roughly halve them if it passes our gradient-fidelity gate. AWS list pricing runs 1.5–2.5× market rental per GPU-hour, with spot or capacity-block reservations closing most of that gap. And checkpoints every 5k steps bound any failure to the segment since the last one. A one-day calibration on your instances (~$500) would replace these estimates with measurements; we would be glad to run that first. +Checkpoints every 5k steps bound any failure to the segment since the last one. A one-day calibration on your instances (~$500) would replace these estimates with measurements; we would be glad to run that first. -This pricing changes the scale question. Within $50k, the 1B milestone and its control leave room for one of: **3B at full Chinchilla scaling (~$30–40k at market rates), or 7–8B at a GPT-3-style token budget (~$25–35k)** — AWS on-demand pricing adds roughly half on top, and the mixed-precision variant, if it validates, subtracts it back. Our intention is unchanged — complete the 1B result first and let it argue for the larger run — but the envelope does reach one of the two. +At these prices the scale question sharpens. Within $50k on AWS, the 1B milestone and its control leave room for either **3B at full Chinchilla scaling (~$40k)** or **7–8B at a GPT-3-style token budget (~$30k)**; at market rates either fits with wide margin. Our intention is unchanged — complete the 1B result first and let it argue for the larger run — but the envelope now reaches one of the two on your infrastructure alone. Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** Every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it. @@ -41,6 +41,8 @@ Yuren 2. 附 a1_demo.ipynb（token 已内置） 3. ~~gap 数字核对~~ RESOLVED: f3e3cont 终值 1.2883 > 1.2808 → "0.050 nats" 保持不变 4. 3.2× 的对照句("18-day → ~4-day")口吻是否合适你再定——它来自他们自己的 Appendix B,准确但直接 -5. 成本表 2026-07-12 已按用户实测锚点重校准（1B×10B BP H200 租赁 ≈$1k → $16.7k/EF 全包上限; - 市场价 H100/H200 TF32 路径 $6-9k/EF)。旧表锚 AWS-A100-savings,每 FLOP 贵 2.5-3×,已废。 - 完整推导见 docs/campaign/COST_MODEL.md +5. 成本表 2026-07-12 v2.1: 按当日查证实价（市场 H100 $1.87-2.99/h; AWS p5e blocks $4.97/GPU·h, + 7月1日+20%后）+ 实测 amp 1.56×。旧 v1 锚 AWS-A100-savings 已废; 用户 H200 数据点按指示不进邮件, + 仅作 $16.7/EF 最差上限留档。完整推导+价格出处: docs/campaign/COST_MODEL.md +6. **发送门: stage1b_amp (58.8k 全 epoch, GPU1) 完成且 ≈1.28 档才发**——邮件说了 "validated + against the fp32 reference this week"。若 amp 在 epoch 尺度掉链子 → 表格回退 TF32 (×1.5) 并删该句 diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 8b52369..d604fc6 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -562,3 +562,24 @@ thesis in one number). CAVEAT: static probes at a trained checkpoint (eval CE + direction), not training-under-fault; wave-2 = co-training with faults injected from step 0 (expectation from the literature and from (c): tolerances IMPROVE). Feeds COMPONENT_HW_MAP.md (per-row status updated) + UIUC outreach dossier. + +### RESULT 15 (2026-07-12): bf16 MIXED PRECISION (--amp) VALIDATED — lossless at 4k, 1.56x wall-clock. +The /2-class cost lever, same-day pipeline: amp_gate.py static gate -> trainer flag -> 3-seed A/B. +SEMANTICS (why this lives while naive-cast --bf16 is dead): params/states/displacements/E-accum stay +fp32; ONLY block forwards run under autocast(bf16). RESULT 11's naive-cast death = pure STATE +quantization (wall-1: beta-displacement below bf16 resolution) — exactly as diagnosed. +- Gate (stage1b s55000, fp64 cosine): amp cos(EP,BP_fp32) 0.9682 vs fp32-EP 0.9687 (zero loss); + beta=3e-3 -> 0.9878, 1e-2 -> 0.9966 (bigger beta ACTIVELY better — wall-1 SNR physics); + bf16 fwd valCE -0.0002; BP_amp baseline 0.9993. amp_last (fp32 final rebuild) buys nothing -> + amp_all everywhere; the E-subtraction term is not binding at production beta (fbnoise-tolerance + prediction from RESULT 14 held: relative noise on forces is invisible). +- 3-seed 4k A/B (bsign flagset + --amp): 1.7086/1.7569/1.7562 mean 1.7406 vs fp32 3v3 mean 1.7314 + (Delta +0.009 inside the seed-noise band; amp_s1 BEAT the BP+Muon mean 1.7098). In-trainer bp_gate + cos 0.9999 at step 0. Zero guard events. +- SPEED (solo GPU3/A6000, C512): amp 2.785 it/s vs fp32 1.789 it/s = 1.56x wall-clock; grows with + width (tensor-core-bound share) -> treat 1.5x as the floor for 1-3B on H100. +- CAVEAT + confirm step: 4k is short-horizon; stage1b_amp (58.8k full epoch, identical recipe+seed, + GPU1) running as the epoch-scale confirm vs 1.2808. EMAIL_BEN_DRAFT2 send-gate #6 waits on it. +- Cost consequence: COST_MODEL.md v2.1 (sourced July-2026 prices: market H100 $1.87-2.99/GPU.h, + AWS p5e blocks $4.97/GPU.h post-hike) — with amp measured, 3B-Chinchilla ~$40k / 7Bx20B ~$30k + on AWS blocks: BOTH inside the $50k envelope individually. amp is the Stage-2 default. diff --git a/docs/campaign/COST_MODEL.md b/docs/campaign/COST_MODEL.md index 2b9d8b6..fdca732 100644 --- a/docs/campaign/COST_MODEL.md +++ b/docs/campaign/COST_MODEL.md @@ -1,47 +1,52 @@ -# Cost model v2 (2026-07-12) — recalibrated on a measured rental datapoint +# Cost model v2.1 (2026-07-12) — sourced July-2026 prices + measured amp speedup -Supersedes the v1 table in EMAIL_BEN_DRAFT2.md (drafts ≤5) and the numbers quoted in the -2026-07-11 scale discussion. **v1's error: anchored $/FLOP on AWS-A100-savings TF32 -(~$19.7k/EF) — the wrong reference class for 2026.** Market H100/H200 rentals are 2.5–3× -cheaper per FLOP. +Supersedes v2 (same day; fixed a 1000× unit-label error — exaFLOP = 1e18, so the user's +anchor is $16.7/EF not "$16.7k/EF"; all dollar TOTALS in v2 were computed as ratios and +were correct) and v1 (anchored on AWS-A100-savings, ~2.5-3× too expensive per FLOP). -## Anchors +## Sourced prices (July 2026) -| anchor | value | source | +| item | price | source | |---|---|---| -| **Empirical all-in ceiling** | **$16.7k/EF** | user's real run: 1B × 10B tok BP on rented H200s ≈ $1k all-in; 6ND = 0.06 EF. Implied throughput only 42–58 TF/s eff (4–6% of bf16 peak) — an UN-optimized run, so this is a ceiling, not a target | -| Market rental rate | $2–3.5/GPU·h | H100/H200 marketplace (Vast/RunPod class), 2026 | -| Our stack's utilization | ~19% of TF32 peak | measured on A6000 (14 TF eff / 75 TF TF32 peak), eager TF32 | -| → projected H200 throughput | 75–125 TF/s eff | 15–25% × 494 TF TF32 peak | -| → **working rate (TF32 path)** | **$6–9k/EF** (mid $7.5k) | $2.5–3/h ÷ 75–125 TF/s | -| EP/BP FLOP ratio | **3.2×** (measured) | EP ≈ 19.2N FLOPs/tok vs BP 6N; single-sided, K=3 | -| Failure allowance | 1.3× | failure modes characterized; ckpt every 5k steps | -| AWS multiplier | 1.5–2.5× on-demand; 1.2–1.5× spot/capacity-block | vs marketplace $/GPU·h | -| Mixed-precision upside | ÷~2 | proper autocast (bf16 matmul, fp32 states + E-accum) — UNVALIDATED for EP; naive-cast is dead (cos ≤0.67) | - -Cross-check: v1's per-EF rate ($19.7k) ≈ the empirical ceiling ($16.7k) — v1 rows were not -order-of-magnitude wrong, they were "un-optimized-run" priced. The row deltas people remember -($1k vs $12k) decompose as: ×2 tokens (20B vs 10B) × 3.2 EP × 1.4 buffer ≈ ×9. - -## Table (market rates, TF32 path, ×1.3 buffer; AWS on-demand ≈ +50–150%) - -| run | EP FLOPs (19.2ND) | cost | note | +| Market H100 | $1.87–2.99/GPU·h | Vast $1.87, RunPod PCIe $1.99 / SXM $2.69, Lambda $2.99 (intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison, runpod.io/pricing, lambda.ai/pricing) | +| Market H200 | $2.30–4.39/GPU·h, median ~$4 | same survey; FluidStack low end | +| **AWS p5e (H200) capacity blocks** | **$39.80/instance·h = $4.97/GPU·h** (Ohio; US-West $6.22) | aws.amazon.com/ec2/capacityblocks/pricing — **+20% hike effective 2026-07-01** (p5en $5.20) | +| AWS/marketplace ratio | **~2–2.5×** | above (v2's "1.5–2.5×, blocks close the gap" is stale — blocks got hiked) | + +H100 and H200 have identical compute (H200 = more/faster HBM); for 1–7B models rent H100 +at $2–2.7 — strictly better $/FLOP than H200. + +## Rates (our stack, measured) + +- Anchor (empirical ceiling): user's real run, 1B×10B BP on rented H200 ≈ $1k = 60 EF → **$16.7/EF** + all-in with NO stack optimization (implied util 4–6% of bf16 peak). Kept as the worst case. +- Utilization: measured ~19% of TF32 peak on A6000 eager → H100 TF32 (494 TF) at 15–25% = + 75–125 TF/s eff → **TF32 rate: $6–9/EF market**. +- **amp (bf16 autocast, fp32 states/d/E): VALIDATED 2026-07-12** — gate cos 0.9682 vs fp32 0.9687; + 3-seed 4k A/B mean 1.7406 vs fp32 1.7314 (inside seed-noise band, amp_s1 beat the BP mean); + **measured 1.56× wall-clock at C512/A6000** (2.785 vs 1.789 it/s solo). Conservatively apply + 1.5× (tensor-core-bound share grows with model width, so this is a floor at 1–3B): + **amp rate: $4–6/EF market; $9–14/EF AWS blocks**. Full-epoch confirm (stage1b_amp) running. +- EP/BP FLOP ratio 3.2× (measured); EP FLOPs = 19.2·N·D. Failure allowance 1.3× (ckpt every 5k). + +## Table (amp rates; buffered ×1.3) + +| run | EP FLOPs | market | AWS blocks | |---|---|---|---| -| 300M × 6B (recipe validation) | 0.035 EF | **~$400** | | -| **1B × 20B Chinchilla EP** | 0.384 EF | **~$4k** | ≤$8k at the empirical-ceiling rate | -| 1B × 20B BP control | 0.12 EF (6ND) | **~$1.5k** | user's datapoint directly: 10B = $1k | -| **3B × 60B full-Chinchilla EP** | 3.46 EF | **~$30–40k** | inside $50k alone (market); AWS on-demand $45–60k → needs spot or bf16 | -| **7B × 20B (GPT-3-style) EP** | 2.69 EF | **~$25–35k** | inside envelope | -| 7B × 140B full-Chinchilla EP | 18.8 EF | $110–170k | out of scope; dedicated funding | +| 300M × 6B (recipe validation) | 34.6 EF | **~$300** | ~$500 | +| **1B × 20B Chinchilla EP** | 384 EF | **~$2.5k** | **~$5k** | +| 1B × 20B BP control | 120 EF (6ND) | ~$700 | ~$1.5k | +| **3B × 60B full-Chinchilla EP** | 3,456 EF | **~$18–27k** | **~$35–48k** | +| **7B × 20B (GPT-3-style) EP** | 2,688 EF | **~$14–21k** | **~$27–38k** | +| 7B × 140B full-Chinchilla EP | 18,816 EF | ~$100–150k | ~$190–260k | + +TF32 fallback (if stage1b_amp fails at epoch scale): multiply by ~1.5. ## Consequences -1. **$50k envelope reaches ONE of {3B full-Chinchilla, 7–8B reduced-token} after the 1B milestone** — - the scale conversation upgrades from speculative to budgetable. -2. The **calibration day (~$500 on Ben's AWS instances)** now resolves a 2× spread - (working rate vs empirical ceiling, AWS vs market) — worth more than before. -3. **bf16 mixed-precision validation is the highest-leverage engineering item**: ÷2 across the - table brings BOTH large options comfortably inside AWS on-demand pricing. Distinct from the - dead naive-cast: fp32 states + fp32 energy accumulation, bf16 matmuls only, gate = cos(EP,BPTT). -4. Throughput assumptions are conservative (eager, no compile/flash in the nudged phase); - the speed-profile levers (--holofast, --sdpa ≈ 1.5×) are not priced in. +1. **With amp landed, BOTH large options fit the $50k AWS envelope individually** + (3B-Chinchilla ~$40k, 7B×20B ~$30k, each incl. the 1B milestone ~$6.5k); at market rates + either fits with wide margin — quote AWS to Ben (his credits), note market is ~2× cheaper. +2. The calibration day (~$500 on his instances) resolves the util spread (15–25%) → ±15%. +3. bf16-amp moved from "upside" to "measured" — the ÷1.5 is in the table, not a promise. + Residual upside: compile/flash-attn in the relax loop (not priced), util growth at width. |