1 files changed, 44 insertions, 39 deletions
diff --git a/docs/campaign/COST_MODEL.md b/docs/campaign/COST_MODEL.md
index 2b9d8b6..fdca732 100644
--- a/docs/campaign/COST_MODEL.md
+++ b/docs/campaign/COST_MODEL.md
@@ -1,47 +1,52 @@
-# Cost model v2 (2026-07-12) — recalibrated on a measured rental datapoint
+# Cost model v2.1 (2026-07-12) — sourced July-2026 prices + measured amp speedup
 
-Supersedes the v1 table in EMAIL_BEN_DRAFT2.md (drafts ≤5) and the numbers quoted in the
-2026-07-11 scale discussion. **v1's error: anchored $/FLOP on AWS-A100-savings TF32
-(~$19.7k/EF) — the wrong reference class for 2026.** Market H100/H200 rentals are 2.5–3×
-cheaper per FLOP.
+Supersedes v2 (same day; fixed a 1000× unit-label error — exaFLOP = 1e18, so the user's
+anchor is $16.7/EF not "$16.7k/EF"; all dollar TOTALS in v2 were computed as ratios and
+were correct) and v1 (anchored on AWS-A100-savings, ~2.5-3× too expensive per FLOP).
 
-## Anchors
+## Sourced prices (July 2026)
 
-| anchor | value | source |
+| item | price | source |
 |---|---|---|
-| **Empirical all-in ceiling** | **$16.7k/EF** | user's real run: 1B × 10B tok BP on rented H200s ≈ $1k all-in; 6ND = 0.06 EF. Implied throughput only 42–58 TF/s eff (4–6% of bf16 peak) — an UN-optimized run, so this is a ceiling, not a target |
-| Market rental rate | $2–3.5/GPU·h | H100/H200 marketplace (Vast/RunPod class), 2026 |
-| Our stack's utilization | ~19% of TF32 peak | measured on A6000 (14 TF eff / 75 TF TF32 peak), eager TF32 |
-| → projected H200 throughput | 75–125 TF/s eff | 15–25% × 494 TF TF32 peak |
-| → **working rate (TF32 path)** | **$6–9k/EF** (mid $7.5k) | $2.5–3/h ÷ 75–125 TF/s |
-| EP/BP FLOP ratio | **3.2×** (measured) | EP ≈ 19.2N FLOPs/tok vs BP 6N; single-sided, K=3 |
-| Failure allowance | 1.3× | failure modes characterized; ckpt every 5k steps |
-| AWS multiplier | 1.5–2.5× on-demand; 1.2–1.5× spot/capacity-block | vs marketplace $/GPU·h |
-| Mixed-precision upside | ÷~2 | proper autocast (bf16 matmul, fp32 states + E-accum) — UNVALIDATED for EP; naive-cast is dead (cos ≤0.67) |
-
-Cross-check: v1's per-EF rate ($19.7k) ≈ the empirical ceiling ($16.7k) — v1 rows were not
-order-of-magnitude wrong, they were "un-optimized-run" priced. The row deltas people remember
-($1k vs $12k) decompose as: ×2 tokens (20B vs 10B) × 3.2 EP × 1.4 buffer ≈ ×9.
-
-## Table (market rates, TF32 path, ×1.3 buffer; AWS on-demand ≈ +50–150%)
-
-| run | EP FLOPs (19.2ND) | cost | note |
+| Market H100 | $1.87–2.99/GPU·h | Vast $1.87, RunPod PCIe $1.99 / SXM $2.69, Lambda $2.99 (intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison, runpod.io/pricing, lambda.ai/pricing) |
+| Market H200 | $2.30–4.39/GPU·h, median ~$4 | same survey; FluidStack low end |
+| **AWS p5e (H200) capacity blocks** | **$39.80/instance·h = $4.97/GPU·h** (Ohio; US-West $6.22) | aws.amazon.com/ec2/capacityblocks/pricing — **+20% hike effective 2026-07-01** (p5en $5.20) |
+| AWS/marketplace ratio | **~2–2.5×** | above (v2's "1.5–2.5×, blocks close the gap" is stale — blocks got hiked) |
+
+H100 and H200 have identical compute (H200 = more/faster HBM); for 1–7B models rent H100
+at $2–2.7 — strictly better $/FLOP than H200.
+
+## Rates (our stack, measured)
+
+- Anchor (empirical ceiling): user's real run, 1B×10B BP on rented H200 ≈ $1k = 60 EF → **$16.7/EF**
+  all-in with NO stack optimization (implied util 4–6% of bf16 peak). Kept as the worst case.
+- Utilization: measured ~19% of TF32 peak on A6000 eager → H100 TF32 (494 TF) at 15–25% =
+  75–125 TF/s eff → **TF32 rate: $6–9/EF market**.
+- **amp (bf16 autocast, fp32 states/d/E): VALIDATED 2026-07-12** — gate cos 0.9682 vs fp32 0.9687;
+  3-seed 4k A/B mean 1.7406 vs fp32 1.7314 (inside seed-noise band, amp_s1 beat the BP mean);
+  **measured 1.56× wall-clock at C512/A6000** (2.785 vs 1.789 it/s solo). Conservatively apply
+  1.5× (tensor-core-bound share grows with model width, so this is a floor at 1–3B):
+  **amp rate: $4–6/EF market; $9–14/EF AWS blocks**. Full-epoch confirm (stage1b_amp) running.
+- EP/BP FLOP ratio 3.2× (measured); EP FLOPs = 19.2·N·D. Failure allowance 1.3× (ckpt every 5k).
+
+## Table (amp rates; buffered ×1.3)
+
+| run | EP FLOPs | market | AWS blocks |
 |---|---|---|---|
-| 300M × 6B (recipe validation) | 0.035 EF | **~$400** | |
-| **1B × 20B Chinchilla EP** | 0.384 EF | **~$4k** | ≤$8k at the empirical-ceiling rate |
-| 1B × 20B BP control | 0.12 EF (6ND) | **~$1.5k** | user's datapoint directly: 10B = $1k |
-| **3B × 60B full-Chinchilla EP** | 3.46 EF | **~$30–40k** | inside $50k alone (market); AWS on-demand $45–60k → needs spot or bf16 |
-| **7B × 20B (GPT-3-style) EP** | 2.69 EF | **~$25–35k** | inside envelope |
-| 7B × 140B full-Chinchilla EP | 18.8 EF | $110–170k | out of scope; dedicated funding |
+| 300M × 6B (recipe validation) | 34.6 EF | **~$300** | ~$500 |
+| **1B × 20B Chinchilla EP** | 384 EF | **~$2.5k** | **~$5k** |
+| 1B × 20B BP control | 120 EF (6ND) | ~$700 | ~$1.5k |
+| **3B × 60B full-Chinchilla EP** | 3,456 EF | **~$18–27k** | **~$35–48k** |
+| **7B × 20B (GPT-3-style) EP** | 2,688 EF | **~$14–21k** | **~$27–38k** |
+| 7B × 140B full-Chinchilla EP | 18,816 EF | ~$100–150k | ~$190–260k |
+
+TF32 fallback (if stage1b_amp fails at epoch scale): multiply by ~1.5.
 
 ## Consequences
 
-1. **$50k envelope reaches ONE of {3B full-Chinchilla, 7–8B reduced-token} after the 1B milestone** —
-   the scale conversation upgrades from speculative to budgetable.
-2. The **calibration day (~$500 on Ben's AWS instances)** now resolves a 2× spread
-   (working rate vs empirical ceiling, AWS vs market) — worth more than before.
-3. **bf16 mixed-precision validation is the highest-leverage engineering item**: ÷2 across the
-   table brings BOTH large options comfortably inside AWS on-demand pricing. Distinct from the
-   dead naive-cast: fp32 states + fp32 energy accumulation, bf16 matmuls only, gate = cos(EP,BPTT).
-4. Throughput assumptions are conservative (eager, no compile/flash in the nudged phase);
-   the speed-profile levers (--holofast, --sdpa ≈ 1.5×) are not priced in.
+1. **With amp landed, BOTH large options fit the $50k AWS envelope individually**
+   (3B-Chinchilla ~$40k, 7B×20B ~$30k, each incl. the 1B milestone ~$6.5k); at market rates
+   either fits with wide margin — quote AWS to Ben (his credits), note market is ~2× cheaper.
+2. The calibration day (~$500 on his instances) resolves the util spread (15–25%) → ±15%.
+3. bf16-amp moved from "upside" to "measured" — the ÷1.5 is in the table, not a promise.
+   Residual upside: compile/flash-attn in the relax loop (not priced), util growth at width.