diff options
Diffstat (limited to 'EMAIL_BEN_DRAFT2.md')
| -rw-r--r-- | EMAIL_BEN_DRAFT2.md | 24 |
1 files changed, 13 insertions, 11 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md index 0b6fc4a..a8f20ce 100644 --- a/EMAIL_BEN_DRAFT2.md +++ b/EMAIL_BEN_DRAFT2.md @@ -16,17 +16,17 @@ The results promised yesterday, concretely: - **The gap ledger.** At matched hyperparameters, 4k steps: EP statistically indistinguishable from its tuned backprop control (three seeds each). Full epoch: 0.050 nats behind, with the mechanism identified — the late-training decline of estimator signal-to-noise as the true gradient shrinks against a fixed precision floor (the quantified version of your β remark) — and a β-schedule that removes most of it. Muon's improvement over AdamW transfers to EP gradients at full magnitude. - **Solver cost.** Our nudged phase runs the full EP step at 3.2× the FLOPs of a backprop step (measured; single-sided, K=3 full-depth sweeps per nudge). For calibration against your Appendix B: that ratio is what turns an 18-day run into a ~4-day one at equal hardware. -On costs — calibrated against a measured datapoint rather than list pricing (a 1B × 10B-token backprop run we executed on rented H200s cost ≈$1k all-in), combined with the measured 3.2× EP/BP FLOP ratio: +On costs, at current (July 2026) rental prices — market H100 at $2–2.7/GPU-hour; AWS H200 capacity blocks at ≈$5/GPU-hour after this month's increase — with our measured utilization, the measured 3.2× EP/BP FLOP ratio, mixed-precision EP validated against the fp32 reference this week (gradient cosine unchanged; 1.56× measured wall-clock), and a 1.3× failure allowance: -| run | cost at market H100/H200 rates, incl. 1.3× failure allowance | -|---|---| -| 300M × 6B tokens — recipe validation on a standard corpus | ~$400 | -| **1B × 20B tokens, Chinchilla-optimal** | **~$4k** (≤$8k even at the uncalibrated worst case) | -| 1B backprop control, same tokens | ~$1.5k | +| run | AWS capacity blocks | market rental | +|---|---|---| +| 300M × 6B tokens — recipe validation on a standard corpus | ~$500 | ~$300 | +| **1B × 20B tokens, Chinchilla-optimal** | **~$5k** | ~$2.5k | +| 1B backprop control, same tokens | ~$1.5k | ~$700 | -Three qualifications. The numbers assume the TF32 path we have verified bit-exact against fp32; a standard mixed-precision variant (bf16 matrix products, fp32 states and energy accumulation) is not yet validated for EP and would roughly halve them if it passes our gradient-fidelity gate. AWS list pricing runs 1.5–2.5× market rental per GPU-hour, with spot or capacity-block reservations closing most of that gap. And checkpoints every 5k steps bound any failure to the segment since the last one. A one-day calibration on your instances (~$500) would replace these estimates with measurements; we would be glad to run that first. +Checkpoints every 5k steps bound any failure to the segment since the last one. A one-day calibration on your instances (~$500) would replace these estimates with measurements; we would be glad to run that first. -This pricing changes the scale question. Within $50k, the 1B milestone and its control leave room for one of: **3B at full Chinchilla scaling (~$30–40k at market rates), or 7–8B at a GPT-3-style token budget (~$25–35k)** — AWS on-demand pricing adds roughly half on top, and the mixed-precision variant, if it validates, subtracts it back. Our intention is unchanged — complete the 1B result first and let it argue for the larger run — but the envelope does reach one of the two. +At these prices the scale question sharpens. Within $50k on AWS, the 1B milestone and its control leave room for either **3B at full Chinchilla scaling (~$40k)** or **7–8B at a GPT-3-style token budget (~$30k)**; at market rates either fits with wide margin. Our intention is unchanged — complete the 1B result first and let it argue for the larger run — but the envelope now reaches one of the two on your infrastructure alone. Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** Every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it. @@ -41,6 +41,8 @@ Yuren 2. 附 a1_demo.ipynb（token 已内置） 3. ~~gap 数字核对~~ RESOLVED: f3e3cont 终值 1.2883 > 1.2808 → "0.050 nats" 保持不变 4. 3.2× 的对照句("18-day → ~4-day")口吻是否合适你再定——它来自他们自己的 Appendix B,准确但直接 -5. 成本表 2026-07-12 已按用户实测锚点重校准（1B×10B BP H200 租赁 ≈$1k → $16.7k/EF 全包上限; - 市场价 H100/H200 TF32 路径 $6-9k/EF)。旧表锚 AWS-A100-savings,每 FLOP 贵 2.5-3×,已废。 - 完整推导见 docs/campaign/COST_MODEL.md +5. 成本表 2026-07-12 v2.1: 按当日查证实价（市场 H100 $1.87-2.99/h; AWS p5e blocks $4.97/GPU·h, + 7月1日+20%后）+ 实测 amp 1.56×。旧 v1 锚 AWS-A100-savings 已废; 用户 H200 数据点按指示不进邮件, + 仅作 $16.7/EF 最差上限留档。完整推导+价格出处: docs/campaign/COST_MODEL.md +6. **发送门: stage1b_amp (58.8k 全 epoch, GPU1) 完成且 ≈1.28 档才发**——邮件说了 "validated + against the fp32 reference this week"。若 amp 在 epoch 尺度掉链子 → 表格回退 TF32 (×1.5) 并删该句 |