From e2e636f36bd75378c6bb630b76d3b497e03ae401 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Sat, 11 Jul 2026 09:09:16 -0500 Subject: Ben reply draft 2: news block (largest fully BP-free LM, parity, Muon transfer) + measured cost table + calibration pilot + envelope questions --- EMAIL_BEN_DRAFT2.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 EMAIL_BEN_DRAFT2.md (limited to 'EMAIL_BEN_DRAFT2.md') diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md new file mode 100644 index 0000000..a78c073 --- /dev/null +++ b/EMAIL_BEN_DRAFT2.md @@ -0,0 +1,31 @@ +# Draft 2 — reply to Ben (costs + news), 2026-07-11 + +Subject: Re: costs — numbers you can put in front of accounting, and some news since the draft + +Hi Ben, + +Great to hear — and yes, AsymEP it is ;) + +News first, because it changes what the credits would buy. Since the draft, the scaling line moved to a cascade form: L *distinct* standard transformer blocks, inference is a plain forward pass, EP only in training. + +- **A standard 12-block transformer (42.7M) trained through a full epoch with zero backpropagation anywhere in the loop — and it tells coherent stories.** A one-click notebook that samples from it is ready whenever you'd like to play with it. +- As far as we can verify, **it is already the largest fully backprop-free language model trained to date**: the 1.07B optical-DFA result trains 400M of its parameters and keeps BP at the readout; the ES line's from-scratch model is a ~2M integer GRU. +- **At matched tuning, EP is indistinguishable from its tuned backprop twin** (4k steps); at full-epoch the gap is 0.050 nats — and every centinat of it has a named mechanism and a measured dial (estimator SNR vs. nudge amplitude: the same physics an analog chip will meet, rehearsed in fp32). +- **Muon's advantage over AdamW transfers to EP gradients at full magnitude** (−0.13 CE at this scale) — backprop-free training runs on a modern optimizer stack. + +On costs, anchored on measured throughput (TF32 validated bit-exact for this trainer; EP = 3.2× BP FLOPs, measured, not estimated): + +| run | cost, 1.4× failure buffer included | +|---|---| +| 300M × 6B tok — recipe validation on a real corpus | ~$1k | +| **1B × 20B tok (Chinchilla-optimal)** | **~$12k — inside the cap** | +| 3B × 25B tok | ~$25–30k (two capped, checkpointed segments) | + +The buffer is small because the failure model is not a guess: the three instabilities we met this quarter are each mechanistically diagnosed and closed (a β-schedule, a block-architecture change, an amplitude floor), and runs checkpoint every 5k steps — **a failed run costs a segment, not the run.** One day on your instances (~$500) converts these numbers from ±30% to ±10–15%; happy to run that calibration first. + +Two questions so we can plan honestly: is the ~$20k a per-run cap inside a larger envelope, or the envelope itself? And does the backprop-twin control (≈1/3 the EP cost, needed for the head-to-head claim) draw from the same pool? + +No rush on the draft — it has since been restructured around a control-map spine, so I'd send you the updated PDF before you invest reading time. + +Best, +Yuren -- cgit v1.2.3