diff options
| -rw-r--r-- | EMAIL_BEN_DRAFT2.md | 31 |
1 files changed, 31 insertions, 0 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md new file mode 100644 index 0000000..a78c073 --- /dev/null +++ b/EMAIL_BEN_DRAFT2.md @@ -0,0 +1,31 @@ +# Draft 2 — reply to Ben (costs + news), 2026-07-11 + +Subject: Re: costs — numbers you can put in front of accounting, and some news since the draft + +Hi Ben, + +Great to hear — and yes, AsymEP it is ;) + +News first, because it changes what the credits would buy. Since the draft, the scaling line moved to a cascade form: L *distinct* standard transformer blocks, inference is a plain forward pass, EP only in training. + +- **A standard 12-block transformer (42.7M) trained through a full epoch with zero backpropagation anywhere in the loop — and it tells coherent stories.** A one-click notebook that samples from it is ready whenever you'd like to play with it. +- As far as we can verify, **it is already the largest fully backprop-free language model trained to date**: the 1.07B optical-DFA result trains 400M of its parameters and keeps BP at the readout; the ES line's from-scratch model is a ~2M integer GRU. +- **At matched tuning, EP is indistinguishable from its tuned backprop twin** (4k steps); at full-epoch the gap is 0.050 nats — and every centinat of it has a named mechanism and a measured dial (estimator SNR vs. nudge amplitude: the same physics an analog chip will meet, rehearsed in fp32). +- **Muon's advantage over AdamW transfers to EP gradients at full magnitude** (−0.13 CE at this scale) — backprop-free training runs on a modern optimizer stack. + +On costs, anchored on measured throughput (TF32 validated bit-exact for this trainer; EP = 3.2× BP FLOPs, measured, not estimated): + +| run | cost, 1.4× failure buffer included | +|---|---| +| 300M × 6B tok — recipe validation on a real corpus | ~$1k | +| **1B × 20B tok (Chinchilla-optimal)** | **~$12k — inside the cap** | +| 3B × 25B tok | ~$25–30k (two capped, checkpointed segments) | + +The buffer is small because the failure model is not a guess: the three instabilities we met this quarter are each mechanistically diagnosed and closed (a β-schedule, a block-architecture change, an amplitude floor), and runs checkpoint every 5k steps — **a failed run costs a segment, not the run.** One day on your instances (~$500) converts these numbers from ±30% to ±10–15%; happy to run that calibration first. + +Two questions so we can plan honestly: is the ~$20k a per-run cap inside a larger envelope, or the envelope itself? And does the backprop-twin control (≈1/3 the EP cost, needed for the head-to-head claim) draw from the same pool? + +No rush on the draft — it has since been restructured around a control-map spine, so I'd send you the updated PDF before you invest reading time. + +Best, +Yuren |