1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
# Draft 3 — reply to Ben (costs + news), 2026-07-11
Subject: Re: costs — numbers for accounting, and some news since the draft
Hi Ben,
Great to hear — and yes, AsymEP it is ;)
News first, because it changes what the credits buy. Since the draft:
- **A standard 12-layer transformer trained through a full epoch with zero backpropagation anywhere in the training loop — and it tells coherent stories.** Inference is a plain forward pass; a one-click notebook that samples from it is ready whenever you'd like to play.
- As far as we can verify, **it is already the largest neural network fully trained without backpropagation** — the 1.07B optical-DFA result trains 400M of its parameters and keeps BP at the readout; the ES line's billion-scale results are fine-tuning of BP-pretrained models, its from-scratch model a ~2M integer GRU.
- **Every previous no-backprop attempt, at any scale, concedes a quality gap to backprop. We measure none.** At matched tuning EP is statistically indistinguishable from its tuned backprop twin; at full-epoch the residual is 0.050 nats — and every centinat of it has a named mechanism and a measured dial (estimator SNR vs. nudge amplitude: the same physics an analog chip will meet, rehearsed in fp32). Muon's advantage over AdamW transfers to EP gradients at full magnitude — this runs on a modern optimizer stack.
On costs, anchored on measured throughput (TF32 validated bit-exact; EP = 3.2× BP FLOPs, measured):
| run | cost, 1.4× failure buffer included |
|---|---|
| 300M × 6B tok — recipe validation on a real corpus | ~$1k |
| **1B × 20B tok, Chinchilla-optimal** | **~$12k — inside the cap** |
The buffer is small because the failure model is not a guess: the three instabilities we met this quarter are each diagnosed and mechanistically closed, and runs checkpoint every 5k steps — **a failed run costs a segment, not the run.** One day on your instances (~$500) converts ±30% into ±10–15%; happy to run that calibration first.
The honest question on our minds is the next octave: **is 3B at full Chinchilla thinkable, and is 7–8B thinkable at all?** (~$25–30k and ~$50k at today's measured throughput — beyond a single capped run.) We are not asking for that now. The proposal is to deliver everything above within the cap first, and let those results decide whether the bigger conversation is worth having.
Separately — and this may interest Rain directly — **we are planning a small analog-hardware LM demo, and the recipe was deliberately designed so that every trained operation has an analog implementation**: divisive (current-mode) normalizations, fixed I/Q rotations for position, translinear gates, the classic subthreshold-exp/KCL circuit for attention's softmax, and learning itself as EP's two local phases. A component-by-component hardware map exists; happy to share it.
Two small accounting questions: is the ~$20k a per-run cap inside a larger envelope, or the envelope itself? And does the backprop-twin control (≈1/3 the EP cost, needed for the head-to-head claim) draw from the same pool?
On the draft — it has since been restructured, so I'd send you the updated PDF before you invest reading time. And as you read, one strategic question we would genuinely value your judgment on. The dynamics story in your hands and the scaling demonstration above are separable: the report stands on its own, and only a slice of it is load-bearing for the LM line. We currently lean toward two papers — the dynamics and its control map now, and the scale demonstration later as a capstone aimed at **the broadest venue the result can carry**, with the dynamics paper as its mechanistic backbone. Merged into one, though, it is a different kind of artifact. You have better instincts than we do about which package travels further.
Best,
Yuren
|
