1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
|
# Draft 4 — reply to Ben (academic register), 2026-07-11
Subject: Re: costs — updated numbers, and some results since the draft
Hi Ben,
Great to hear — and yes, AsymEP it is ;)
First, results obtained since the draft, since they bear on what the credits would fund:
- **A standard 12-layer transformer was trained for a full epoch with no backpropagation at any point in the training loop, and it generates coherent text.** Inference is a standard forward pass. A notebook that samples from the trained model is ready if you would like to try it.
- To the best of our knowledge, **this is the largest neural network fully trained without backpropagation to date.** The closest claims do not hold under inspection: the 1.07B optical-DFA model trains only 400M of its parameters and retains backpropagation at the readout layer, and the billion-scale evolution-strategies results are fine-tuning of backprop-pretrained models (their from-scratch model is a ~2M-parameter integer GRU).
- **Previous backpropagation-free methods report a quality gap relative to backpropagation at every scale tested. We observe none.** At matched hyperparameters, EP training is statistically indistinguishable from its backpropagation control at 4k steps; over a full epoch the difference is 0.050 nats, and each contribution to that residual has an identified mechanism and a demonstrated mitigation (estimator signal-to-noise as a function of nudge amplitude — the same quantity that will govern analog implementations). Muon's improvement over AdamW transfers to EP gradients at full magnitude, so the method is compatible with current optimizer practice.
On costs, based on measured throughput (TF32 verified bit-exact for this trainer; EP requires 3.2× the FLOPs of backpropagation, measured rather than estimated):
| run | cost, including a 1.4× failure allowance |
|---|---|
| 300M × 6B tokens — recipe validation on a standard corpus | ~$1k |
| **1B × 20B tokens, Chinchilla-optimal** | **~$12k — within the cap** |
The failure allowance is modest because the failure modes are characterized rather than assumed: the three instabilities encountered this quarter were each diagnosed and resolved (a nudge-amplitude schedule, an architecture choice, an amplitude floor), and training checkpoints every 5k steps, so a failure loses only the segment since the last checkpoint. A one-day calibration on your instances (~$500) would reduce the uncertainty of these estimates from ±30% to ±10–15%; we would be glad to run that first.
The question we are weighing is the next scale step: **whether 3B at full Chinchilla, or 7–8B at reduced token count, is feasible** (~$25–30k and ~$50k respectively at current measured throughput — beyond a single capped run). We are not requesting this now; our intention is to complete the runs above within the cap, and to revisit the larger scale on the basis of those results.
Separately, and possibly of direct interest to Rain: **we are planning a small analog-hardware LM demonstration, and the training recipe was designed so that every trained operation has a known analog implementation** — divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule itself. A component-by-component hardware mapping is written up; we can share it.
Two accounting questions: is the ~$20k a per-run cap within a larger total, or the total itself? And would a backpropagation control run (about one third of the EP cost, required for the comparison) draw from the same allocation?
On the draft: it has been substantially restructured since the version you have, so we would send the updated PDF before you invest reading time. One question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable.
Best,
Yuren
|
