1 files changed, 15 insertions, 16 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md
index de63afd..265d979 100644
--- a/EMAIL_BEN_DRAFT2.md
+++ b/EMAIL_BEN_DRAFT2.md
@@ -1,33 +1,32 @@
-# Draft 4 — reply to Ben (academic register), 2026-07-11
+# Draft 5 — reply to Ben (post-reading 2606.03584), 2026-07-11
 
-Subject: Re: costs — updated numbers, and some results since the draft
+Subject: Re: costs — updated numbers, your PCN paper, and some results since the draft
 
 Hi Ben,
 
-Great to hear — and yes, AsymEP it is ;)
+$50k total understood — that fits well: hyperparameter work happens at 150–300M where runs cost hundreds of dollars, so most of the envelope survives for the 1B run and its backprop control.
 
-First, results obtained since the draft, since they bear on what the credits would fund:
+Overleaf with comment rights it is. I will send the link shortly — the restructured version, so your comments land on current text.
 
-- **A standard 12-layer transformer was trained for a full epoch with no backpropagation at any point in the training loop, and it generates coherent text.** Inference is a standard forward pass. A notebook that samples from the trained model is ready if you would like to try it.
-- To the best of our knowledge, **this is the largest neural network fully trained without backpropagation to date.** The closest claims do not hold under inspection: the 1.07B optical-DFA model trains only 400M of its parameters and retains backpropagation at the readout layer, and the billion-scale evolution-strategies results are fine-tuning of backprop-pretrained models (their from-scratch model is a ~2M-parameter integer GRU).
-- **Previous backpropagation-free methods report a quality gap relative to backpropagation at every scale tested. We observe none.** At matched hyperparameters, EP training is statistically indistinguishable from its backpropagation control at 4k steps; over a full epoch the difference is 0.050 nats, and each contribution to that residual has an identified mechanism and a demonstrated mitigation (estimator signal-to-noise as a function of nudge amplitude — the same quantity that will govern analog implementations). Muon's improvement over AdamW transfers to EP gradients at full magnitude, so the method is compatible with current optimizer practice.
+Thank you for the pointer to the PCN-ImageNet paper — we know this object well, from the other direction. **The energy of your Eq. (10) is exactly the training objective of our scaling line**, with transformer blocks in place of conv layers and an autoregressive cross-entropy at the output. Concretely, since the draft:
 
-On costs, based on measured throughput (TF32 verified bit-exact for this trainer; EP requires 3.2× the FLOPs of backpropagation, measured rather than estimated):
+- **A standard 12-layer transformer was trained for a full epoch with no backpropagation at any point in the training loop, and it generates coherent text.** Inference is a standard forward pass; a notebook that samples from the trained model is ready if you would like to try it. To our knowledge this is the first transformer language model fully trained without backpropagation (the 1.07B optical-DFA result trains 400M of its parameters and keeps BP at the readout; the billion-scale ES results are fine-tuning) — your VGG10 is, we think, its closest relative, on the vision side.
+- **Previous backpropagation-free results, including the strongest ones, retain a quality gap; we measure none.** At matched hyperparameters EP is statistically indistinguishable from its backpropagation control at 4k steps; over a full epoch the difference is 0.050 nats with an identified mechanism and a demonstrated mitigation. On that mechanism: your remark that β must be small for gradient accuracy yet large enough for perturbations to survive machine precision is a sentence we have spent two weeks quantifying — it produces a specific late-training failure mode (the effective gradient signal shrinks as the loss converges while the precision floor does not), and a β-schedule removes it. Muon's improvement over AdamW also transfers to EP gradients at full magnitude.
+
+On your question — with your paper in hand the answer is short. On the Eq. (10) energy the augmented system is conservative, so the antisymmetric term that AsymEP adds vanishes identically, and AsymEP reduces to vanilla EP: **in the feedforward case the two coincide**, including the single-equilibrium simplification (the free-state energy derivative is zero). They part exactly where the field stops being a gradient — untied feedback, or recurrence inside a block, which is the report's subject — and there the asymmetric correction is the term that restores the gradient. One convergence that may amuse you: our nudged-phase solver arrived, independently, at essentially your mod-PGD with asynchronous traversal.
+
+On costs, based on measured throughput (TF32 verified bit-exact; EP requires 3.2× the FLOPs of backpropagation, measured):
 
 | run | cost, including a 1.4× failure allowance |
 |---|---|
 | 300M × 6B tokens — recipe validation on a standard corpus | ~$1k |
-| **1B × 20B tokens, Chinchilla-optimal** | **~$12k — within the cap** |
-
-The failure allowance is modest because the failure modes are characterized rather than assumed: the three instabilities encountered this quarter were each diagnosed and resolved (a nudge-amplitude schedule, an architecture choice, an amplitude floor), and training checkpoints every 5k steps, so a failure loses only the segment since the last checkpoint. A one-day calibration on your instances (~$500) would reduce the uncertainty of these estimates from ±30% to ±10–15%; we would be glad to run that first.
-
-The question we are weighing is the next scale step: **whether 3B at full Chinchilla, or 7–8B at reduced token count, is feasible** (~$25–30k and ~$50k respectively at current measured throughput — beyond a single capped run). We are not requesting this now; our intention is to complete the runs above within the cap, and to revisit the larger scale on the basis of those results.
+| **1B × 20B tokens, Chinchilla-optimal** | **~$12k** |
 
-Separately, and possibly of direct interest to Rain: **we are planning a small analog-hardware LM demonstration, and the training recipe was designed so that every trained operation has a known analog implementation** — divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule itself. A component-by-component hardware mapping is written up; we can share it.
+The allowance is modest because the failure modes are characterized rather than assumed, and training checkpoints every 5k steps, so a failure loses only the segment since the last checkpoint. A one-day calibration on your instances (~$500) would reduce the uncertainty from ±30% to ±10–15%; we would be glad to run that first. Within $50k this leaves room for a second seed or a first 1.5B segment. The question we are weighing beyond that is **whether 3B at full Chinchilla, or 7–8B at reduced token count, is feasible** (~$25–30k and ~$50k respectively); we are not requesting this now — our intention is to complete the runs above and revisit the larger scale on the basis of those results.
 
-Two accounting questions: is the ~$20k a per-run cap within a larger total, or the total itself? And would a backpropagation control run (about one third of the EP cost, required for the comparison) draw from the same allocation?
+Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** We are planning a small analog-hardware LM demonstration, and every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it.
 
-On the draft: it has been substantially restructured since the version you have, so we would send the updated PDF before you invest reading time. One question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable.
+On the draft, one question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable.
 
 Best,
 Yuren