diff options
| -rw-r--r-- | EMAIL_BEN_DRAFT2.md | 37 |
1 files changed, 22 insertions, 15 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md index 265d979..2db8e9f 100644 --- a/EMAIL_BEN_DRAFT2.md +++ b/EMAIL_BEN_DRAFT2.md @@ -1,32 +1,39 @@ -# Draft 5 — reply to Ben (post-reading 2606.03584), 2026-07-11 +# Draft 5 FINAL — follow-up with Overleaf link (morning after the quick reply), 2026-07-12 -Subject: Re: costs — updated numbers, your PCN paper, and some results since the draft +Subject: Re: Overleaf link + the numbers Hi Ben, -$50k total understood — that fits well: hyperparameter work happens at 150–300M where runs cost hundreds of dollars, so most of the envelope survives for the 1B run and its backprop control. +Here is the Overleaf: [LINK] — comment rights enabled. It is the restructured version: the paper is now organized around a control map (what each protection clamps × when its signal acts), the two systems of yesterday's email are kept explicit throughout (the looped transformer is the paper's non-conservative subject; the scaling line is the layered-energy route, and your ImageNet paper is cited as its closest relative), and the experiments your review asked for are in (the jacreg head-to-head, the no-drive collapse arms, the configuration appendix). -Overleaf with comment rights it is. I will send the link shortly — the restructured version, so your comments land on current text. +The results promised yesterday, concretely: -Thank you for the pointer to the PCN-ImageNet paper — we know this object well, from the other direction. **The energy of your Eq. (10) is exactly the training objective of our scaling line**, with transformer blocks in place of conv layers and an autoregressive cross-entropy at the output. Concretely, since the draft: +- **A standard 12-layer transformer (42.7M) trained for a full epoch — 59k steps, 361M tokens — with no backpropagation at any point in the training loop; it generates coherent text.** Inference is a standard forward pass. To our knowledge the first transformer language model fully trained this way (the 1.07B optical-DFA model trains 400M of its parameters and keeps BP at the readout; billion-scale ES results are fine-tuning). A notebook that samples from the trained model is attached — one click, no setup. +- **The gap ledger.** At matched hyperparameters, 4k steps: EP statistically indistinguishable from its tuned backprop control (three seeds each). Full epoch: 0.050 nats behind, with the mechanism identified — the late-training decline of estimator signal-to-noise as the true gradient shrinks against a fixed precision floor (the quantified version of your β remark) — and a β-schedule that removes most of it. Muon's improvement over AdamW transfers to EP gradients at full magnitude. +- **Solver cost.** Our nudged phase runs the full EP step at 3.2× the FLOPs of a backprop step (measured; single-sided, K=3 full-depth sweeps per nudge). For calibration against your Appendix B: that ratio is what turns an 18-day run into a ~4-day one at equal hardware. -- **A standard 12-layer transformer was trained for a full epoch with no backpropagation at any point in the training loop, and it generates coherent text.** Inference is a standard forward pass; a notebook that samples from the trained model is ready if you would like to try it. To our knowledge this is the first transformer language model fully trained without backpropagation (the 1.07B optical-DFA result trains 400M of its parameters and keeps BP at the readout; the billion-scale ES results are fine-tuning) — your VGG10 is, we think, its closest relative, on the vision side. -- **Previous backpropagation-free results, including the strongest ones, retain a quality gap; we measure none.** At matched hyperparameters EP is statistically indistinguishable from its backpropagation control at 4k steps; over a full epoch the difference is 0.050 nats with an identified mechanism and a demonstrated mitigation. On that mechanism: your remark that β must be small for gradient accuracy yet large enough for perturbations to survive machine precision is a sentence we have spent two weeks quantifying — it produces a specific late-training failure mode (the effective gradient signal shrinks as the loss converges while the precision floor does not), and a β-schedule removes it. Muon's improvement over AdamW also transfers to EP gradients at full magnitude. +On costs, anchored on measured throughput (TF32 verified bit-exact): -On your question — with your paper in hand the answer is short. On the Eq. (10) energy the augmented system is conservative, so the antisymmetric term that AsymEP adds vanishes identically, and AsymEP reduces to vanilla EP: **in the feedforward case the two coincide**, including the single-equilibrium simplification (the free-state energy derivative is zero). They part exactly where the field stops being a gradient — untied feedback, or recurrence inside a block, which is the report's subject — and there the asymmetric correction is the term that restores the gradient. One convergence that may amuse you: our nudged-phase solver arrived, independently, at essentially your mod-PGD with asynchronous traversal. - -On costs, based on measured throughput (TF32 verified bit-exact; EP requires 3.2× the FLOPs of backpropagation, measured): - -| run | cost, including a 1.4× failure allowance | +| run | cost, incl. 1.4× failure allowance | |---|---| | 300M × 6B tokens — recipe validation on a standard corpus | ~$1k | | **1B × 20B tokens, Chinchilla-optimal** | **~$12k** | +| 1B backprop control | ~$4k | -The allowance is modest because the failure modes are characterized rather than assumed, and training checkpoints every 5k steps, so a failure loses only the segment since the last checkpoint. A one-day calibration on your instances (~$500) would reduce the uncertainty from ±30% to ±10–15%; we would be glad to run that first. Within $50k this leaves room for a second seed or a first 1.5B segment. The question we are weighing beyond that is **whether 3B at full Chinchilla, or 7–8B at reduced token count, is feasible** (~$25–30k and ~$50k respectively); we are not requesting this now — our intention is to complete the runs above and revisit the larger scale on the basis of those results. +Within $50k this leaves room for a second seed or a first 1.5B segment. The allowance is modest because the failure modes are characterized rather than assumed, and training checkpoints every 5k steps — a failure loses only the segment since the last checkpoint. A one-day calibration on your instances (~$500) would reduce the uncertainty from ±30% to ±10–15%; we would be glad to run that first. -Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** We are planning a small analog-hardware LM demonstration, and every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it. +The question we are weighing beyond that: **whether 3B at full Chinchilla, or 7–8B at reduced token count, is feasible** (~$25–30k and ~$50k respectively at current measured throughput). We are not requesting this now — the intention is to complete the runs above and revisit the larger scale on the basis of those results. -On the draft, one question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable. +Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** Every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it. + +One question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable. Best, Yuren + +--- +## 发送前 checklist（内部,不进邮件） +1. [LINK] 换成真实 Overleaf 链接（zip 已备好: overleaf_dynamics_v3.zip,含两对象改写+KHS26引用+floss重调数字） +2. 附 a1_demo.ipynb（token 已内置） +3. gap 数字核对: f3e3cont 终值若 <1.2808 → "0.050 nats" 段落改为新 gap + "a β-schedule removes most of it" 升级为具体数字 +4. 3.2× 的对照句("18-day → ~4-day")口吻是否合适你再定——它来自他们自己的 Appendix B,准确但直接 |