# Draft 6 — follow-up: Overleaf link + formulation correction + note (2026-07-12)

Subject: Re: Overleaf link + the numbers

Hi Ben,

Here is the Overleaf: [LINK] — comment rights enabled. It is the restructured version: the paper is now organized around a control map (what each protection clamps × when its signal acts), the two systems of yesterday's email are kept explicit throughout (the looped transformer is the paper's non-conservative subject; the scaling line is the layered-energy route, and your ImageNet paper is cited as its closest relative), and the experiments your review asked for are in (the jacreg head-to-head, the no-drive collapse arms, the configuration appendix).

First, your correction — you are right, and I withdraw the shortcut. To your direct question: **we use the energy formulation.** Our trainer defines E of your Eq. (10), relaxes the nudged stationarity with a fixed-point scheme of the same family as your mod-PGD, and reads dE/dtheta at the nudged state, single-sided (legal since the free-state read vanishes, as in your Eq. (11)). My "AsymEP reduces to vanilla EP" remark was, as you say, contentless for that case.

Your sharper question — whether AsymEP on the directly-defined field F_k = f_k(h_{k-1}) − h_k yields the same algorithm — turned out to have a more interesting answer than "yes": **same gradient, different algorithms.** Composing your group's exactness theorem (arXiv:2602.03670, Eqs. 22–25) with the classical EP theorem gives equality of the gradients in the small-β limit; but the two nudged equilibria differ at first order — the energy formulation displaces by −β[(I−L)ᵀ(I−L)]⁻¹∇C (the Gauss–Newton curvature survives exactly because the free-point errors vanish), the corrected field by −β(I−Lᵀ)⁻¹∇C (your Eq. 24) — and the readout operators and bias structures differ correspondingly. The attached one-page note states this precisely, with a numerical check (predicted resolvents match measured displacements to cos = 1.000000; the two states differ at cos 0.93, norm ratio 1.8; both reads converge to autograd). Consistently, your paper's observation that uncorrected VF trains only the last layer on feedforward structure is the degenerate case: without the correction the nudged equilibrium moves no interior layer. If the note survives your reading, it may make a useful appendix for either paper. And quite right on scope — the report's ensemble evidence is MLP/CNN/RNN at 200 seeds; the looped transformer is its in-vivo instance.

The results promised yesterday, concretely:

- **A standard 12-layer transformer (42.7M) trained for a full epoch — 59k steps, 361M tokens — with no backpropagation at any point in the training loop; it generates coherent text.** Inference is a standard forward pass. To our knowledge the first transformer language model fully trained this way (the 1.07B optical-DFA model trains 400M of its parameters and keeps BP at the readout; billion-scale ES results are fine-tuning). A notebook that samples from the trained model is attached — one click, no setup.
- **The gap ledger.** At matched hyperparameters, 4k steps: EP statistically indistinguishable from its tuned backprop control (three seeds each). Full epoch: 0.050 nats behind, with the mechanism identified — the late-training decline of estimator signal-to-noise as the true gradient shrinks against a fixed precision floor (the quantified version of your β remark) — and a β-schedule that removes most of it. Muon's improvement over AdamW transfers to EP gradients at full magnitude.
- **Solver cost.** Our nudged phase runs the full EP step at 3.2× the FLOPs of a backprop step (measured; single-sided, K=3 full-depth sweeps per nudge). For calibration against your Appendix B: that ratio is what turns an 18-day run into a ~4-day one at equal hardware.

On costs, at current (July 2026) rental prices — market H100 at $2–2.7/GPU-hour; AWS H200 capacity blocks at ≈$5/GPU-hour after this month's increase — with our measured utilization, the measured 3.2× EP/BP FLOP ratio, mixed-precision EP validated against the fp32 reference this week (gradient cosine unchanged; 1.56× measured wall-clock), and a 1.3× failure allowance:

| run | AWS capacity blocks | market rental |
|---|---|---|
| 300M × 6B tokens — recipe validation on a standard corpus | ~$500 | ~$300 |
| **1B × 20B tokens, Chinchilla-optimal** | **~$5k** | ~$2.5k |
| 1B backprop control, same tokens | ~$1.5k | ~$700 |

Checkpoints every 5k steps bound any failure to the segment since the last one. A one-day calibration on your instances (~$500) would replace these estimates with measurements; we would be glad to run that first.

At these prices the scale question sharpens. Within $50k on AWS, the 1B milestone and its control leave room for either **3B at full Chinchilla scaling (~$40k)** or **7–8B at a GPT-3-style token budget (~$30k)**; at market rates either fits with wide margin. Our intention is unchanged — complete the 1B result first and let it argue for the larger run — but the envelope now reaches one of the two on your infrastructure alone.

Separately, and possibly of direct interest to Rain: your paper leaves open whether F_PCN can be realized efficiently in hardware — **that question is the one our recipe was designed around.** Every trained operation was chosen to have a known analog implementation: divisive (current-mode) normalization, fixed rotations for the position encoding, translinear multipliers for the gated MLP, the subthreshold-exponential/KCL circuit for attention's softmax, and EP's two relaxation phases for the learning rule. A component-by-component hardware mapping is written up; we can share it.

One question we would value your judgment on as you read: the dynamics results and the scaling demonstration are separable, and only part of the former is required for the latter. Our current inclination is two papers — the dynamics and its taxonomy of stability controls now, and the scale demonstration later, submitted to the most general venue the result supports, with the dynamics paper as its theoretical reference. A single merged paper is the alternative. Your judgment on which is the stronger structure would be valuable.

Best,
Yuren

---
## 发送前 checklist（内部,不进邮件）
1. [LINK] 换成真实 Overleaf 链接（zip = **~/aep-dynamics/arxiv_ep_dynamics.zip**,arXiv 模板结构版,
   含两对象改写+KHS26引用+floss重调数字; share 给 benjamin.scellier@gmail.com,开 comment 权限）
2. 附 a1_demo.ipynb（token 已内置）
3. ~~gap 数字核对~~ RESOLVED: f3e3cont 终值 1.2883 > 1.2808 → "0.050 nats" 保持不变
4. 3.2× 的对照句("18-day → ~4-day")口吻是否合适你再定——它来自他们自己的 Appendix B,准确但直接
5. 成本表 2026-07-12 v2.1: 按当日查证实价（市场 H100 $1.87-2.99/h; AWS p5e blocks $4.97/GPU·h,
   7月1日+20%后）+ 实测 amp 1.56×。旧 v1 锚 AWS-A100-savings 已废; 用户 H200 数据点按指示不进邮件,
   仅作 $16.7/EF 最差上限留档。完整推导+价格出处: docs/campaign/COST_MODEL.md
6. **发送门: stage1b_amp (58.8k 全 epoch, GPU1) 完成且 ≈1.28 档才发**——邮件说了 "validated
   against the fp32 reference this week"。若 amp 在 epoch 尺度掉链子 → 表格回退 TF32 (×1.5) 并删该句

---
# Draft 7 — reply to Ben's "LLM bullshit" Overleaf-comments email (2026-07-12 晚)

CONTEXT: costs email (Draft 6 派生, subject "Costs — updated numbers, and some results since the
draft") 已发出; Overleaf 已建并 share; Ben 读了 intro, Overleaf 留了评论: (a) "flavour of LLM
bullshit" — 要求技术概念平语定义 + 合适引用; (b) 合作标准 = 引的每篇都读过; (c) appendix 想法他
接受 ("we should write an appendix"); (d) 问题本身 "interesting and useful"。

Hi Ben,

Thank you — all comments accepted. To be direct about the document's state: the current report is
an LLM-assisted progress artifact, written for speed while the experiments were running, and it
reads like one. The paper version is being rewritten by hand, in the same Overleaf, and your
comments will be addressed there as I go. Concretely: every technical concept will get a
plain-language definition at first use — a supercritical Hopf bifurcation: a settling point that,
as a parameter crosses a threshold, gives way to sustained oscillation of growing-then-saturating
amplitude — with the textbook reference (Kuznetsov, *Elements of Applied Bifurcation Theory*)
rather than an incidental paper; and every citation will be one I have read and can defend. I
agree that this is the standard for working together; it is the standard I want anyway. That the
introduction left the proposed solution unclear is the most useful datapoint I have — the rewrite
will lead with what we actually built.

On one paper versus two, a finding since the draft changes how I would frame the question. The
scaled feedforward line turned out not to need the stability machinery of the dynamics report: on
the layered energy the relaxation is a gradient flow with a symmetric Jacobian, so the oscillatory
instabilities are structurally absent, and the two failure modes it does have — the β-versus-
precision floor, and a curvature/step-size limit of the discretized relaxation — were closed by a
β-schedule and by norm placement, not by the report's controls. The honest relationship is
therefore: the two lines share the estimator theory (finite-β bias and SNR), and they divide the
territory — the dynamics results concern the genuinely recurrent/equilibrium class (DEQ-style
models, physical relaxation substrates), which the feedforward-energy class escapes by
construction. To me that reads as two papers with a shared spine rather than one; but I hold the
view loosely and would take your judgment.

Glad the appendix idea stands — I will bring the note into the Overleaf once the restructuring
settles.

Best,
Yuren

## 要点备忘（内部）
- "I will email you at that time" 类关门句已删——他在 Overleaf 活跃,保持同文档滚动修改
- Hopf 平语定义 + Kuznetsov 教科书引用 = 当场示范非-slop;论文里 normal form 的正确引用就是这本书
- 两-vs-一给的是类边界论证(保守梯度流 vs 非保守循环类),不说"关联不大"
- 手工重写 = 真承诺: report_v3 的重写必须人工过每一句、每条引用逐篇核实