diff options
Diffstat (limited to 'EMAIL_BEN_DRAFT2.md')
| -rw-r--r-- | EMAIL_BEN_DRAFT2.md | 6 |
1 files changed, 5 insertions, 1 deletions
diff --git a/EMAIL_BEN_DRAFT2.md b/EMAIL_BEN_DRAFT2.md index 2db8e9f..f367b12 100644 --- a/EMAIL_BEN_DRAFT2.md +++ b/EMAIL_BEN_DRAFT2.md @@ -1,4 +1,4 @@ -# Draft 5 FINAL — follow-up with Overleaf link (morning after the quick reply), 2026-07-12 +# Draft 6 — follow-up: Overleaf link + formulation correction + note (2026-07-12) Subject: Re: Overleaf link + the numbers @@ -6,6 +6,10 @@ Hi Ben, Here is the Overleaf: [LINK] — comment rights enabled. It is the restructured version: the paper is now organized around a control map (what each protection clamps × when its signal acts), the two systems of yesterday's email are kept explicit throughout (the looped transformer is the paper's non-conservative subject; the scaling line is the layered-energy route, and your ImageNet paper is cited as its closest relative), and the experiments your review asked for are in (the jacreg head-to-head, the no-drive collapse arms, the configuration appendix). +First, your correction — you are right, and I withdraw the shortcut. To your direct question: **we use the energy formulation.** Our trainer defines E of your Eq. (10), relaxes the nudged stationarity with a fixed-point scheme of the same family as your mod-PGD, and reads dE/dtheta at the nudged state, single-sided (legal since the free-state read vanishes, as in your Eq. (11)). My "AsymEP reduces to vanilla EP" remark was, as you say, contentless for that case. + +Your sharper question — whether AsymEP on the directly-defined field F_k = f_k(h_{k-1}) − h_k yields the same algorithm — turned out to have a more interesting answer than "yes": **same gradient, different algorithms.** Composing your group's exactness theorem (arXiv:2602.03670, Eqs. 22–25) with the classical EP theorem gives equality of the gradients in the small-β limit; but the two nudged equilibria differ at first order — the energy formulation displaces by −β[(I−L)ᵀ(I−L)]⁻¹∇C (the Gauss–Newton curvature survives exactly because the free-point errors vanish), the corrected field by −β(I−Lᵀ)⁻¹∇C (your Eq. 24) — and the readout operators and bias structures differ correspondingly. The attached one-page note states this precisely, with a numerical check (predicted resolvents match measured displacements to cos = 1.000000; the two states differ at cos 0.93, norm ratio 1.8; both reads converge to autograd). Consistently, your paper's observation that uncorrected VF trains only the last layer on feedforward structure is the degenerate case: without the correction the nudged equilibrium moves no interior layer. If the note survives your reading, it may make a useful appendix for either paper. And quite right on scope — the report's ensemble evidence is MLP/CNN/RNN at 200 seeds; the looped transformer is its in-vivo instance. + The results promised yesterday, concretely: - **A standard 12-layer transformer (42.7M) trained for a full epoch — 59k steps, 361M tokens — with no backpropagation at any point in the training loop; it generates coherent text.** Inference is a standard forward pass. To our knowledge the first transformer language model fully trained this way (the 1.07B optical-DFA model trains 400M of its parameters and keeps BP at the readout; billion-scale ES results are fine-tuning). A notebook that samples from the trained model is attached — one click, no setup. |
