技术摘要(给Eder/慈善出资方): 五尺寸对照表+读出地板结果+算力分档+开放交付物清单

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn

1 files changed, 106 insertions, 0 deletions

diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
new file mode 100644
index 0000000..4ac89fc
--- /dev/null
+++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
@@ -0,0 +1,106 @@
+\documentclass[10pt]{article}
+\usepackage[letterpaper,margin=0.75in,top=0.65in,bottom=0.65in]{geometry}
+\usepackage[T1]{fontenc}
+\usepackage{newpxtext,newpxmath}
+\usepackage{booktabs,array,enumitem,xcolor,titlesec}
+\definecolor{accent}{RGB}{138,31,31}
+\titleformat{\section}{\normalsize\bfseries\color{accent}}{}{0pt}{}
+\titlespacing{\section}{0pt}{10pt}{4pt}
+\setlist[enumerate]{leftmargin=1.2em,itemsep=2pt,topsep=2pt,parsep=0pt}
+\setlist[itemize]{leftmargin=1.1em,itemsep=2pt,topsep=2pt,parsep=0pt}
+\pagestyle{empty}
+\setlength{\parindent}{0pt}
+\setlength{\parskip}{3.5pt}
+\begin{document}
+
+{\large\bfseries Training language models without backpropagation}\\
+{\small Technical summary \,\textperiodcentered\, Yuren Hao, University of Illinois Urbana-Champaign
+\,\textperiodcentered\, August 2026 \,\textperiodcentered\, yurenh2@illinois.edu}
+
+\section{What we do}
+We train standard transformer language models from scratch with Equilibrium Propagation (EP), a
+learning rule in which the training signal comes from the system relaxing twice and reading local
+differences, with no backward pass anywhere. The trained model is an ordinary transformer and runs
+ordinary forward inference. Every design choice is constrained to operations a physical substrate
+can perform, because the point of the exercise is a learning rule that analog and other
+physics based hardware can run natively.
+
+\textbf{Why this question is open.} Backpropagation requires a global backward pass that a physical
+system cannot perform on itself, so an analog accelerator must either digitize every intermediate
+state or carry a second adjoint copy of the hardware. Both destroy the energy advantage that
+motivates the substrate. Learning rules that a substrate can run natively do exist, and the
+published evidence for them stops at small vision models. Whether they hold at language model scale
+is the gap this project addresses.
+
+\section{Results to date (all against matched backprop twins)}
+Twin discipline: identical architecture, tokenizer, data order, optimizer, step budget, and
+evaluation. The only difference is the training rule. Two seeds per arm at the smaller sizes, three
+backprop seeds at 72M. Corpus is FineWeb-Edu with a 32k vocabulary.
+
+\begin{center}
+\begin{tabular}{@{}lccc@{}}
+\toprule
+\textbf{Model} & \textbf{EP (val CE)} & \textbf{Backprop twin} & \textbf{Perplexity difference}\\
+\midrule
+11M  & 4.194 & 4.197 & $-0.4\%$ (EP ahead)\\
+18M  & 3.945 & 3.933 & $+1.2\%$\\
+36M  & 3.649 & 3.648 & $+0.1\%$\\
+72M  & rerun in progress & 3.288 & tracks the twins at matched progress\\
+135M & rerun queued & 3.090 & \\
+\bottomrule
+\end{tabular}
+\end{center}
+
+Differences at the first three sizes are smaller than the spread between seeds, so EP and
+backpropagation are statistically indistinguishable there. Wall clock cost of EP training in
+simulation is about twice backpropagation. To our knowledge the 135M model is the largest language
+model trained from scratch with no backpropagation at any layer.
+
+\section{A result from this week, and why it matters for how we work}
+Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every
+algorithmic remedy we tried, and looked exactly like a scaling limit of the learning rule. A per
+layer probe traced it instead to arithmetic in our own simulator. The training signal is a small
+displacement added to a much larger activation; in single precision the components of that
+displacement below machine epsilon are destroyed by the addition, and the readout, which recovered
+the displacement by subtracting the activation back out, returned the damaged copy without any error
+signal. Wider models put more layers below that threshold, which is why the loss looked like a
+scaling wall.
+
+Reading the training signal from the stored displacement directly, never as the difference of two
+large numbers, removes the effect at no computational cost. Verification was pre registered: a
+double precision control, a test that shifted the affected layers exactly as predicted when the
+nudge amplitude was scaled by eight in each direction, a five width probe showing the damage appear
+precisely where the displacement ratio crosses the precision threshold, complete closure of the
+measured loss on a controlled instrument, and a null test confirming the fix changes nothing at the
+sizes that were never below the threshold.
+
+We report this here because it is representative of how the project is run and because it is likely
+to matter to others. Any implementation of this family of learning rules that recovers the contrast
+by differencing two large states will hit the same floor, and the symptoms mimic an algorithmic
+limitation closely enough that it would be easy to publish the wrong conclusion.
+
+\section{What compute would buy}
+\begin{enumerate}
+\item \textbf{Validation at 300M} on 6B tokens with a matched backprop control. Report within three
+weeks of access. Roughly 250 H100 hours by measured throughput.
+\item \textbf{A matched ladder from 150M to 600M}, two seeds per arm, with the hardware relevant
+measurements (quantization, injected device noise, operating windows) run at every size rather than
+as a separate track. Roughly 2,750 H100 hours.
+\item \textbf{A 1B stage}, 2,000 to 3,000 H100 hours, conditioned on agreed loss and stability gates
+from the first two.
+\end{enumerate}
+
+\section{What gets released}
+Checkpoints at every size for both the EP models and their backprop twins, full training logs,
+training and measurement code, the probe that produced the precision result above, and the negative
+results. The intent is a reference artifact for the physical learning and neuromorphic communities:
+the empirical answer to whether hardware compatible learning scales, in a form other groups can
+build on without spending the compute again. Publication is open by default.
+
+\section{People}
+Yuren Hao (UIUC) leads the project, advised by ChengXiang Zhai on the language modeling side and by
+Rainer Engelken on learning dynamics. Collaborators include Alexi Gladstone (UIUC, energy based
+models), Xiang Wan (Stanford), and Zeyi Liu (UIUC). The hardware measurement program is being
+designed in consultation with experimentalists in physical learning.
+
+\end{document}