From 7245628fa745c36c07ff33522f9237713deab586 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Sat, 1 Aug 2026 12:41:01 -0500 Subject: =?UTF-8?q?=E6=8A=80=E6=9C=AF=E6=91=98=E8=A6=81(=E7=BB=99Eder/?= =?UTF-8?q?=E6=85=88=E5=96=84=E5=87=BA=E8=B5=84=E6=96=B9):=20=E4=BA=94?= =?UTF-8?q?=E5=B0=BA=E5=AF=B8=E5=AF=B9=E7=85=A7=E8=A1=A8+=E8=AF=BB?= =?UTF-8?q?=E5=87=BA=E5=9C=B0=E6=9D=BF=E7=BB=93=E6=9E=9C+=E7=AE=97?= =?UTF-8?q?=E5=8A=9B=E5=88=86=E6=A1=A3+=E5=BC=80=E6=94=BE=E4=BA=A4?= =?UTF-8?q?=E4=BB=98=E7=89=A9=E6=B8=85=E5=8D=95?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 106 ++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/outreach/EPT_TECHNICAL_SUMMARY.tex (limited to 'docs/outreach/EPT_TECHNICAL_SUMMARY.tex') diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex new file mode 100644 index 0000000..4ac89fc --- /dev/null +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -0,0 +1,106 @@ +\documentclass[10pt]{article} +\usepackage[letterpaper,margin=0.75in,top=0.65in,bottom=0.65in]{geometry} +\usepackage[T1]{fontenc} +\usepackage{newpxtext,newpxmath} +\usepackage{booktabs,array,enumitem,xcolor,titlesec} +\definecolor{accent}{RGB}{138,31,31} +\titleformat{\section}{\normalsize\bfseries\color{accent}}{}{0pt}{} +\titlespacing{\section}{0pt}{10pt}{4pt} +\setlist[enumerate]{leftmargin=1.2em,itemsep=2pt,topsep=2pt,parsep=0pt} +\setlist[itemize]{leftmargin=1.1em,itemsep=2pt,topsep=2pt,parsep=0pt} +\pagestyle{empty} +\setlength{\parindent}{0pt} +\setlength{\parskip}{3.5pt} +\begin{document} + +{\large\bfseries Training language models without backpropagation}\\ +{\small Technical summary \,\textperiodcentered\, Yuren Hao, University of Illinois Urbana-Champaign +\,\textperiodcentered\, August 2026 \,\textperiodcentered\, yurenh2@illinois.edu} + +\section{What we do} +We train standard transformer language models from scratch with Equilibrium Propagation (EP), a +learning rule in which the training signal comes from the system relaxing twice and reading local +differences, with no backward pass anywhere. The trained model is an ordinary transformer and runs +ordinary forward inference. Every design choice is constrained to operations a physical substrate +can perform, because the point of the exercise is a learning rule that analog and other +physics based hardware can run natively. + +\textbf{Why this question is open.} Backpropagation requires a global backward pass that a physical +system cannot perform on itself, so an analog accelerator must either digitize every intermediate +state or carry a second adjoint copy of the hardware. Both destroy the energy advantage that +motivates the substrate. Learning rules that a substrate can run natively do exist, and the +published evidence for them stops at small vision models. Whether they hold at language model scale +is the gap this project addresses. + +\section{Results to date (all against matched backprop twins)} +Twin discipline: identical architecture, tokenizer, data order, optimizer, step budget, and +evaluation. The only difference is the training rule. Two seeds per arm at the smaller sizes, three +backprop seeds at 72M. Corpus is FineWeb-Edu with a 32k vocabulary. + +\begin{center} +\begin{tabular}{@{}lccc@{}} +\toprule +\textbf{Model} & \textbf{EP (val CE)} & \textbf{Backprop twin} & \textbf{Perplexity difference}\\ +\midrule +11M & 4.194 & 4.197 & $-0.4\%$ (EP ahead)\\ +18M & 3.945 & 3.933 & $+1.2\%$\\ +36M & 3.649 & 3.648 & $+0.1\%$\\ +72M & rerun in progress & 3.288 & tracks the twins at matched progress\\ +135M & rerun queued & 3.090 & \\ +\bottomrule +\end{tabular} +\end{center} + +Differences at the first three sizes are smaller than the spread between seeds, so EP and +backpropagation are statistically indistinguishable there. Wall clock cost of EP training in +simulation is about twice backpropagation. To our knowledge the 135M model is the largest language +model trained from scratch with no backpropagation at any layer. + +\section{A result from this week, and why it matters for how we work} +Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every +algorithmic remedy we tried, and looked exactly like a scaling limit of the learning rule. A per +layer probe traced it instead to arithmetic in our own simulator. The training signal is a small +displacement added to a much larger activation; in single precision the components of that +displacement below machine epsilon are destroyed by the addition, and the readout, which recovered +the displacement by subtracting the activation back out, returned the damaged copy without any error +signal. Wider models put more layers below that threshold, which is why the loss looked like a +scaling wall. + +Reading the training signal from the stored displacement directly, never as the difference of two +large numbers, removes the effect at no computational cost. Verification was pre registered: a +double precision control, a test that shifted the affected layers exactly as predicted when the +nudge amplitude was scaled by eight in each direction, a five width probe showing the damage appear +precisely where the displacement ratio crosses the precision threshold, complete closure of the +measured loss on a controlled instrument, and a null test confirming the fix changes nothing at the +sizes that were never below the threshold. + +We report this here because it is representative of how the project is run and because it is likely +to matter to others. Any implementation of this family of learning rules that recovers the contrast +by differencing two large states will hit the same floor, and the symptoms mimic an algorithmic +limitation closely enough that it would be easy to publish the wrong conclusion. + +\section{What compute would buy} +\begin{enumerate} +\item \textbf{Validation at 300M} on 6B tokens with a matched backprop control. Report within three +weeks of access. Roughly 250 H100 hours by measured throughput. +\item \textbf{A matched ladder from 150M to 600M}, two seeds per arm, with the hardware relevant +measurements (quantization, injected device noise, operating windows) run at every size rather than +as a separate track. Roughly 2,750 H100 hours. +\item \textbf{A 1B stage}, 2,000 to 3,000 H100 hours, conditioned on agreed loss and stability gates +from the first two. +\end{enumerate} + +\section{What gets released} +Checkpoints at every size for both the EP models and their backprop twins, full training logs, +training and measurement code, the probe that produced the precision result above, and the negative +results. The intent is a reference artifact for the physical learning and neuromorphic communities: +the empirical answer to whether hardware compatible learning scales, in a form other groups can +build on without spending the compute again. Publication is open by default. + +\section{People} +Yuren Hao (UIUC) leads the project, advised by ChengXiang Zhai on the language modeling side and by +Rainer Engelken on learning dynamics. Collaborators include Alexi Gladstone (UIUC, energy based +models), Xiang Wan (Stanford), and Zeyi Liu (UIUC). The hardware measurement program is being +designed in consultation with experimentalists in physical learning. + +\end{document} -- cgit v1.2.3