From 7245628fa745c36c07ff33522f9237713deab586 Mon Sep 17 00:00:00 2001
From: Yuren Hao <yurenh2@illinois.edu>
Date: Sat, 1 Aug 2026 12:41:01 -0500
Subject: =?UTF-8?q?=E6=8A=80=E6=9C=AF=E6=91=98=E8=A6=81(=E7=BB=99Eder/?=
 =?UTF-8?q?=E6=85=88=E5=96=84=E5=87=BA=E8=B5=84=E6=96=B9):=20=E4=BA=94?=
 =?UTF-8?q?=E5=B0=BA=E5=AF=B8=E5=AF=B9=E7=85=A7=E8=A1=A8+=E8=AF=BB?=
 =?UTF-8?q?=E5=87=BA=E5=9C=B0=E6=9D=BF=E7=BB=93=E6=9E=9C+=E7=AE=97?=
 =?UTF-8?q?=E5=8A=9B=E5=88=86=E6=A1=A3+=E5=BC=80=E6=94=BE=E4=BA=A4?=
 =?UTF-8?q?=E4=BB=98=E7=89=A9=E6=B8=85=E5=8D=95?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
---
 docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 106 ++++++++++++++++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 docs/outreach/EPT_TECHNICAL_SUMMARY.tex

(limited to 'docs/outreach/EPT_TECHNICAL_SUMMARY.tex')

diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
new file mode 100644
index 0000000..4ac89fc
--- /dev/null
+++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
@@ -0,0 +1,106 @@
+\documentclass[10pt]{article}
+\usepackage[letterpaper,margin=0.75in,top=0.65in,bottom=0.65in]{geometry}
+\usepackage[T1]{fontenc}
+\usepackage{newpxtext,newpxmath}
+\usepackage{booktabs,array,enumitem,xcolor,titlesec}
+\definecolor{accent}{RGB}{138,31,31}
+\titleformat{\section}{\normalsize\bfseries\color{accent}}{}{0pt}{}
+\titlespacing{\section}{0pt}{10pt}{4pt}
+\setlist[enumerate]{leftmargin=1.2em,itemsep=2pt,topsep=2pt,parsep=0pt}
+\setlist[itemize]{leftmargin=1.1em,itemsep=2pt,topsep=2pt,parsep=0pt}
+\pagestyle{empty}
+\setlength{\parindent}{0pt}
+\setlength{\parskip}{3.5pt}
+\begin{document}
+
+{\large\bfseries Training language models without backpropagation}\\
+{\small Technical summary \,\textperiodcentered\, Yuren Hao, University of Illinois Urbana-Champaign
+\,\textperiodcentered\, August 2026 \,\textperiodcentered\, yurenh2@illinois.edu}
+
+\section{What we do}
+We train standard transformer language models from scratch with Equilibrium Propagation (EP), a
+learning rule in which the training signal comes from the system relaxing twice and reading local
+differences, with no backward pass anywhere. The trained model is an ordinary transformer and runs
+ordinary forward inference. Every design choice is constrained to operations a physical substrate
+can perform, because the point of the exercise is a learning rule that analog and other
+physics based hardware can run natively.
+
+\textbf{Why this question is open.} Backpropagation requires a global backward pass that a physical
+system cannot perform on itself, so an analog accelerator must either digitize every intermediate
+state or carry a second adjoint copy of the hardware. Both destroy the energy advantage that
+motivates the substrate. Learning rules that a substrate can run natively do exist, and the
+published evidence for them stops at small vision models. Whether they hold at language model scale
+is the gap this project addresses.
+
+\section{Results to date (all against matched backprop twins)}
+Twin discipline: identical architecture, tokenizer, data order, optimizer, step budget, and
+evaluation. The only difference is the training rule. Two seeds per arm at the smaller sizes, three
+backprop seeds at 72M. Corpus is FineWeb-Edu with a 32k vocabulary.
+
+\begin{center}
+\begin{tabular}{@{}lccc@{}}
+\toprule
+\textbf{Model} & \textbf{EP (val CE)} & \textbf{Backprop twin} & \textbf{Perplexity difference}\\
+\midrule
+11M  & 4.194 & 4.197 & $-0.4\%$ (EP ahead)\\
+18M  & 3.945 & 3.933 & $+1.2\%$\\
+36M  & 3.649 & 3.648 & $+0.1\%$\\
+72M  & rerun in progress & 3.288 & tracks the twins at matched progress\\
+135M & rerun queued & 3.090 & \\
+\bottomrule
+\end{tabular}
+\end{center}
+
+Differences at the first three sizes are smaller than the spread between seeds, so EP and
+backpropagation are statistically indistinguishable there. Wall clock cost of EP training in
+simulation is about twice backpropagation. To our knowledge the 135M model is the largest language
+model trained from scratch with no backpropagation at any layer.
+
+\section{A result from this week, and why it matters for how we work}
+Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every
+algorithmic remedy we tried, and looked exactly like a scaling limit of the learning rule. A per
+layer probe traced it instead to arithmetic in our own simulator. The training signal is a small
+displacement added to a much larger activation; in single precision the components of that
+displacement below machine epsilon are destroyed by the addition, and the readout, which recovered
+the displacement by subtracting the activation back out, returned the damaged copy without any error
+signal. Wider models put more layers below that threshold, which is why the loss looked like a
+scaling wall.
+
+Reading the training signal from the stored displacement directly, never as the difference of two
+large numbers, removes the effect at no computational cost. Verification was pre registered: a
+double precision control, a test that shifted the affected layers exactly as predicted when the
+nudge amplitude was scaled by eight in each direction, a five width probe showing the damage appear
+precisely where the displacement ratio crosses the precision threshold, complete closure of the
+measured loss on a controlled instrument, and a null test confirming the fix changes nothing at the
+sizes that were never below the threshold.
+
+We report this here because it is representative of how the project is run and because it is likely
+to matter to others. Any implementation of this family of learning rules that recovers the contrast
+by differencing two large states will hit the same floor, and the symptoms mimic an algorithmic
+limitation closely enough that it would be easy to publish the wrong conclusion.
+
+\section{What compute would buy}
+\begin{enumerate}
+\item \textbf{Validation at 300M} on 6B tokens with a matched backprop control. Report within three
+weeks of access. Roughly 250 H100 hours by measured throughput.
+\item \textbf{A matched ladder from 150M to 600M}, two seeds per arm, with the hardware relevant
+measurements (quantization, injected device noise, operating windows) run at every size rather than
+as a separate track. Roughly 2,750 H100 hours.
+\item \textbf{A 1B stage}, 2,000 to 3,000 H100 hours, conditioned on agreed loss and stability gates
+from the first two.
+\end{enumerate}
+
+\section{What gets released}
+Checkpoints at every size for both the EP models and their backprop twins, full training logs,
+training and measurement code, the probe that produced the precision result above, and the negative
+results. The intent is a reference artifact for the physical learning and neuromorphic communities:
+the empirical answer to whether hardware compatible learning scales, in a form other groups can
+build on without spending the compute again. Publication is open by default.
+
+\section{People}
+Yuren Hao (UIUC) leads the project, advised by ChengXiang Zhai on the language modeling side and by
+Rainer Engelken on learning dynamics. Collaborators include Alexi Gladstone (UIUC, energy based
+models), Xiang Wan (Stanford), and Zeyi Liu (UIUC). The hardware measurement program is being
+designed in consultation with experimentalists in physical learning.
+
+\end{document}
-- 
cgit v1.2.3

