From 14e2adbd11c75098d74f1733f28bbeb9443891a7 Mon Sep 17 00:00:00 2001 From: Yuren Hao <97327730+YurenHao0426@users.noreply.github.com> Date: Thu, 30 Jul 2026 14:51:50 +0700 Subject: Rewrite grant project page (#1) --- index.html | 1758 ++++++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 1429 insertions(+), 329 deletions(-) (limited to 'index.html') diff --git a/index.html b/index.html index 334895b..bd76651 100644 --- a/index.html +++ b/index.html @@ -1,366 +1,1466 @@ - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Training Transformer Language Models Without Backpropagation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + EPT-LM · Physical learning for Transformer language models + + - - - - - -
- -
-
-

More Works from Our Lab

- +
+ +
+ +
+
+
+
+

Grant project brief · July 2026

+

Language models that learn by settling.

+

+ EPT-LM is a practical route to train standard Transformer language models without a + global backward pass. During training, a small physical nudge turns local state + differences into weight updates. At inference, the model is an ordinary Transformer. +

+
+ See the evidence + What funding unlocks
- - - -
-
Paper Title 2
-

Brief description of the work and its main contribution.

- Conference/Journal 2023 +

+ Yuren Hao · University of Illinois Urbana-Champaign +

+
+ + -
- -
-
-
-
-
-
-

Training Transformer Language Models Without Backpropagation

-
- Yuren Hao1, - Xiang Wan2, - Alexi Gladstone1, - Zeyi Liu1, - ChengXiang Zhai1 -
- -
- 1University of Illinois Urbana-Champaign  ·  2Stanford University
2026 -
- -
-
- - - - Paper: in preparation - - - - - Code: on request - - - - - Contact - + +
+
+ Training + local physical read
+
+
+ Free state + model settles +
+
Δ
+
+ Nudged state + target applied +
+
+
Each block updates from its own measured difference
+ +

+ No backward activation tape through the stack. No separate inference architecture. + The same bidirectional operations map naturally onto analog matrix hardware. +

-
-
-
- - - -
-
-
-
-

TL;DR

-
-
    -
  • The largest models trained from scratch without backpropagation at any level. - Every parameter update follows the Equilibrium Propagation (EP) rule: no layer, block, or - output head is trained with a backprop rule. The models are standard transformer LMs - (OLMo2-style blocks, 32k vocabulary, FineWeb-Edu); the trained network runs ordinary forward inference.
    • -
  • Matches backprop within a 4-5% perplexity band at 72M parameters, against a backprop - twin trained on identical data, steps, and optimizer: with multi-seed controls on both sides.
    • -
  • First transformer language model in the EP family, at 2.15× the size of the largest prior - EP-family result (VGG10 ImageNet classifier, ~63M params): models trained up to 135M - parameters on 2.7B tokens.
    • -
  • Over 5× cheaper per training step than the closest EP-family work: one nudged phase - with 3 relaxation sweeps per step, versus two phases with 10 iterations each (20 total) in the - prior state of the art.
    • -
  • Verified gradient fidelity: the EP update maintains cosine ≈0.99 to the true backprop - gradient throughout training: measured, at scale. Larger non-backprop transformers in the - literature train most parameters with local backprop inside blocks; ours use none.
    • -
  • Hardware compatible, with measurements to show it: 8-bit quantization shows no EP-specific - penalty against an equally-quantized backprop twin; under injected analog faults (1% forward noise, - 10% error-channel noise, device-tolerance mismatch) the EP estimator tracks the faulted network at - cosine ≈0.97: learning co-adapts to the hardware.
    • -
+
+ +
+
+
+ 72M + parameter matched-control language model
-
-
-
- - - - -
-
-
-
-

Abstract

-
-

- Analog and physical accelerators promise order-of-magnitude energy savings for training, but they - cannot run backpropagation natively: exact gradients on a physical substrate require per-step - digitization or an adjoint copy of the hardware. Equilibrium Propagation extracts gradients from the - physics itself: two relaxations and local reads: and is the only member of its family with a - gradient-equivalence guarantee that we verify directly at scale. What the field has lacked is scale and - rigor: EP results stopped at mid-size vision models, without matched controls. We train standard - transformer language models end-to-end with EP: 72M parameters within 4-5% perplexity of a - matched backprop twin, models up to 135M, with backprop twins, - multi-seed discipline, and hardware-relevant ablations (quantization, analog faults, nudge operating - windows, energy accounting) at every stage. Scaling a physical learning rule also surfaces new science: - the frozen recipe fails one width step up, with sharp measurable structure. Locating the mechanism, - fitting a transfer law, and validating it blind at the next width is the current phase; the - instruments for that campaign (a 90-minute estimator-quality screen and a stability probe) are built - and calibrated. -

+
+ 1.44B + FineWeb-Edu training tokens
-
-
-
-
- - - - -
-
-
-
-

Headline results

-
- - - - - - -
ModelDataEP (val CE)BP twinGap
72M transformer LMFineWeb-Edu, 1.4B tok3.333.29+4-5% ppl
135M transformer LMFineWeb-Edu, 2.7B toktrains stably end to end; the frozen - 72M recipe does not transfer (gap grows to roughly 30% ppl). See the scaling note below.
-

Twin discipline: identical architecture, tokenizer, data order, - optimizer, steps, and evaluation; multi-seed on both sides (BP n=3, band ±0.006; EP n=2).

- - - - - - - -
Training cost, EP familyThis workClosest EP work (VGG10, ImageNet)
Nudged phases per step12
Relaxation iterations per step320
Model class and scaletransformer LM, 72M and 135Mconvolutional classifier, ~63M
+
+ 4–5% + perplexity gap to its backprop twin +
+
+ ~2× + measured training wall-clock vs backprop
-
-
-
- -
-
-
-
-

The scaling note

-
-

Scaling the frozen 72M recipe one width step (512 to 768) fails: the EP to BP gap grows from - 4-5% to roughly 30% perplexity. We treat this as the central result of the current phase and the - central research object. It has sharp structure: the loss is fully localized to the top half of the - network, is invisible to every per-step alignment metric, is untouched by estimator symmetrization, - precision, relaxation depth, and output-head interventions, and is removed completely by - substituting true gradients in the top half. A mechanism-identification campaign across four widths - is in progress, with a pre-registered transfer law and a blind holdout at the next width. Positive - scaling claims are suspended until the law predicts an unseen width with zero tuning.

+
+ +
+
+

The opportunity

+

Move learning into the machine.

+
+
+

+ Analog and in-memory accelerators can perform matrix operations where the weights + physically reside. Today, training still routes those operations through a digital + backpropagation pipeline: store activations, build a global backward graph, and + shuttle data between compute and memory. +

+

+ Equilibrium Propagation offers a different interface. The hardware runs twice, + once freely and once with a weak target signal. The difference between those settled + states supplies a local learning signal. The device that performs inference can also + participate directly in training. +

+

+ Until now, the evidence stopped at vision-scale demonstrations. EPT-LM brings the + method to standard autoregressive Transformers, modern language data, and controlled + comparisons that make the result useful to both algorithm and hardware reviewers. +

+
+
- +
+ +
+
+

The approach

+

A local update for a standard Transformer.

+

+ The free state is exactly the model’s forward activations. Training adds a weak loss + signal at the output, lets the layer states re-equilibrate, and reads each block locally. + Three operations make up the full step. +

+ +
+
+ 01 / FORWARD +

Run the model normally.

+

+ Tokens pass through distinct, standard Transformer blocks. This is both the free + equilibrium and the deployment-time inference path. +

+
+
+ 02 / NUDGE +

Apply the target once.

+

+ A small output nudge propagates through the bidirectional training dynamics. A few + local relaxation sweeps produce the perturbed state. +

+
+
+ 03 / READ +

Update each block locally.

+

+ Every weight update comes from the block’s own free-to-nudged difference. The + training rule does not store or traverse a global backward tape. +

+
-

Hardware line

-
-
    -
  • Measured energy projection for an integrated weight-stationary realization: - 0.21-0.63 pJ/MAC (SPICE-measured analog core + datasheet periphery), against a - 0.3-1 pJ/MAC digital INT8 system envelope.
    • -
  • Single-column analog prototype: SPICE-modeled, discrete multiplying-DAC parts list: kept at the - “hardware someone can actually build” level.
    • -
  • Nudge-amplitude operating windows and their evolution over training are mapped: the - dynamic-range spec an analog implementation must meet.
    • -
+ +
+ Hardware fit + + Forward and transpose matrix reads, local state storage, and difference measurements + are native operations for bidirectional analog arrays. +
-

Roadmap

-
-

Mechanism first: identify the width-scaling term across four measured widths, freeze a transfer - law with its thresholds and budgets, then validate blind at 270M with from-scratch runs and two - seeds. The scaling ladder resumes only on a zero-tuning hit at the holdout width. In parallel: the - algorithm-to-regime map across the activity-difference family, and a bounded single-column hardware - feasibility study.

+
+
+ +
+
+

Evidence in hand

+

Backprop-class quality at 72M parameters.

+

+ The flagship experiment trains an OLMo2-style, 32k-vocabulary language model from + scratch on FineWeb-Edu. Its control is not a literature number. It is a backprop twin + built and trained in the same codebase. To our knowledge, this is the first Transformer + language model trained end-to-end with Equilibrium Propagation. +

+ +
+
+

Validation cross-entropy · lower is better

+
+
+ 3.29 + Backpropagation twin +
+
+ 3.33 + Equilibrium Propagation +
+
+
+ Difference at the sealed 72M rung + 4–5% perplexity +
+

+ 1.44B tokens · FineWeb-Edu · 12 layers · width 512 · Muon hybrid optimizer · + matched evaluation protocol +

+
+ +
+ +
+
+ ≈0.99 +

Gradient audit

+

+ The local EP update tracks the corresponding backprop update throughout validated + training runs. +

+
+
+ 8-bit +

Compute tolerance

+

+ Matched quantization probes show no additional EP-specific quality penalty at + 8-bit weight and compute precision. +

+
+
+ 135M +

Scaling testbed

+

+ The next model rung has completed a 2.7B-token training budget and now serves as + the mechanism and transfer-law testbed. +

+
-
-
- + +
+
+

What funding unlocks

+

From a compelling result to a predictive engineering program.

+

+ The funded objective is not a collection of per-model recipes. It is one frozen, + measurable training rule that predicts its operating point before the next model is run. + The work is organized around three deliverables. +

+
+
+
01
+
+

Predictive scaling law

+ Deliverable · 150M–600M ladder +
+

+ Derive dimensionless controls for nudge amplitude, state displacement, and + layer-to-layer transmission. Freeze the rule on smaller models, then test it + prospectively on held-out larger widths with matched backprop twins. +

+
- -
-
-
-

BibTeX

- -
- 
@misc{ept2026,
-  title={Training Transformer Language Models Without Backpropagation},
-  author={Hao, Yuren and Wan, Xiang and Gladstone, Alexi and Liu, Zeyi and Zhai, ChengXiang},
-  year={2026},
-  note={Project page}
-}
-
-
- +
+
02
+
+

Hardware-realistic trainer

+ Deliverable · tolerance specification +
+

+ Train under measured device constraints rather than idealized noise: finite + precision, update asymmetry, dynamic range, ADC noise, and device mismatch. The + output is a component-level operating envelope tied to model quality. +

+
+
+
03
+
+

Physical learning demonstrator

+ Deliverable · closed-loop column prototype +
+

+ Build and validate a bidirectional mixed-signal column that performs forward reads, + transpose reads, nudging, and local update extraction. Compare measured behavior + directly against the simulator’s device model. +

+
+
-
+
+
+
+

Train where the weights live.

- This page was built using the Academic Project Page Template which was adopted from the Nerfies project page. - You are free to borrow the source code of this website, we just ask that you link back to this page in the footer.
This website is licensed under a Creative - Commons Attribution-ShareAlike 4.0 International License. + EPT-LM has crossed the first credibility threshold: a standard 72M-parameter language + model, trained from scratch with local equilibrium updates, remains close to its + backprop twin. The next award turns that result into a scaling law and a hardware + specification.

-
+ + Discuss the project +
- - - - - - - +
+ - - - - + + + -- cgit v1.2.3