Training Transformer Language Models Without Backpropagation
TL;DR
- The largest models trained from scratch without backpropagation at any level. Every parameter update follows the Equilibrium Propagation (EP) rule: no layer, block, or output head is trained with a backprop rule. The models are standard transformer LMs (OLMo2-style blocks, 32k vocabulary, FineWeb-Edu); the trained network runs ordinary forward inference.
- Matches backprop within a 4-5% perplexity band at 72M parameters, against a backprop twin trained on identical data, steps, and optimizer: with multi-seed controls on both sides.
- First transformer language model in the EP family, at 2.15× the size of the largest prior EP-family result (VGG10 ImageNet classifier, ~63M params): models trained up to 135M parameters on 2.7B tokens.
- Over 5× cheaper per training step than the closest EP-family work: one nudged phase with 3 relaxation sweeps per step, versus two phases with 10 iterations each (20 total) in the prior state of the art.
- Verified gradient fidelity: the EP update maintains cosine ≈0.99 to the true backprop gradient throughout training: measured, at scale. Larger non-backprop transformers in the literature train most parameters with local backprop inside blocks; ours use none.
- Hardware compatible, with measurements to show it: 8-bit quantization shows no EP-specific penalty against an equally-quantized backprop twin; under injected analog faults (1% forward noise, 10% error-channel noise, device-tolerance mismatch) the EP estimator tracks the faulted network at cosine ≈0.97: learning co-adapts to the hardware.
Abstract
Analog and physical accelerators promise order-of-magnitude energy savings for training, but they cannot run backpropagation natively: exact gradients on a physical substrate require per-step digitization or an adjoint copy of the hardware. Equilibrium Propagation extracts gradients from the physics itself: two relaxations and local reads: and is the only member of its family with a gradient-equivalence guarantee that we verify directly at scale. What the field has lacked is scale and rigor: EP results stopped at mid-size vision models, without matched controls. We train standard transformer language models end-to-end with EP: 72M parameters within 4-5% perplexity of a matched backprop twin, models up to 135M, with backprop twins, multi-seed discipline, and hardware-relevant ablations (quantization, analog faults, nudge operating windows, energy accounting) at every stage. Scaling a physical learning rule also surfaces new science: we identified a width-scaling loss in the EP gradient invisible to per-step alignment metrics, built an instrument that measures it in 90 minutes per candidate recipe, mapped its dose-response law, and demonstrated an estimator-side treatment that recovers 97% of it without touching the model or the cost budget.
Headline results
| Model | Data | EP (val CE) | BP twin | Gap |
|---|---|---|---|---|
| 72M transformer LM | FineWeb-Edu, 1.4B tok | 3.33 | 3.29 | +4-5% ppl |
| 135M transformer LM | FineWeb-Edu, 2.7B tok | trains stably end to end; the frozen 72M recipe does not transfer (gap grows to roughly 30% ppl). See the scaling note below. | ||
Twin discipline: identical architecture, tokenizer, data order, optimizer, steps, and evaluation; multi-seed on both sides (BP n=3, band ±0.006; EP n=2).
| Training cost, EP family | This work | Closest EP work (VGG10, ImageNet) |
|---|---|---|
| Nudged phases per step | 1 | 2 |
| Relaxation iterations per step | 3 | 20 |
| Model class and scale | transformer LM, 72M and 135M | convolutional classifier, ~63M |
The scaling note
Scaling the frozen 72M recipe one width step (512 to 768) fails: the EP to BP gap grows from 4-5% to roughly 30% perplexity. We treat this as the central result of the current phase and the central research object. It has sharp structure: the loss is fully localized to the top half of the network, is invisible to every per-step alignment metric, is untouched by estimator symmetrization, precision, relaxation depth, and output-head interventions, and is removed completely by substituting true gradients in the top half. A mechanism-identification campaign across four widths is in progress, with a pre-registered transfer law and a blind holdout at the next width. Positive scaling claims are suspended until the law predicts an unseen width with zero tuning.
Hardware line
- Measured energy projection for an integrated weight-stationary realization: 0.21-0.63 pJ/MAC (SPICE-measured analog core + datasheet periphery), against a 0.3-1 pJ/MAC digital INT8 system envelope.
- Single-column analog prototype: SPICE-modeled, discrete multiplying-DAC parts list: kept at the “hardware someone can actually build” level.
- Nudge-amplitude operating windows and their evolution over training are mapped: the dynamic-range spec an analog implementation must meet.
Roadmap
Mechanism first: identify the width-scaling term across four measured widths, freeze a transfer law with its thresholds and budgets, then validate blind at 270M with from-scratch runs and two seeds. The scaling ladder resumes only on a zero-tuning hit at the holdout width. In parallel: the algorithm-to-regime map across the activity-difference family, and a bounded single-column hardware feasibility study.
BibTeX
@misc{ept2026,
title={Training Transformer Language Models Without Backpropagation},
author={Hao, Yuren and Wan, Xiang and Gladstone, Alexi and Liu, Zeyi and Zhai, ChengXiang},
year={2026},
note={Project page}
}