Training Transformer Language Models Without Backpropagation
TL;DR
- The largest models trained from scratch without backpropagation at any level. Every parameter update follows the Equilibrium Propagation (EP) rule — no layer, block, or output head is trained with a backprop rule. The models are standard transformer LMs (OLMo2-style blocks, 32k vocabulary, FineWeb-Edu); the trained network runs ordinary forward inference.
- Matches backprop within a 4–5% perplexity band at 72M parameters, against a backprop twin trained on identical data, steps, and optimizer — with multi-seed controls on both sides.
- First transformer language model in the EP family, at 2.15× the size of the largest prior EP-family result (VGG10 ImageNet classifier, ~63M params): models trained up to 135M parameters on 2.7B tokens.
- 2× wall-clock overhead versus backprop — where the closest prior EP work pays 6.7–12×: one nudged phase × 3 relaxation sweeps per step versus two phases × 10 iterations.
- Verified gradient fidelity: the EP update maintains cosine ≈0.99 to the true backprop gradient throughout training — measured, at scale. Larger non-backprop transformers in the literature train most parameters with local backprop inside blocks; ours use none.
- Hardware-ready by measurement, not assumption: 8-bit quantization shows no EP-specific penalty against an equally-quantized backprop twin; under injected analog faults (1% forward noise, 10% error-channel noise, device-tolerance mismatch) the EP estimator tracks the faulted network at cosine ≈0.97 — learning co-adapts to the hardware.
Abstract
Analog and physical accelerators promise order-of-magnitude energy savings for training, but they cannot run backpropagation natively: exact gradients on a physical substrate require per-step digitization or an adjoint copy of the hardware. Equilibrium Propagation extracts gradients from the physics itself — two relaxations and local reads — and is the only member of its family with a gradient-equivalence guarantee that we verify directly at scale. What the field has lacked is scale and rigor: EP results stopped at mid-size vision models, without matched controls. We train standard transformer language models end-to-end with EP — 72M parameters within 4–5% perplexity of a matched backprop twin, models up to 135M — at 2× backprop wall-clock, with backprop twins, multi-seed discipline, and hardware-relevant ablations (quantization, analog faults, nudge operating windows, energy accounting) at every stage. Scaling a physical learning rule also surfaces new science: we identified a width-scaling loss in the EP gradient invisible to per-step alignment metrics, built an instrument that measures it in 90 minutes per candidate recipe, mapped its dose–response law, and demonstrated an estimator-side treatment that recovers 97% of it without touching the model or the cost budget.
Headline results
| Model | Data | EP (val CE) | BP twin | Gap |
|---|---|---|---|---|
| 72M transformer LM | FineWeb-Edu, 1.4B tok | 3.33 | 3.29 | +4–5% ppl |
| 135M transformer LM | FineWeb-Edu, 2.7B tok | trained end-to-end, zero instability events; scaling analysis below | ||
Twin discipline: identical architecture, tokenizer, data order, optimizer, steps, and evaluation; multi-seed on both sides (BP n=3, band ±0.006; EP n=2).
| Cost vs backprop | This work | Closest EP work (VGG10, ImageNet) |
|---|---|---|
| Wall-clock overhead | 2.0× | 6.7× (single-sided) / 12× (centered) |
| Relaxation iterations / step | 3 (one phase) | 20 (two phases × 10) |
The scaling science
Scaling a physical learning rule surfaces phenomena backprop never meets. Between widths 512 and 768 we identified a width-scaling loss in the EP gradient — localized to the top half of the network, invisible to every per-step alignment metric, and traced to response components that finite nudge displacement under-reaches. We built a screening instrument that measures this leak in 90 minutes per candidate recipe, mapped its dose–response law (logarithmic across two decades of displacement amplification), and demonstrated a pure estimator-side treatment that closes 97% of it — no change to the model, its inference path, or the cost budget. The same instruments provide the go/no-go protocol for each next rung of the ladder.
Hardware line
- Measured energy projection for an integrated weight-stationary realization: 0.21–0.63 pJ/MAC (SPICE-measured analog core + datasheet periphery), against a 0.3–1 pJ/MAC digital INT8 system envelope.
- Single-column analog prototype: SPICE-modeled, discrete multiplying-DAC parts list — kept at the “hardware someone can actually build” level.
- Nudge-amplitude operating windows and their evolution over training are mapped — the dynamic-range spec an analog implementation must meet.
Roadmap
Staged scaling with matched BP controls and hardware-relevant ablations at every rung: a 150M–600M ladder (does the gap grow or shrink with scale — measured, not assumed), then 1B–3B; each stage gated on the previous stage’s loss, alignment, and throughput numbers. In parallel: the algorithm→regime map across the activity-difference family (contrastive / coupled-learning arms on the same harness), and a bounded single-column hardware feasibility study.
BibTeX
@misc{ept2026,
title={Training Transformer Language Models Without Backpropagation},
author={(author list forthcoming)},
year={2026},
note={Project page}
}