From 10badce756a2a293e39cb3db742179b5dcf25b85 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Wed, 29 Jul 2026 18:49:34 -0500 Subject: EPT project page v1 (noindex, grant-review mode) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- index.html | 147 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ robots.txt | 2 + 2 files changed, 149 insertions(+) create mode 100644 index.html create mode 100644 robots.txt diff --git a/index.html b/index.html new file mode 100644 index 0000000..eaa8f7c --- /dev/null +++ b/index.html @@ -0,0 +1,147 @@ + + + + + + +Training Transformer Language Models Without Backpropagation + + + + +

Training Transformer Language Models Without Backpropagation

Standard LLMs, trained end-to-end by Equilibrium Propagation — with matched backprop controls at every scale, and every design choice made under analog-hardware constraints.

+ +

Author list forthcoming · University of Illinois Urbana-Champaign

+ Paper — in preparation + Code — on request + Training curves — on request +

+ +

TL;DR

The largest models ever trained from scratch without backpropagation at any level. + Every parameter update follows the Equilibrium Propagation (EP) rule; no layer, block, or output head + is trained with a backprop rule. Our models are standard transformer LMs (OLMo2-style blocks, + 32k-token vocabulary, FineWeb-Edu) — the trained network runs ordinary forward inference.
Matches backprop within a 4–5% perplexity band at 72M parameters, against a + backprop twin trained on identical data, steps, and optimizer — with multi-seed controls on both sides.
First transformer language model in the EP family, and 2.15× the size of the + largest prior EP-family result (VGG10 image classifier, ∼63M params, Kerjan–Høier–Scellier 2026): + models trained up to 135M parameters on 2.7B tokens.
2× wall-clock overhead versus backprop — where the closest prior EP work pays + 6.7–12×. One nudged phase and 3 relaxation sweeps per step versus their two phases × 10 + iterations: 6.7× fewer relaxation iterations per step.
Verified gradient fidelity, not alignment by luck: the EP update maintains cosine + ≈0.99 to the true backprop gradient throughout training — measured, at scale. Prior non-backprop + transformer results at larger sizes train most parameters with local backprop inside blocks; + ours use none.
Hardware-ready by measurement, not assumption: 8-bit weight/compute quantization shows + no EP-specific penalty against an equally-quantized backprop twin; under injected analog faults + (1% forward noise, 10% error-channel noise, device-tolerance mismatch) the EP estimator tracks the + faulted network at cosine ≈0.97 — learning co-adapts to the hardware.

+ +

Why this matters

Analog and physical accelerators promise order-of-magnitude energy savings for training, but they cannot +run backpropagation natively: exact gradients on a physical substrate require either per-step digitization +or a second, adjoint copy of the hardware. Equilibrium Propagation extracts gradients from the physics +itself — two relaxations and local reads — and is the only member of its family with a +gradient-equivalence guarantee we can verify at scale. The missing evidence has always been scale and +rigor: EP results stopped at mid-size vision models, without matched controls. This project supplies both, +on the model class that matters commercially: language models.

+ +

+The claim we defend: a standard transformer LM can be trained to backprop-class quality with a +physics-compatible learning rule, at 2× backprop wall-clock in simulation — and the constraints that +matter for analog hardware (quantization, noise, nudge operating windows, energy) are measured quantities +in our stack, not assumptions. +

+ +

Headline results

+ + + + + +

Model	Data	EP (val CE)	BP twin	Gap
72M transformer LM	FineWeb-Edu, 1.4B tokens	3.33	3.29	+4–5% ppl
135M transformer LM	FineWeb-Edu, 2.7B tokens	trained end-to-end, zero + instability events; scaling analysis below

Twin discipline: identical architecture, tokenizer, data order, optimizer (Muon hybrid), +steps, and evaluation; multi-seed on both sides (BP n=3, band ±0.006; EP n=2). Comparison row uses the +sealed flagship pair.

+ + + + + +

Cost vs backprop (wall-clock)	This work	Closest EP work (ImageNet VGG10)
Training overhead	2.0×	6.7× (single-sided) / 12× (centered)
Relaxation iterations / step	3 (one phase)	20 (two phases × 10)

+ +

The scaling science

Scaling a physical learning rule surfaces phenomena backprop never meets. Between 72M (width 512) and +135M (width 768) we identified a width-scaling loss in the EP gradient — localized to the top half of +the network, invisible to every per-step alignment metric, and traced to response components that finite +nudge displacement under-reaches. We built a screening instrument that measures this leak in 90 minutes +per candidate recipe, mapped its dose–response law, and demonstrated a pure estimator-side treatment +that recovers 97% of it without touching the model, its inference path, or the cost budget. Full-schedule +validation of the treated recipe is running now; the same instruments give the go/no-go protocol for each +next rung of the ladder (300M → 1B).

+ +

Hardware line

Measured energy projection for an integrated weight-stationary realization: 0.21–0.63 pJ/MAC + (SPICE-measured analog core + datasheet periphery), against a 0.3–1 pJ/MAC digital INT8 system envelope.
Single-column analog prototype: SPICE-modeled, discrete multiplying-DAC parts list — deliberately + kept at the “hardware someone can actually build” level.
Nudge-amplitude operating windows and their evolution over training are mapped — the dynamic-range + spec an analog implementation must meet.

+ +

Roadmap

Staged scaling with matched BP controls and hardware-relevant ablations at every rung: 150M–600M +scaling ladder (does the gap grow or shrink with scale — measured, not assumed), then 1B–3B. +Each stage is gated on the previous stage’s loss, alignment, and throughput numbers. In parallel: the +algorithm→regime map across the activity-difference family (contrastive / coupled-learning arms on the +same harness), and the single-column hardware feasibility study.

+ +

BibTeX

@misc{ept2026,
+  title  = {Training Transformer Language Models Without Backpropagation},
+  author = {(author list forthcoming)},
+  year   = {2026},
+  note   = {Project page}
+}

+ + + + + diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..1f53798 --- /dev/null +++ b/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: / -- cgit v1.2.3