Replay the network backward.
The forward pass stores the activations of every block. Training then revisits the whole stack in reverse to assemble one coordinated gradient.
storereverseupdate
Grant project brief · July 2026
Language models that learn by settling.
EPT-LM is a practical route to train standard Transformer language models without a global backward pass. We gently tug the output toward the correct next token; each layer learns from its own response. At inference, the model is an ordinary Transformer.
Same Transformer
Two training interfaces
The model
inference is identical
tokens
→
OLMo2-style
Transformer blocks
Transformer blocks
→
logits
What changes
during training
Backpropagation
store → reverse
One global activation tape
or
EPT-LM
nudge → measure
One local response per layer
The deployed network is unchanged
Backprop reconstructs a global chain of derivatives in software. EPT-LM asks each layer a local question: how did you move when the answer was nudged?
72M
parameter matched-control language model
1.44B
FineWeb-Edu training tokens
BP match
matched quality at 72M parameters
0.4×
training wall-clock vs the closest EP-family baseline
The opportunity
Move learning into the machine.
Analog and in-memory accelerators can perform matrix operations where the weights physically reside. Today, training still routes those operations through a digital backpropagation pipeline: store activations, build a global backward graph, and shuttle data between compute and memory.
Equilibrium Propagation offers a different interface. Run the model, then tug its answer slightly toward the target. Every layer moves in response, and that local before-and-after change supplies its learning signal. The device that performs inference can participate directly in training.
Until now, the evidence stopped at vision-scale demonstrations. EPT-LM brings the method to standard autoregressive Transformers, modern language data, and controlled comparisons that make the result useful to both algorithm and hardware reviewers.
The approach
Two ways to train the same model.
Backpropagation sends one instruction backward through the entire network. EPT-LM perturbs the answer and lets every layer observe its own response. The inference graph, model architecture, and output stay the same.
Give the answer a small tug.
Pull the next-token output slightly toward the correct answer. Each block measures how its own activity changed and converts that response into a local weight update.
runnudgemeasure locally
Hardware fit
Analog arrays already expose forward reads, transpose reads, and local electrical
responses. EPT-LM turns those physical operations into the training interface.
Evidence in hand
Matches backprop at 72M parameters.
The flagship experiment trains an OLMo2-style, 32k-vocabulary language model from scratch on FineWeb-Edu. Its control is not a literature number. It is a backprop twin built and trained in the same codebase. To our knowledge, this is the first Transformer language model trained end-to-end with Equilibrium Propagation.
Validation cross-entropy
3.29
Backpropagation twin
3.33
Equilibrium Propagation
Matched-control result at the sealed 72M rung
BP-matched quality
1.44B tokens · FineWeb-Edu · 12 layers · width 512 · Muon hybrid optimizer · matched evaluation protocol
Gradient audit
The local EP update tracks the corresponding backprop update throughout validated training runs.
Compute tolerance
Matched quantization probes show no additional EP-specific quality penalty at 8-bit weight and compute precision.
Scaling testbed
The next model rung has completed a 2.7B-token training budget and now serves as the mechanism and transfer-law testbed.
What funding unlocks
From a compelling result to a predictive engineering program.
The funded objective is not a collection of per-model recipes. It is one frozen, measurable training rule that predicts its operating point before the next model is run. The work is organized around three deliverables.
01
Predictive scaling law
Deliverable · 0.6B ladder → 1B & 8B validationDerive dimensionless controls for nudge amplitude, state displacement, and layer-to-layer transmission. Map the scaling behavior through 0.6B with matched backprop twins. Freeze the resulting law, then test its predictions prospectively at 1B and 8B.
02
Hardware-realistic trainer
Deliverable · tolerance specificationTrain under measured device constraints rather than idealized noise: finite precision, update asymmetry, dynamic range, ADC noise, and device mismatch. The output is a component-level operating envelope tied to model quality.
03
Physical learning demonstrator
Deliverable · closed-loop column prototypeBuild and validate a bidirectional mixed-signal column that performs forward reads, transpose reads, nudging, and local update extraction. Compare measured behavior directly against the simulator’s device model.
Gate A
Quality transfer
A frozen rule lands inside its preregistered EP–BP quality band.
Gate B
Cost transfer
Relaxation count and precision remain inside the quoted training budget.
Gate C
Device transfer
Measured hardware behavior agrees with the simulator’s tolerance envelope.
Train where the weights live.
EPT-LM has crossed the first credibility threshold: a standard 72M-parameter language model, trained from scratch with local equilibrium updates, matches its backprop twin. The next award turns that result into a scaling law and a hardware specification.