Run the model normally.
Tokens pass through distinct, standard Transformer blocks. This is both the free equilibrium and the deployment-time inference path.
Grant project brief · July 2026
Language models that learn by settling.
EPT-LM is a practical route to train standard Transformer language models without a global backward pass. During training, a small physical nudge turns local state differences into weight updates. At inference, the model is an ordinary Transformer.
Standard model
Training changes · inference does not
Inference
one forward pass
tokens
→
OLMo2-style
Transformer blocks
Transformer blocks
→
logits
Training
local physical read
Free state
model settles
Δ
Nudged state
target applied
Each block updates from its own measured difference
No backward activation tape through the stack. No separate inference architecture. The same bidirectional operations map naturally onto analog matrix hardware.
72M
parameter matched-control language model
1.44B
FineWeb-Edu training tokens
BP match
matched quality at 72M parameters
0.4×
training wall-clock vs the closest EP-family baseline
The opportunity
Move learning into the machine.
Analog and in-memory accelerators can perform matrix operations where the weights physically reside. Today, training still routes those operations through a digital backpropagation pipeline: store activations, build a global backward graph, and shuttle data between compute and memory.
Equilibrium Propagation offers a different interface. The hardware runs twice, once freely and once with a weak target signal. The difference between those settled states supplies a local learning signal. The device that performs inference can also participate directly in training.
Until now, the evidence stopped at vision-scale demonstrations. EPT-LM brings the method to standard autoregressive Transformers, modern language data, and controlled comparisons that make the result useful to both algorithm and hardware reviewers.
The approach
A local update for a standard Transformer.
The free state is exactly the model’s forward activations. Training adds a weak loss signal at the output, lets the layer states re-equilibrate, and reads each block locally. Three operations make up the full step.
Apply the target once.
A small output nudge propagates through the bidirectional training dynamics. A few local relaxation sweeps produce the perturbed state.
Update each block locally.
Every weight update comes from the block’s own free-to-nudged difference. The training rule does not store or traverse a global backward tape.
Hardware fit
Forward and transpose matrix reads, local state storage, and difference measurements
are native operations for bidirectional analog arrays.
Evidence in hand
Matches backprop at 72M parameters.
The flagship experiment trains an OLMo2-style, 32k-vocabulary language model from scratch on FineWeb-Edu. Its control is not a literature number. It is a backprop twin built and trained in the same codebase. To our knowledge, this is the first Transformer language model trained end-to-end with Equilibrium Propagation.
Validation cross-entropy · lower is better
3.29
Backpropagation twin
3.33
Equilibrium Propagation
Matched-control result at the sealed 72M rung
BP-matched quality
1.44B tokens · FineWeb-Edu · 12 layers · width 512 · Muon hybrid optimizer · matched evaluation protocol
Gradient audit
The local EP update tracks the corresponding backprop update throughout validated training runs.
Compute tolerance
Matched quantization probes show no additional EP-specific quality penalty at 8-bit weight and compute precision.
Scaling testbed
The next model rung has completed a 2.7B-token training budget and now serves as the mechanism and transfer-law testbed.
What funding unlocks
From a compelling result to a predictive engineering program.
The funded objective is not a collection of per-model recipes. It is one frozen, measurable training rule that predicts its operating point before the next model is run. The work is organized around three deliverables.
01
Predictive scaling law
Deliverable · 0.6B ladder → 1B & 8B validationDerive dimensionless controls for nudge amplitude, state displacement, and layer-to-layer transmission. Map the scaling behavior through 0.6B with matched backprop twins. Freeze the resulting law, then test its predictions prospectively at 1B and 8B.
02
Hardware-realistic trainer
Deliverable · tolerance specificationTrain under measured device constraints rather than idealized noise: finite precision, update asymmetry, dynamic range, ADC noise, and device mismatch. The output is a component-level operating envelope tied to model quality.
03
Physical learning demonstrator
Deliverable · closed-loop column prototypeBuild and validate a bidirectional mixed-signal column that performs forward reads, transpose reads, nudging, and local update extraction. Compare measured behavior directly against the simulator’s device model.
Gate A
Quality transfer
A frozen rule lands inside its preregistered EP–BP quality band.
Gate B
Cost transfer
Relaxation count and precision remain inside the quoted training budget.
Gate C
Device transfer
Measured hardware behavior agrees with the simulator’s tolerance envelope.
Train where the weights live.
EPT-LM has crossed the first credibility threshold: a standard 72M-parameter language model, trained from scratch with local equilibrium updates, matches its backprop twin. The next award turns that result into a scaling law and a hardware specification.