1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
|
<!DOCTYPE html><html><head><meta charset='utf-8'><style>
body { font-family: 'Helvetica Neue', Arial, sans-serif; font-size: 10.5pt; line-height: 1.38;
max-width: 46em; margin: 0 auto; color: #111; }
h1 { font-size: 15pt; border-bottom: 2px solid #333; padding-bottom: 4px; margin-top: 0.4em;}
h2 { font-size: 12pt; margin-top: 1.1em; border-bottom: 1px solid #bbb; padding-bottom: 2px; }
h3 { font-size: 11pt; margin-top: 1em; }
table { border-collapse: collapse; margin: 0.6em 0; font-size: 9.5pt; }
th, td { border: 1px solid #999; padding: 3px 7px; text-align: left; }
th { background: #eee; }
code, pre { font-family: Menlo, Consolas, monospace; font-size: 9pt; background: #f4f4f4; }
pre { padding: 7px 10px; border: 1px solid #ddd; overflow-x: hidden; white-space: pre-wrap; }
blockquote { border-left: 3px solid #888; margin-left: 0; padding-left: 12px; color: #333; }
li { margin: 2px 0; }
@page { margin: 16mm 15mm; }
</style></head><body><h1>Backprop-free training of transformers — results and a staged hardware program</h1>
<p><strong>One-page brief for hardware-side collaborators · rev. 2026-07-12 · Yuren Hao (UIUC)</strong></p>
<h2>The idea in three sentences</h2>
<p>We train <strong>standard multi-layer transformers</strong> with <strong>Equilibrium Propagation</strong> on a layered energy:
training consists of two relaxation phases and a <strong>local</strong> contrast update per weight — no
backpropagation anywhere — and inference is an ordinary forward pass. On GPU this now works at
language-model scale with essentially no quality gap to backprop. The hardware program starts with
the cheapest object that can validate the physical learning rule — a <strong>clockless twin-network
analog tile (~$300)</strong> — and climbs rung by rung to an in-memory-compute transformer block.</p>
<h2>GPU-scale results (2026-07, measured)</h2>
<ul>
<li><strong>A 12-layer, 42.7M-parameter transformer LM trained for a full epoch (59k steps, 361M tokens)
with no backpropagation in the training loop; it generates coherent text.</strong> To our knowledge the
first transformer language model trained fully this way.</li>
<li><strong>Gap to a tuned, same-architecture backprop control: 0.05 nats</strong> (at 4k steps: statistically
indistinguishable, 3 seeds/arm). Prior backprop-free attempts at scale all report qualitative gaps.</li>
<li>EP step = <strong>3.2× backprop FLOPs</strong> (measured); mixed-precision training validated; the two known
EP-specific instabilities are mechanistically diagnosed and closed (an estimator-SNR floor with a
β-schedule law; a relaxation-contractivity crossing eliminated by norm placement).</li>
<li><strong>Every trained operation chosen analog-implementable:</strong> crossbar MVM, divisive normalization,
fixed I/Q rotations (position code), translinear gated MLP, subthreshold-exponential softmax,
two-phase relaxation for the learning rule.</li>
</ul>
<h2>Measured fault tolerances (fault injection at the trained model)</h2>
<table>
<thead>
<tr>
<th>fault</th>
<th>free</th>
<th>marginal</th>
<th>dead</th>
</tr>
</thead>
<tbody>
<tr>
<td>weight precision</td>
<td><strong>8-bit</strong> (ΔCE +0.004)</td>
<td>6-bit (+0.05)</td>
<td>4-bit</td>
</tr>
<tr>
<td>forward state noise</td>
<td><strong>1%</strong></td>
<td>—</td>
<td>—</td>
</tr>
<tr>
<td>error-channel (nudge) noise</td>
<td><strong>10% relative</strong></td>
<td>30%</td>
<td>—</td>
</tr>
<tr>
<td>divider mismatch / gate gain / phase error</td>
<td>3% / 10% / 0.03 rad</td>
<td>10% / — / 0.1 rad</td>
<td>—</td>
</tr>
</tbody>
</table>
<p>Under every non-fatal fault the learning signal tracks the <em>faulted</em> network (gradient cosine ≈ 0.97
invariant): <strong>the rule co-adapts to the device.</strong> The only hard spec is ~7-bit effective weights.</p>
<h2>The hardware ladder (each rung publishable alone)</h2>
<ol>
<li><strong>One-edge metrology tile ($70–130):</strong> twin MOSFET edge, shared weight capacitor, exact
(difference-of-squares) and sign-only local update channels, OTA current nudge — no processor,
converter, clock, or sampled memory in the learning loop.</li>
<li><strong>8-edge twin network ($170–300):</strong> nonlinear regression; EP current-nudge vs Coupled-Learning
voltage-clamp on one board; exact-vs-sign update comparison; measured bias-vs-nudge-magnitude
curve (the same β-SNR law we measured in simulation).</li>
<li><strong>32-edge network ($450–900):</strong> replication-class nonlinear tasks, robustness study.</li>
<li><strong>Reciprocal attention microcell (+$100–250):</strong> two tokens, one head, energy-based attention.</li>
<li><strong>CIM transformer block (partner phase):</strong> analog MVM + in-situ two-phase EP weight update —
the piece no shipping analog-AI chip has (all are inference-only or on-chip-backprop).</li>
<li><strong>North star: a few-M-parameter TinyStories LM trained on analog hardware.</strong></li>
</ol>
<h2>What we bring / what we ask</h2>
<p><strong>Bring:</strong> the trained models and recipe, the estimator theory (β-SNR law, stability walls), the
measured tolerance ledger, SPICE-first costed build plan, and parts funding (rungs 1–3 are <$1k).
<strong>Ask (rungs 1–3):</strong> bench access, analog-design mentorship, and/or a student who enjoys discrete
analog — six-week plan, instruments = a scope and a DMM. <strong>Ask (rung 5):</strong> a CIM/mixed-signal
partnership where the substrate expertise is yours and the learning rule is ours.</p>
<p><em>(Detail: CLOCKLESS_ANALOG_MVP_PLAN.md — full BOM, schedule, acceptance criteria, claim limits;
COMPONENT_HW_MAP.md — per-operation analog mapping + tolerance status.)</em></p></body></html>
|
