diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-15 10:32:26 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-15 10:32:26 -0500 |
| commit | a6d88a18a4bfd46ab0dd487730b72dcbc3b8b5b8 (patch) | |
| tree | ad6a724c445958c8e5ddca1906fdd9bc63a764a2 /assets/hw/COLLABORATOR_BRIEF.html | |
| parent | 82da86dbd49a093a24e0827331667e8e9e6217df (diff) | |
assets/: organize into decks|figs|hw|refs|misc; scripts + gitignore repointed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'assets/hw/COLLABORATOR_BRIEF.html')
| -rw-r--r-- | assets/hw/COLLABORATOR_BRIEF.html | 98 |
1 files changed, 98 insertions, 0 deletions
diff --git a/assets/hw/COLLABORATOR_BRIEF.html b/assets/hw/COLLABORATOR_BRIEF.html new file mode 100644 index 0000000..48df78c --- /dev/null +++ b/assets/hw/COLLABORATOR_BRIEF.html @@ -0,0 +1,98 @@ +<!DOCTYPE html><html><head><meta charset='utf-8'><style> +body { font-family: 'Helvetica Neue', Arial, sans-serif; font-size: 10.5pt; line-height: 1.38; + max-width: 46em; margin: 0 auto; color: #111; } +h1 { font-size: 15pt; border-bottom: 2px solid #333; padding-bottom: 4px; margin-top: 0.4em;} +h2 { font-size: 12pt; margin-top: 1.1em; border-bottom: 1px solid #bbb; padding-bottom: 2px; } +h3 { font-size: 11pt; margin-top: 1em; } +table { border-collapse: collapse; margin: 0.6em 0; font-size: 9.5pt; } +th, td { border: 1px solid #999; padding: 3px 7px; text-align: left; } +th { background: #eee; } +code, pre { font-family: Menlo, Consolas, monospace; font-size: 9pt; background: #f4f4f4; } +pre { padding: 7px 10px; border: 1px solid #ddd; overflow-x: hidden; white-space: pre-wrap; } +blockquote { border-left: 3px solid #888; margin-left: 0; padding-left: 12px; color: #333; } +li { margin: 2px 0; } +@page { margin: 16mm 15mm; } +</style></head><body><h1>Backprop-free training of transformers — results and a staged hardware program</h1> +<p><strong>One-page brief for hardware-side collaborators · rev. 2026-07-12 · Yuren Hao (UIUC)</strong></p> +<h2>The idea in three sentences</h2> +<p>We train <strong>standard multi-layer transformers</strong> with <strong>Equilibrium Propagation</strong> on a layered energy: +training consists of two relaxation phases and a <strong>local</strong> contrast update per weight — no +backpropagation anywhere — and inference is an ordinary forward pass. On GPU this now works at +language-model scale with essentially no quality gap to backprop. The hardware program starts with +the cheapest object that can validate the physical learning rule — a <strong>clockless twin-network +analog tile (~$300)</strong> — and climbs rung by rung to an in-memory-compute transformer block.</p> +<h2>GPU-scale results (2026-07, measured)</h2> +<ul> +<li><strong>A 12-layer, 42.7M-parameter transformer LM trained for a full epoch (59k steps, 361M tokens) + with no backpropagation in the training loop; it generates coherent text.</strong> To our knowledge the + first transformer language model trained fully this way.</li> +<li><strong>Gap to a tuned, same-architecture backprop control: 0.05 nats</strong> (at 4k steps: statistically + indistinguishable, 3 seeds/arm). Prior backprop-free attempts at scale all report qualitative gaps.</li> +<li>EP step = <strong>3.2× backprop FLOPs</strong> (measured); mixed-precision training validated; the two known + EP-specific instabilities are mechanistically diagnosed and closed (an estimator-SNR floor with a + β-schedule law; a relaxation-contractivity crossing eliminated by norm placement).</li> +<li><strong>Every trained operation chosen analog-implementable:</strong> crossbar MVM, divisive normalization, + fixed I/Q rotations (position code), translinear gated MLP, subthreshold-exponential softmax, + two-phase relaxation for the learning rule.</li> +</ul> +<h2>Measured fault tolerances (fault injection at the trained model)</h2> +<table> +<thead> +<tr> +<th>fault</th> +<th>free</th> +<th>marginal</th> +<th>dead</th> +</tr> +</thead> +<tbody> +<tr> +<td>weight precision</td> +<td><strong>8-bit</strong> (ΔCE +0.004)</td> +<td>6-bit (+0.05)</td> +<td>4-bit</td> +</tr> +<tr> +<td>forward state noise</td> +<td><strong>1%</strong></td> +<td>—</td> +<td>—</td> +</tr> +<tr> +<td>error-channel (nudge) noise</td> +<td><strong>10% relative</strong></td> +<td>30%</td> +<td>—</td> +</tr> +<tr> +<td>divider mismatch / gate gain / phase error</td> +<td>3% / 10% / 0.03 rad</td> +<td>10% / — / 0.1 rad</td> +<td>—</td> +</tr> +</tbody> +</table> +<p>Under every non-fatal fault the learning signal tracks the <em>faulted</em> network (gradient cosine ≈ 0.97 +invariant): <strong>the rule co-adapts to the device.</strong> The only hard spec is ~7-bit effective weights.</p> +<h2>The hardware ladder (each rung publishable alone)</h2> +<ol> +<li><strong>One-edge metrology tile ($70–130):</strong> twin MOSFET edge, shared weight capacitor, exact + (difference-of-squares) and sign-only local update channels, OTA current nudge — no processor, + converter, clock, or sampled memory in the learning loop.</li> +<li><strong>8-edge twin network ($170–300):</strong> nonlinear regression; EP current-nudge vs Coupled-Learning + voltage-clamp on one board; exact-vs-sign update comparison; measured bias-vs-nudge-magnitude + curve (the same β-SNR law we measured in simulation).</li> +<li><strong>32-edge network ($450–900):</strong> replication-class nonlinear tasks, robustness study.</li> +<li><strong>Reciprocal attention microcell (+$100–250):</strong> two tokens, one head, energy-based attention.</li> +<li><strong>CIM transformer block (partner phase):</strong> analog MVM + in-situ two-phase EP weight update — + the piece no shipping analog-AI chip has (all are inference-only or on-chip-backprop).</li> +<li><strong>North star: a few-M-parameter TinyStories LM trained on analog hardware.</strong></li> +</ol> +<h2>What we bring / what we ask</h2> +<p><strong>Bring:</strong> the trained models and recipe, the estimator theory (β-SNR law, stability walls), the +measured tolerance ledger, SPICE-first costed build plan, and parts funding (rungs 1–3 are <$1k). +<strong>Ask (rungs 1–3):</strong> bench access, analog-design mentorship, and/or a student who enjoys discrete +analog — six-week plan, instruments = a scope and a DMM. <strong>Ask (rung 5):</strong> a CIM/mixed-signal +partnership where the substrate expertise is yours and the learning rule is ours.</p> +<p><em>(Detail: CLOCKLESS_ANALOG_MVP_PLAN.md — full BOM, schedule, acceptance criteria, claim limits; +COMPONENT_HW_MAP.md — per-operation analog mapping + tolerance status.)</em></p></body></html>
\ No newline at end of file |