path: root/docs/hardware/HW_RESEARCH_FINDINGS.md

# Analog-hardware substrate research — findings (2026-06-21)

Deep-research run (108 agents, 25 sources, 118 claims → 22 adversarially-verified 3-0/2-1).
Raw verified claims + source URLs + quotes: `hw_research_claims.json`. Synthesis below is mine.
(The run's auto-synthesis step died on a mid-run /login 401; no DATA lost — all 22 verified claims recovered.)

## THE decisive split confirmed: TRAINABLE-but-small vs LARGE-but-fixed
The single most important filter — does the substrate support **in-situ weight update** (EP needs it) — cleanly partitions the market:

### LARGE but FIXED-WEIGHT (inference-only — fail EP's in-situ filter as-is)
- **Mythic M1076** (analog flash CIM): **80M weights/chip**, eval boards / M.2 / PCIe cards exist. BUT explicitly **inference-only** — train off-device, program once. [mythic.ai, 3-0]
- **IBM HERMES** (PCM, 14nm, 64×256×256 = **4.2M weights**, mixed-signal): research chip, **inference-only**, weights programmed once via hardware-aware training. [Nature Electronics 2023, 3-0]
- **MRAM / PCM crossbars** generally: program-once, fixed during inference; authors state in-situ training "increases energy + degrades device lifespan" → why the whole field avoids it. [Science, NCBI, 3-0]
- → These give SCALE (the tens-of-M you want) but can't do EP's repeated local updates without re-flashing.

### TRAINABLE in-situ (small, but the EP-correct regime)
- **Bulk-switching memristor CIM module** (arXiv 2305.14547): experimentally implements **on-chip mixed-precision TRAINING** with in-situ VMM. KEY mechanism: **digital high-precision update accumulation, physically program the memristor only when accumulated Δw exceeds a threshold** — exactly the hybrid scheme that limits write/endurance stress in an EP loop. [3-0] ← **this is the template for our update path.**
- **In-situ training demonstrated** on memristor crossbars for MLP/CNN/LSTM/RL — local in-array updates during a learning loop are physically real. [arXiv, 3-0]
- Constraints to design around: limited NVM **endurance**, **asymmetric/nonlinear** weight update, variability, retention, stuck-at-faults. Compensation methods exist (stochastic rounding etc.). [escholarship, 2-1/3-0]

## EP / equilibrium learning ALREADY physically realized (precedent exists!)
- **PNAS — self-learning analog resistor network** (Coupled Learning, EP-cousin): XOR + nonlinear regression learned **fully in-situ, NO computer, NO backprop**. Weights = transistor gate-voltage on a local 22µF cap, updated by on-edge circuitry from the **local free-vs-clamped difference**. Forward = physical settling, **τ≈1µs**; learning on 18ms timescale. [PNAS, 3-0] ← **proof the whole concept works in COTS-buildable analog.**
- **EP on D-Wave** (quantum annealer Ising machine): the physical machine does both free + nudge relaxation to steady state (settling is physical). Learning rule is **local** (updates from the two equilibrium states, no backprop). Caveat (1-1): weights live on the classical computer; only couplings loaded per phase → hybrid, not fully in-situ. [Nature, 3-0 on the local-rule claim]
- → EP/local learning on physical equilibrium hardware is **demonstrated**, not speculative. Our contribution would be doing it for a TRANSFORMER block at scale.

## Softmax/attention in analog (the hard part)
- Confirmed open challenge: Transformers need frequent Q/K/V updates, which **conflicts with crossbars' weakness at reprogramming** — flagged as an open HW problem. [arXiv, 3-0]
- (The energy/Hopfield-attention analog-native route verification was among the 3 claims killed by the 401 — needs a re-run. The pragmatic mixed-signal answer — softmax/LN/GELU in FPGA, linear+relaxation in analog — was the framing, not contradicted.)

## BOTTOM LINE for our build (synthesis)
The market splits exactly as feared: **you cannot buy one module that is both tens-of-M AND in-situ-trainable.** So:
- **Phase 1 (trainable, small) — DO THIS FIRST.** Stitch a **bulk-switching/memristor CIM eval module** (in-situ, threshold-accumulated update) + an **FPGA** (softmax/LN/GELU + the EP control loop: settle→nudge→settle→local Δθ). Prove ONE equilibrium-transformer block trains end-to-end via EP in analog. The PNAS resistor-network + the memristor-training paper together show every piece is real.
- **Phase 2 (scale) — LARGE-but-fixed used cleverly.** Use Mythic-80M / HERMES-class for the bulk fixed linear MVM (the relaxation forward), and keep ONLY the trainable/updated weights on the in-situ substrate, OR do mixed-signal "analog-forward, digital-accumulate, periodic-reflash" updates (the threshold-program trick) to tolerate their write limits.
- **Update path = the crux.** Adopt the verified hybrid: **accumulate Δθ in digital high-precision, physically program the analog weight only when |Δθ|>threshold.** This is what makes EP survive endurance limits.
- **De-risk in sim first (free):** the code's `--fnoise` already models multiplicative analog noise — sweep device noise / quantization / asymmetric-update in the 1B sim before buying anything.

## Re-run #2 (2026-06-21, focused) — GAP 1 SOLVED, GAPs 2/3 still thin
Raw: `hw_research_claims2.json`. 107 agents, clean run (no auth drop).

### GAP 1 — analog attention: ANSWERED. It exists, across substrates, but all inference-only.
- **Real fabricated silicon**: UCSD **65nm charge-based SRAM-CIM attention** chip (Moradifirouzabadi/Dodla/Kang, arXiv 2409.04940, ESSERC 2024) — first charge-based analog CIM in SRAM for transformers, **measured 14.8 TOPS/W**, 9-T bitcell does Q·Kᵀ via capacitor charge-sharing. [high]
- **Jülich gain-cell in-memory attention** (Leroux et al., arXiv 2409.19315, Nature Comp Sci 2025): charge-on-capacitor, **~70,000× energy / ~100× speed vs GPU** (simulated). [high]
- Memristor: Nature Sci Reports 2024 self-attention accel (128×128, 2-bit); **STAR RRAM softmax engine** (arXiv 2401.17582). Photonic: TFLN-microring softmax PROPOSAL (arXiv 2603.12934, not fabricated). [high/med]
- **Softmax IS analog-realizable in principle**: a subthreshold source-coupled differential-pair / WTA network computes normalized-exp **"for free" via KCL at the shared tail node** (translinear). [high] — so an energy/LSE-attention analog route is physically grounded.
- **BUT GAP 1(c) CONFIRMED**: real prototypes **overwhelmingly use the mixed-signal split we proposed** — softmax/LN/normalization in DIGITAL/LUT/FPGA, only the linear maps + dot-products in analog. So our architecture choice is the validated one. [high]
- **EVERY analog-attention implementation found is INFERENCE-ONLY / fixed-weight** (Jülich uses offline HW-aware init + offline backprop fine-tune before deploy). Reinforces: nobody has done in-situ-trained analog attention → that IS our novel contribution. [high]
- Noise budget datapoint: a variation-aware memristor-ViT sim tolerates **~35% compute error + ~10% conductance variation** while matching digital Top-1 (MDPI Electronics 2026) — encouraging for the `--fnoise` de-risk. [med]
- Caveat (Sillman, arXiv 2305.13649): an analog softmax block only pays off INSIDE a fully-analog system; isolating it behind ADC/DAC dwarfs the saving → keep softmax digital UNLESS going fully analog. [med]

### GAP 2 (buy-now SKUs) + GAP 3 (endurance/ECRAM) — STILL OPEN
The re-run did NOT substantively verify these (its own summary says so). The one product claim (Knowm $800 kit) was REFUTED/split. So procurement (TetraMem/Mythic/Anadigm/Aspinity SKU+price+order-today) and the **make-or-break endurance budget (RRAM/PCM/FeFET/Flash vs ECRAM writes-to-failure)** remain genuinely unanswered. Indirect signal only: NVM rejected for KV-cache because of slow/high-energy/low-endurance writes; gain-cells chosen for endurance.

### Still to pin (3rd focused pass — procurement + endurance ONLY)
1. SKU-level buy-now: TetraMem MX100, Mythic dev kit, Anadigm AN231E04 board, Aspinity AML100, any RRAM eval kit — orderable today? price? (deep-research struggles here — may need vendor sites / direct contact, not web search.)
2. Per-device **write endurance**: RRAM/PCM/FeFET/Flash/**ECRAM** cycles-to-failure; is ECRAM the symmetric-update + endurance fix, and is it available outside research labs? (Likely research-only — flag if so.)
3. With digital-accumulate-then-threshold-program, how many physical writes does a ~30k-step EP run actually incur, vs device endurance?

## UIUC ECE collaboration map (2026-06-21, user-directed — the hardware-side gap)
User's reachable hardware groups (ALL ECE — this is the team's missing layer). The key insight: the
three UIUC ECE groups span EXACTLY the three layers an in-situ-EP analog demo needs, and together they
SOLVE the market's fatal gap (you can't BUY an in-situ-trainable analog array — but you can fab one in-house):
- **Wenjuan Zhu (UIUC ECE) = DEVICE layer** [user-confirmed]: memristor/RRAM/FeFET / 2D-material devices.
  This is the in-situ-trainable substrate that is research-only on the market — her group can FABRICATE it.
- **Naresh Shanbhag (UIUC ECE) = CIRCUIT/ARCH layer**: SRAM in-memory compute (DIMA/C3SRAM line) — the analog MVM.
- **Pavan Hanumolu (UIUC ECE) = MIXED-SIGNAL GLUE**: ADC/DAC, PLL, switched-cap — the converters + analog
  integrator for the relaxation/control loop (settle→nudge→settle→local Δθ).
- Tao Chen (USTC) = hardware but NOT EP; Stanford = student can broker intros (Wong RRAM / Murmann-legacy CIM / etc.).
STRATEGY SHIFT: not "buy a board" — it's in-house fab of the trainable substrate (Zhu) + CIM circuit (Shanbhag)
+ converter glue (Hanumolu) + our FPGA/EP control loop. Sourcing deep-research w1kuw4zmz profiling all + industry;
its "Zhu" angle mis-targets ML-accel (wrong layer) — corrected to Wenjuan-Zhu-device here; will run a focused
pass on her device work + merge.

### Sourcing run RESULTS (w1kuw4zmz, 11 high-conf named-paper findings; raw: hw_groups_claims.json)
**HEADLINE: Shanbhag (UIUC) is the closest match of ANY named group — and it's the ONLY group silicon that already does analog MVM + genuine on-chip in-situ weight update.**
- **Shanbhag DIMA chip** (Gonugondla/Kang/Shanbhag, **JSSC 2018**, "Variation-Tolerant In-Memory ML Classifier via On-Chip Training"): 65nm, 16kB 6T-SRAM, analog MVM via "functional read" + charge-sharing, AND a **dedicated on-chip digital trainer doing SGD + writing weights back to the array each batch** (random→within 1% of FP in ~400 batches). In-situ training gave **2.4× energy cut** at iso-accuracy. ⇒ the two EP-critical mechanisms (analog MVM + on-chip weight write-back) ALREADY in one fabricated UIUC chip. CAVEAT: it's a single-layer SVM, batch-SGD — **no relaxation/settling loop, no multilayer credit assignment**. We'd add the equilibrium dynamics + two-phase EP rule on top. [high]
- Shanbhag also: **C3SRAM** (w/ ASU, JSSC 2020) — capacitive-coupling XNOR-MAC, but inference-only binary.
- **Hanumolu** — (the run under-covered him; he's the ADC/DAC+integrator glue, still the right converter-layer partner; pin specific silicon separately.)
- **"Zhu"/Lin correction CONFIRMED**: Yingyan **Celine Lin** is UIUC-PhD, now **Georgia Tech** (not current UIUC faculty, surname Lin not Zhu), digital-accel/co-design, **no analog/RRAM/in-situ silicon**. ⇒ the device-layer partner is **Wenjuan Zhu** (user-confirmed), NOT Lin.
- **USTC Tao Chen (陈涛)**: device/materials "in-materio" reservoir computing (disordered dopant-atom networks in Si, Nature 2020). Real device work but NOT circuit-CIM, NOT EP — confirms user's "no EP hardware." Possible device-physics collaborator, not a demo host.
- **Stanford NeuRRAM** (Wong + Raina + UCSD Cauwenberghs, **Nature 2022**): 48-core, ~3M-cell RRAM analog-CIM — the most EP-relevant *substrate* (analog MVM at scale), **BUT INFERENCE-ONLY** (weights programmed offline; only chip-in-the-loop forward fine-tune). Gives the MVM primitive, not native in-situ learning.
  - **Wong = H.-S. Philip Wong (Hon-Sum Philip Wong, 黄汉森)**: Stanford EE, Willard R. & Inez Kerr Bell Professor; also **TSMC Chief Scientist**. RRAM/memristor, 3D monolithic integration, in-memory computing; co-author of NeuRRAM + the canonical "Memristive devices for computing" review (Nature Nanotech 2013). THE RRAM-device heavyweight for the trainable-substrate conversation (reach via the Stanford student contact). TSMC tie = a path to real foundry RRAM.
- **Industry = all inference-only**: TetraMem **MX100** (Nature Electronics 2025; 10 cores, 248×256 1T1R RRAM+RISC-V) ships real silicon but **inference-only** (no in-situ update). Mythic/EnCharge class same.
- **DIY in-situ test-chip path**: **SkyWater S130 + Weebit Nano 256Kb ReRAM IP** (JEDEC/AEC-Q100 qualified 2023, open SKY130 PDK) = foundry RRAM access for an MPW — a lab can fab its OWN trainable RRAM array. [med]
- **EP-on-hardware = still only SPICE sim**: "Memristor Crossbar Circuits Implementing Equilibrium Propagation" (Oh et al., Kookmin U) is circuit simulation, NOT silicon. ⇒ **no fabricated EP-transformer hardware exists anywhere — the demo is genuinely novel.** [high]

**BOTTOM LINE FOR THE PITCH:** lead with **Shanbhag** (his JSSC-2018 chip already proves analog-MVM + on-chip-training in one die — the nearest substrate; we add relaxation + EP) + **Wenjuan Zhu** (trainable device) + **Hanumolu** (converter glue) = a complete in-house UIUC-ECE stack. Stanford **Wong** as the RRAM-device escalation (via the student). Industry (TetraMem/Mythic) only useful for the fixed-weight Phase-2 forward path. Nobody has built EP-transformer hardware → first-mover.

### Hanumolu profile (targeted, 2026-06-21)
**Pavan Kumar Hanumolu** — Seendripu Family Professor, UIUC ECE (since 2013; prior Oregon State); member of CSL's Integrated Circuits & Systems Group. "Top-five mixed-signal IC researchers worldwide," NSF CAREER 2010, heavy JSSC/ISSCC record. Work: **energy-efficient analog/mixed-signal — time-based ADCs, continuous-time filters, ultralow-jitter clocking/PLLs, high-speed serial links, switched-cap, DC-DC power conversion.** ⇒ exactly the **converter + analog-integrator + feedback-loop** layer the EP control loop needs (ADC/DAC glue to read settled states + apply the nudge; switched-cap integrators ARE relaxation-dynamics primitives). Note: his published silicon is converters/links/clocking, NOT CIM — he's the glue/control-loop partner, not the MVM substrate. (Also co-founded Omni Design Technologies — converter IP.) [sources: ece.illinois.edu/.../hanumolu, icsg.csl.illinois.edu]

### GAP 3 — endurance budget: the make-or-break number, and it CLEARS the bar
The feasibility question: an EP run does ~tens-of-thousands of update STEPS; with digital-accumulate-then-threshold-program, physical device writes are FEWER than steps. How many cycles do devices survive?
- **HfOx RRAM: up to ~10^10 cycles** endurance (best-in-class metal-oxide). [arxiv 1909.01771, IOP 10.1088/1361-6641/abf29d]
- AlOx / weaker oxides: only ~10^4 — material choice matters a lot.
- **Budget check**: MNIST-class training writes ~10^4 cycles; gradient training can scale to **~10^8** cycles. ⇒ **HfOx (10^10) has ~100× headroom even over a 10^8-write run** — endurance is NOT a blocker IF you use HfOx-class RRAM + the threshold-accumulate scheme (which cuts writes below step-count). [web-search snippets, med-high]
- **Device-nudge insight**: an EP/Coupled-Learning *nudge* changes resistance far less than a full state write, so per-nudge endurance is plausibly >> rated full-write endurance (needs empirical confirmation, but favorable). 
- **ECRAM (electrochemical RAM)** = the symmetric/linear-analog-update + high-endurance technology specifically aimed at in-situ training: "open-loop analog programmable electrochemical memory array" (Nature Comms 2023, s41467-023-41958-4) — but **research-only** (not commercially available; lab/foundry fab). It's the device-physics frontier Wenjuan Zhu / Wong-type collaborators work in.
- **VERDICT: endurance is survivable** with HfOx-class RRAM (10^10) + threshold-program; ECRAM is the better-but-research-only upgrade. The make-or-break risk is NOT endurance — it's **update linearity/symmetry + device variation** (the asymmetric-nonlinear-update problem), which the digital-accumulate scheme + compensation (stochastic rounding) mitigates. [from earlier run + this]

---

## 2026-07-07 addendum — the J^T wall is now a THEOREM (arXiv:2603.26969), and the price list

**Stern, Frim, Candás, Liu, Balasubramanian, "Contrastive learning in tunable dynamical systems"
(UPenn physical-learning group — the same lineage as our Demo-M blueprint):** for dynamics that break
time-reversal symmetry (= non-reciprocal = our non-conservative operators), *exact* gradient descent
via local rules + a causal, scalable supervisor is impossible **in principle** — the gradient needs the
adjoint (backward-in-time / transposed) propagator, which the forward physics of a non-reciprocal
system cannot supply. Their constructive answer: **PAR (Probably Approximately Right)** — a causal
"forward supervisor" whose update kernel is NOT the gradient but is often positively correlated with
it; learning succeeds iff the average projection is positive.

**What this hardens:** our "Wall 3" (exact AsymEP correction needs J^T v) is physics, not engineering
— *for single-ended, forward-dynamics-only hardware*. Our plain-nudge measurement (cos 0.30 at C512)
is their impossibility measured; our corrected estimator's cos 0.88–0.94 is their PAR correlation
parameter at LM scale.

**The price list (escape routes, ranked):**
1. **Bidirectional crossbars** — the transpose of every LINEAR element is physically free by
   reverse-driving the same array (row-drive vs column-drive; classic analog-BP practice); diagonal
   nonlinear gains are self-transpose (sample-and-hold the operating point). Cost: ~2x terminal
   circuitry (drivers/ADCs both sides). This is the Phase-3 chip spec item (Shanbhag conversation).
2. **Conservative corner (CET energy attention)** — J = J^T by construction: the forward system IS its
   own adjoint; plain EP exact; the theorem's exemption zone. The expressivity tax (no causal masking,
   per our Onsager argument) is now weighed against *exact physical trainability* — CET's stock rises.
3. **PAR acceptance** — run plain contrastive, eat the bias. Price being measured NOW at C512
   (runs/par_baseline: warm s2000 + plain nudge + stability guard; the corrected twin reached 1.75).
4. **Mixed-signal (Demo S, current plan)** — correction computed digitally; the theorem says this
   digital slice cannot be eliminated for free, only bought via (1) or avoided via (2)/(3).

**Demo-M reframe (upgrade):** the OTA board's local contrastive rule on non-reciprocal coupling *is* a
PAR process — position the demo as "PAR learning + our spectral/residual stability control on
non-reciprocal analog hardware": complementary to (and citing) their paper, covering exactly the half
they don't (forward stability), on their home turf.

---

## 2026-07-10 — ANALOG SOFTMAX / ATTENTION DOSSIER (deep-research, 12 claims 3-0 verified; synthesis truncated by API limit)

**Verdict: the whole cascade-analog hypothesis chain is CONFIRMED, and the training gap is REAL.**
Every non-weight op of attention has a demonstrated analog primitive; nobody has trained attention in analog.

### (a) softmax = subthreshold-exp + common-node KCL normalizer — CONFIRMED, and reciprocal
- **Elfadel & Wyatt, NIPS 1993** (`proceedings.neurips.cc/paper/1993/hash/352407221afb776e3143e8a1a0577885`):
  explicit subthreshold-CMOS softmax, i_m = I_c·exp(κv_m/V_0) / Σ exp(κv_p/V_0). exp from subthreshold
  MOS drain current; normalization from KCL at a shared node fed by control current I_c. **Exactly our
  hypothesis.** Temperature/process enter via V_0=kT/q and κ (calibration target).
- **★ Reciprocity (KEY for EP):** their Theorem 1 — the *diode-connected* variant ("entropic resistor")
  is **reciprocal, passive, locally passive**, with log-sum-exp co-content Φ(v)=(1/κ)I_c V_0 ln Σexp(κv_m/V_0).
  The naive gate-in/drain-out softmax is NOT reciprocal and can't compose into energy-based networks.
  **⟹ there exists a softmax circuit that composes into reciprocal EP optimization networks.** This is
  the physical realization of "softmax Jacobian diag(p)−ppᵀ is symmetric ⟹ VJP needs no adjoint" — our
  PAR-wall escape for attention is not just algebraic, it has a named 1993 circuit.
- **Softmax Jacobian symmetry** confirmed from the same paper (DF = diag(F)−FFᵀ, symmetric, singular,
  eigenvalues < 1, softmax = ∇ of a convex potential).
- **Gilbert 1984** 16-channel analog array normalizer (IEEE JSSC SC-19(6):956): divisive normalization
  in silicon since 1984.
- **Lazzaro WTA 1988** (the reference I'd assumed) is actually a HARD max (argmax + log(max)), NOT
  softmax — cite Elfadel&Wyatt for soft, Lazzaro only for the common-node inhibition mechanism.
- **Modern (2023, arXiv:2305.13649):** differential-pair softmax, tail-current-normalized. Measured
  ±0.9% error (subthreshold NMOS, SPICE, N=4); breadboard ±4.2%. Only N=4 ever built — **large N is
  open**, our first tolerance question.
- **★ Tolerance transfer coefficient (plug into E-tier sim):** ±1% device (group-constant Δc) mismatch
  → **≤±1% softmax output error, LINEAR.** (arXiv:2305.13649.) Pelgrom σ(ΔVT)=A_VT/√(WL) is the
  underlying law (Kinget JSSC 2005).

### (b) act×act (QKᵀ, P·V) with transient operands = charge-domain gain cells — CONFIRMED (built, sim+device)
- **Leroux et al., Nature Comp. Sci. 2025** (`nature.com/articles/s43588-025-00854-1`, arXiv:2409.19315):
  gain-cell (capacitor/OSFET, volatile charge) crossbars **simultaneously store the KV cache and compute
  QKᵀ and score·V in analog.** K,V as capacitor voltages in 64×64 arrays; Q & scores as PWM; products
  summed as bitline currents. Precision: 4-bit Q, 3-bit stored K/V, 5-bit out. **This is our hypothesis
  (b) already built.** BUT they **replaced softmax with HardSigmoid** (charge-to-pulse), citing
  sigmoid≈softmax attention up to ~7B — i.e. they sidestepped analog softmax, didn't solve it.
- Memristor route (Bettayeb, Sci.Rep. 2024) hits **write-endurance** limits writing transient operands
  into RRAM → confirms crossbars are wrong for act×act; **charge/capacitor is the right primitive** (KV
  is volatile anyway).

### (c) LayerNorm / divisive normalization = Gilbert normalizer / RMS circuits — prior art exists (as above).

### THE GAP — CONFIRMED unclaimed
Every analog/photonic attention accelerator surveyed **offloads softmax to digital or replaces it**, and
**all are inference-only** (weights trained offline in software, mapped to conductances):
Leroux 2025 (HardSigmoid + software adapt), 65nm hybrid chip arXiv:2409.04940 (softmax+PV digital),
Bettayeb 2024 (exp/log via digital LUT), photonic Lightening-Transformer HPCA2024 & arXiv:2603.12934
(GEMMs optical, softmax electronic). **No one has TRAINED softmax attention in analog with any learning
rule.** Pre-transformer analog neuro had no softmax demand; post-transformer analog work is inference-only.
⟹ "standard-form transformer, softmax intact, trained on analog by a local two-phase rule" is open territory.

### EP-in-crossbars precedent (RQ6): Kendall/Scellier/Bengio 2020 (arXiv:2006.01981) — nonlinear
resistive networks (memristor + diodes) are EBMs by Kirchhoff, trainable by EP with local rules; reverse
crossbar drive = Wᵀ. (Kendall → Rain AI.) Our reciprocal "entropic resistor" softmax slots into exactly
this framework.

**Open for the deferred optimizer-state dossier (wceo0z7fy, hit API limit — rerun later):** analog
momentum (leaky integrator / Loihi eligibility trace), Adam-as-AGC (translinear squarer + RC + divider),
IBM Tiki-Taka aux-array momentum, and the Muon-spectral gap. Hypothesis: Adafactor (row/col shared 2nd
moment = per-line AGC) is the analog-native adaptive optimizer.

---

## 2026-07-11 — OPTIMIZER BoM AUDIT: what each cascade block needs in analog, Adam vs Muon (vs Lion baseline)

Per-block inventory for one standard transformer block, cascade-EP training. Weight count per block
= 12C² (QKV 3C², out C², FFN 8C²). Numbers at C=512 (D1 scale, 3.1M weights/block) and C=1536
(sub-B scale, 28M weights/block). [dossier] = literature-anchored; [est] = engineering estimate.

### Common substrate (optimizer-independent)
| item | count per block | primitive |
|---|---|---|
| weight crossbars | 12C² cells | memristor/PCM, reciprocal → transpose free [dossier: Kendall 2020] |
| KV charge arrays | 2·T·C gain cells (volatile) | Leroux 2025 gain-cell tiles [dossier] |
| softmax normalizers | H per-row common-node circuits, T branches each (time-muxed) | Elfadel-Wyatt entropic resistor [dossier] |
| LayerNorm | 2 divisive-norm/RMS circuits, C channels | Gilbert 1984 normalizer family [dossier] |
| bidirectional periphery | 2× crossbar drivers/sense (reverse drive for Jᵀ) | [dossier: analog-BP/EP practice] |
| outer-product write | pulse-coincidence programming per crossbar | standard analog-training periphery |

### ADAM per block
| ingredient | analog form | added components | verdict |
|---|---|---|---|
| momentum m | per-synapse leaky integrator (cap+leak) OR Tiki-Taka fast aux array | **+12C² cells (+100% analog memory)** — 3.1M @C512, 28M @C1536 | feasible; aux-array form is proven practice [Tiki-Taka; pending re-research] |
| second moment v | per-synapse squarer = economic suicide; **factored (Adafactor-style) row/col statistics** | ~16C AGC channels (translinear squarer + RC + divider, ~10–30 transistors + 1 cap each) ≈ 8k channels @C512 | feasible [est]; "Adafactor is the analog-native Adam" |
| m/√v division | per-LINE programmable driver gain (factored) | C-channel gain DACs per matrix | feasible [est] |
| bias correction / schedules | digital global scalars | negligible | — |
| **total Adam tax** | | **~2× analog memory + ~16C AGC + per-line gain** ; ZERO gradient digitization | **the analog-compatible adaptive optimizer** |
| precision risk | AGC mismatch = per-line lr noise (linear transfer [dossier: ±1%→±1%]) | needs E-tier arm | |

### MUON per block
| ingredient | analog form | added components | verdict |
|---|---|---|---|
| momentum | same as Adam | +12C² cells | feasible |
| Newton-Schulz (X↦aX+b(XXᵀ)X+…, 5 iters) | matrix-MATRIX on a transient matrix = ~10–15·C vector-passes per matrix per step; needs a second C×C act×act fabric per matrix or steals the KV arrays | ~1000× the training step's own MVM duty [est, matches GPU-side analysis] | **analog: economically dead** |
| mixed-signal Muon | digitize FULL gradient/momentum matrix every step: 12C² conversions @ 8–16 bit per block per step (3.1M @C512, 37M/step for L12) + digital NS matmul sidecar + full-precision DAC write-back | ADC energy dominates; this is the exact cost Lion's 1-bit updates avoid | **negates the analog-training story; Muon stays a GPU/digital-era tool** |

### LION baseline (for contrast)
momentum array (+12C²) + 1-bit sign comparators per line + fixed-amplitude pulse writes.
**Minimal periphery, zero ADC traffic, native to pulse programming.** Known quality tax vs AdamW
~15–20% PPL-relative [OmniOpt @1B]; MARS-Lion recovers part.

### Cost ladder (per block, relative)
```
SGD/Lion:   1.0× memory-cells, ~0 ADC/step, trivial periphery      ← analog-native
Adam(fact): 2.0× memory-cells, ~0 ADC/step, +16C AGC channels      ← analog-compatible
Muon:       2.0× memory-cells, +12C² ADC conversions/step + digital NS sidecar ← mixed-signal heavy
```
Recommendation for the chip narrative: **Lion (or SGDM) as the on-chip rule; factored-Adam as the
premium tier; Muon explicitly priced out** (its value is in the GPU/simulation era, where it is
nearly free). Pending verification: Tiki-Taka transfer constants, AGC channel specs, Loihi-trace
leakage numbers — the deferred optimizer-state deep-research (rerun when API budget recovers).