# Analog-hardware substrate research — findings (2026-06-21)

Deep-research run (108 agents, 25 sources, 118 claims → 22 adversarially-verified 3-0/2-1).
Raw verified claims + source URLs + quotes: `hw_research_claims.json`. Synthesis below is mine.
(The run's auto-synthesis step died on a mid-run /login 401; no DATA lost — all 22 verified claims recovered.)

## THE decisive split confirmed: TRAINABLE-but-small vs LARGE-but-fixed
The single most important filter — does the substrate support **in-situ weight update** (EP needs it) — cleanly partitions the market:

### LARGE but FIXED-WEIGHT (inference-only — fail EP's in-situ filter as-is)
- **Mythic M1076** (analog flash CIM): **80M weights/chip**, eval boards / M.2 / PCIe cards exist. BUT explicitly **inference-only** — train off-device, program once. [mythic.ai, 3-0]
- **IBM HERMES** (PCM, 14nm, 64×256×256 = **4.2M weights**, mixed-signal): research chip, **inference-only**, weights programmed once via hardware-aware training. [Nature Electronics 2023, 3-0]
- **MRAM / PCM crossbars** generally: program-once, fixed during inference; authors state in-situ training "increases energy + degrades device lifespan" → why the whole field avoids it. [Science, NCBI, 3-0]
- → These give SCALE (the tens-of-M you want) but can't do EP's repeated local updates without re-flashing.

### TRAINABLE in-situ (small, but the EP-correct regime)
- **Bulk-switching memristor CIM module** (arXiv 2305.14547): experimentally implements **on-chip mixed-precision TRAINING** with in-situ VMM. KEY mechanism: **digital high-precision update accumulation, physically program the memristor only when accumulated Δw exceeds a threshold** — exactly the hybrid scheme that limits write/endurance stress in an EP loop. [3-0] ← **this is the template for our update path.**
- **In-situ training demonstrated** on memristor crossbars for MLP/CNN/LSTM/RL — local in-array updates during a learning loop are physically real. [arXiv, 3-0]
- Constraints to design around: limited NVM **endurance**, **asymmetric/nonlinear** weight update, variability, retention, stuck-at-faults. Compensation methods exist (stochastic rounding etc.). [escholarship, 2-1/3-0]

## EP / equilibrium learning ALREADY physically realized (precedent exists!)
- **PNAS — self-learning analog resistor network** (Coupled Learning, EP-cousin): XOR + nonlinear regression learned **fully in-situ, NO computer, NO backprop**. Weights = transistor gate-voltage on a local 22µF cap, updated by on-edge circuitry from the **local free-vs-clamped difference**. Forward = physical settling, **τ≈1µs**; learning on 18ms timescale. [PNAS, 3-0] ← **proof the whole concept works in COTS-buildable analog.**
- **EP on D-Wave** (quantum annealer Ising machine): the physical machine does both free + nudge relaxation to steady state (settling is physical). Learning rule is **local** (updates from the two equilibrium states, no backprop). Caveat (1-1): weights live on the classical computer; only couplings loaded per phase → hybrid, not fully in-situ. [Nature, 3-0 on the local-rule claim]
- → EP/local learning on physical equilibrium hardware is **demonstrated**, not speculative. Our contribution would be doing it for a TRANSFORMER block at scale.

## Softmax/attention in analog (the hard part)
- Confirmed open challenge: Transformers need frequent Q/K/V updates, which **conflicts with crossbars' weakness at reprogramming** — flagged as an open HW problem. [arXiv, 3-0]
- (The energy/Hopfield-attention analog-native route verification was among the 3 claims killed by the 401 — needs a re-run. The pragmatic mixed-signal answer — softmax/LN/GELU in FPGA, linear+relaxation in analog — was the framing, not contradicted.)

## BOTTOM LINE for our build (synthesis)
The market splits exactly as feared: **you cannot buy one module that is both tens-of-M AND in-situ-trainable.** So:
- **Phase 1 (trainable, small) — DO THIS FIRST.** Stitch a **bulk-switching/memristor CIM eval module** (in-situ, threshold-accumulated update) + an **FPGA** (softmax/LN/GELU + the EP control loop: settle→nudge→settle→local Δθ). Prove ONE equilibrium-transformer block trains end-to-end via EP in analog. The PNAS resistor-network + the memristor-training paper together show every piece is real.
- **Phase 2 (scale) — LARGE-but-fixed used cleverly.** Use Mythic-80M / HERMES-class for the bulk fixed linear MVM (the relaxation forward), and keep ONLY the trainable/updated weights on the in-situ substrate, OR do mixed-signal "analog-forward, digital-accumulate, periodic-reflash" updates (the threshold-program trick) to tolerate their write limits.
- **Update path = the crux.** Adopt the verified hybrid: **accumulate Δθ in digital high-precision, physically program the analog weight only when |Δθ|>threshold.** This is what makes EP survive endurance limits.
- **De-risk in sim first (free):** the code's `--fnoise` already models multiplicative analog noise — sweep device noise / quantization / asymmetric-update in the 1B sim before buying anything.

## Re-run #2 (2026-06-21, focused) — GAP 1 SOLVED, GAPs 2/3 still thin
Raw: `hw_research_claims2.json`. 107 agents, clean run (no auth drop).

### GAP 1 — analog attention: ANSWERED. It exists, across substrates, but all inference-only.
- **Real fabricated silicon**: UCSD **65nm charge-based SRAM-CIM attention** chip (Moradifirouzabadi/Dodla/Kang, arXiv 2409.04940, ESSERC 2024) — first charge-based analog CIM in SRAM for transformers, **measured 14.8 TOPS/W**, 9-T bitcell does Q·Kᵀ via capacitor charge-sharing. [high]
- **Jülich gain-cell in-memory attention** (Leroux et al., arXiv 2409.19315, Nature Comp Sci 2025): charge-on-capacitor, **~70,000× energy / ~100× speed vs GPU** (simulated). [high]
- Memristor: Nature Sci Reports 2024 self-attention accel (128×128, 2-bit); **STAR RRAM softmax engine** (arXiv 2401.17582). Photonic: TFLN-microring softmax PROPOSAL (arXiv 2603.12934, not fabricated). [high/med]
- **Softmax IS analog-realizable in principle**: a subthreshold source-coupled differential-pair / WTA network computes normalized-exp **"for free" via KCL at the shared tail node** (translinear). [high] — so an energy/LSE-attention analog route is physically grounded.
- **BUT GAP 1(c) CONFIRMED**: real prototypes **overwhelmingly use the mixed-signal split we proposed** — softmax/LN/normalization in DIGITAL/LUT/FPGA, only the linear maps + dot-products in analog. So our architecture choice is the validated one. [high]
- **EVERY analog-attention implementation found is INFERENCE-ONLY / fixed-weight** (Jülich uses offline HW-aware init + offline backprop fine-tune before deploy). Reinforces: nobody has done in-situ-trained analog attention → that IS our novel contribution. [high]
- Noise budget datapoint: a variation-aware memristor-ViT sim tolerates **~35% compute error + ~10% conductance variation** while matching digital Top-1 (MDPI Electronics 2026) — encouraging for the `--fnoise` de-risk. [med]
- Caveat (Sillman, arXiv 2305.13649): an analog softmax block only pays off INSIDE a fully-analog system; isolating it behind ADC/DAC dwarfs the saving → keep softmax digital UNLESS going fully analog. [med]

### GAP 2 (buy-now SKUs) + GAP 3 (endurance/ECRAM) — STILL OPEN
The re-run did NOT substantively verify these (its own summary says so). The one product claim (Knowm $800 kit) was REFUTED/split. So procurement (TetraMem/Mythic/Anadigm/Aspinity SKU+price+order-today) and the **make-or-break endurance budget (RRAM/PCM/FeFET/Flash vs ECRAM writes-to-failure)** remain genuinely unanswered. Indirect signal only: NVM rejected for KV-cache because of slow/high-energy/low-endurance writes; gain-cells chosen for endurance.

### Still to pin (3rd focused pass — procurement + endurance ONLY)
1. SKU-level buy-now: TetraMem MX100, Mythic dev kit, Anadigm AN231E04 board, Aspinity AML100, any RRAM eval kit — orderable today? price? (deep-research struggles here — may need vendor sites / direct contact, not web search.)
2. Per-device **write endurance**: RRAM/PCM/FeFET/Flash/**ECRAM** cycles-to-failure; is ECRAM the symmetric-update + endurance fix, and is it available outside research labs? (Likely research-only — flag if so.)
3. With digital-accumulate-then-threshold-program, how many physical writes does a ~30k-step EP run actually incur, vs device endurance?

## UIUC ECE collaboration map (2026-06-21, user-directed — the hardware-side gap)
User's reachable hardware groups (ALL ECE — this is the team's missing layer). The key insight: the
three UIUC ECE groups span EXACTLY the three layers an in-situ-EP analog demo needs, and together they
SOLVE the market's fatal gap (you can't BUY an in-situ-trainable analog array — but you can fab one in-house):
- **Wenjuan Zhu (UIUC ECE) = DEVICE layer** [user-confirmed]: memristor/RRAM/FeFET / 2D-material devices.
  This is the in-situ-trainable substrate that is research-only on the market — her group can FABRICATE it.
- **Naresh Shanbhag (UIUC ECE) = CIRCUIT/ARCH layer**: SRAM in-memory compute (DIMA/C3SRAM line) — the analog MVM.
- **Pavan Hanumolu (UIUC ECE) = MIXED-SIGNAL GLUE**: ADC/DAC, PLL, switched-cap — the converters + analog
  integrator for the relaxation/control loop (settle→nudge→settle→local Δθ).
- Tao Chen (USTC) = hardware but NOT EP; Stanford = student can broker intros (Wong RRAM / Murmann-legacy CIM / etc.).
STRATEGY SHIFT: not "buy a board" — it's in-house fab of the trainable substrate (Zhu) + CIM circuit (Shanbhag)
+ converter glue (Hanumolu) + our FPGA/EP control loop. Sourcing deep-research w1kuw4zmz profiling all + industry;
its "Zhu" angle mis-targets ML-accel (wrong layer) — corrected to Wenjuan-Zhu-device here; will run a focused
pass on her device work + merge.

### Sourcing run RESULTS (w1kuw4zmz, 11 high-conf named-paper findings; raw: hw_groups_claims.json)
**HEADLINE: Shanbhag (UIUC) is the closest match of ANY named group — and it's the ONLY group silicon that already does analog MVM + genuine on-chip in-situ weight update.**
- **Shanbhag DIMA chip** (Gonugondla/Kang/Shanbhag, **JSSC 2018**, "Variation-Tolerant In-Memory ML Classifier via On-Chip Training"): 65nm, 16kB 6T-SRAM, analog MVM via "functional read" + charge-sharing, AND a **dedicated on-chip digital trainer doing SGD + writing weights back to the array each batch** (random→within 1% of FP in ~400 batches). In-situ training gave **2.4× energy cut** at iso-accuracy. ⇒ the two EP-critical mechanisms (analog MVM + on-chip weight write-back) ALREADY in one fabricated UIUC chip. CAVEAT: it's a single-layer SVM, batch-SGD — **no relaxation/settling loop, no multilayer credit assignment**. We'd add the equilibrium dynamics + two-phase EP rule on top. [high]
- Shanbhag also: **C3SRAM** (w/ ASU, JSSC 2020) — capacitive-coupling XNOR-MAC, but inference-only binary.
- **Hanumolu** — (the run under-covered him; he's the ADC/DAC+integrator glue, still the right converter-layer partner; pin specific silicon separately.)
- **"Zhu"/Lin correction CONFIRMED**: Yingyan **Celine Lin** is UIUC-PhD, now **Georgia Tech** (not current UIUC faculty, surname Lin not Zhu), digital-accel/co-design, **no analog/RRAM/in-situ silicon**. ⇒ the device-layer partner is **Wenjuan Zhu** (user-confirmed), NOT Lin.
- **USTC Tao Chen (陈涛)**: device/materials "in-materio" reservoir computing (disordered dopant-atom networks in Si, Nature 2020). Real device work but NOT circuit-CIM, NOT EP — confirms user's "no EP hardware." Possible device-physics collaborator, not a demo host.
- **Stanford NeuRRAM** (Wong + Raina + UCSD Cauwenberghs, **Nature 2022**): 48-core, ~3M-cell RRAM analog-CIM — the most EP-relevant *substrate* (analog MVM at scale), **BUT INFERENCE-ONLY** (weights programmed offline; only chip-in-the-loop forward fine-tune). Gives the MVM primitive, not native in-situ learning.
  - **Wong = H.-S. Philip Wong (Hon-Sum Philip Wong, 黄汉森)**: Stanford EE, Willard R. & Inez Kerr Bell Professor; also **TSMC Chief Scientist**. RRAM/memristor, 3D monolithic integration, in-memory computing; co-author of NeuRRAM + the canonical "Memristive devices for computing" review (Nature Nanotech 2013). THE RRAM-device heavyweight for the trainable-substrate conversation (reach via the Stanford student contact). TSMC tie = a path to real foundry RRAM.
- **Industry = all inference-only**: TetraMem **MX100** (Nature Electronics 2025; 10 cores, 248×256 1T1R RRAM+RISC-V) ships real silicon but **inference-only** (no in-situ update). Mythic/EnCharge class same.
- **DIY in-situ test-chip path**: **SkyWater S130 + Weebit Nano 256Kb ReRAM IP** (JEDEC/AEC-Q100 qualified 2023, open SKY130 PDK) = foundry RRAM access for an MPW — a lab can fab its OWN trainable RRAM array. [med]
- **EP-on-hardware = still only SPICE sim**: "Memristor Crossbar Circuits Implementing Equilibrium Propagation" (Oh et al., Kookmin U) is circuit simulation, NOT silicon. ⇒ **no fabricated EP-transformer hardware exists anywhere — the demo is genuinely novel.** [high]

**BOTTOM LINE FOR THE PITCH:** lead with **Shanbhag** (his JSSC-2018 chip already proves analog-MVM + on-chip-training in one die — the nearest substrate; we add relaxation + EP) + **Wenjuan Zhu** (trainable device) + **Hanumolu** (converter glue) = a complete in-house UIUC-ECE stack. Stanford **Wong** as the RRAM-device escalation (via the student). Industry (TetraMem/Mythic) only useful for the fixed-weight Phase-2 forward path. Nobody has built EP-transformer hardware → first-mover.

### Hanumolu profile (targeted, 2026-06-21)
**Pavan Kumar Hanumolu** — Seendripu Family Professor, UIUC ECE (since 2013; prior Oregon State); member of CSL's Integrated Circuits & Systems Group. "Top-five mixed-signal IC researchers worldwide," NSF CAREER 2010, heavy JSSC/ISSCC record. Work: **energy-efficient analog/mixed-signal — time-based ADCs, continuous-time filters, ultralow-jitter clocking/PLLs, high-speed serial links, switched-cap, DC-DC power conversion.** ⇒ exactly the **converter + analog-integrator + feedback-loop** layer the EP control loop needs (ADC/DAC glue to read settled states + apply the nudge; switched-cap integrators ARE relaxation-dynamics primitives). Note: his published silicon is converters/links/clocking, NOT CIM — he's the glue/control-loop partner, not the MVM substrate. (Also co-founded Omni Design Technologies — converter IP.) [sources: ece.illinois.edu/.../hanumolu, icsg.csl.illinois.edu]

### GAP 3 — endurance budget: the make-or-break number, and it CLEARS the bar
The feasibility question: an EP run does ~tens-of-thousands of update STEPS; with digital-accumulate-then-threshold-program, physical device writes are FEWER than steps. How many cycles do devices survive?
- **HfOx RRAM: up to ~10^10 cycles** endurance (best-in-class metal-oxide). [arxiv 1909.01771, IOP 10.1088/1361-6641/abf29d]
- AlOx / weaker oxides: only ~10^4 — material choice matters a lot.
- **Budget check**: MNIST-class training writes ~10^4 cycles; gradient training can scale to **~10^8** cycles. ⇒ **HfOx (10^10) has ~100× headroom even over a 10^8-write run** — endurance is NOT a blocker IF you use HfOx-class RRAM + the threshold-accumulate scheme (which cuts writes below step-count). [web-search snippets, med-high]
- **Device-nudge insight**: an EP/Coupled-Learning *nudge* changes resistance far less than a full state write, so per-nudge endurance is plausibly >> rated full-write endurance (needs empirical confirmation, but favorable). 
- **ECRAM (electrochemical RAM)** = the symmetric/linear-analog-update + high-endurance technology specifically aimed at in-situ training: "open-loop analog programmable electrochemical memory array" (Nature Comms 2023, s41467-023-41958-4) — but **research-only** (not commercially available; lab/foundry fab). It's the device-physics frontier Wenjuan Zhu / Wong-type collaborators work in.
- **VERDICT: endurance is survivable** with HfOx-class RRAM (10^10) + threshold-program; ECRAM is the better-but-research-only upgrade. The make-or-break risk is NOT endurance — it's **update linearity/symmetry + device variation** (the asymmetric-nonlinear-update problem), which the digital-accumulate scheme + compensation (stochastic rounding) mitigates. [from earlier run + this]

---

## 2026-07-07 addendum — the J^T wall is now a THEOREM (arXiv:2603.26969), and the price list

**Stern, Frim, Candás, Liu, Balasubramanian, "Contrastive learning in tunable dynamical systems"
(UPenn physical-learning group — the same lineage as our Demo-M blueprint):** for dynamics that break
time-reversal symmetry (= non-reciprocal = our non-conservative operators), *exact* gradient descent
via local rules + a causal, scalable supervisor is impossible **in principle** — the gradient needs the
adjoint (backward-in-time / transposed) propagator, which the forward physics of a non-reciprocal
system cannot supply. Their constructive answer: **PAR (Probably Approximately Right)** — a causal
"forward supervisor" whose update kernel is NOT the gradient but is often positively correlated with
it; learning succeeds iff the average projection is positive.

**What this hardens:** our "Wall 3" (exact AsymEP correction needs J^T v) is physics, not engineering
— *for single-ended, forward-dynamics-only hardware*. Our plain-nudge measurement (cos 0.30 at C512)
is their impossibility measured; our corrected estimator's cos 0.88–0.94 is their PAR correlation
parameter at LM scale.

**The price list (escape routes, ranked):**
1. **Bidirectional crossbars** — the transpose of every LINEAR element is physically free by
   reverse-driving the same array (row-drive vs column-drive; classic analog-BP practice); diagonal
   nonlinear gains are self-transpose (sample-and-hold the operating point). Cost: ~2x terminal
   circuitry (drivers/ADCs both sides). This is the Phase-3 chip spec item (Shanbhag conversation).
2. **Conservative corner (CET energy attention)** — J = J^T by construction: the forward system IS its
   own adjoint; plain EP exact; the theorem's exemption zone. The expressivity tax (no causal masking,
   per our Onsager argument) is now weighed against *exact physical trainability* — CET's stock rises.
3. **PAR acceptance** — run plain contrastive, eat the bias. Price being measured NOW at C512
   (runs/par_baseline: warm s2000 + plain nudge + stability guard; the corrected twin reached 1.75).
4. **Mixed-signal (Demo S, current plan)** — correction computed digitally; the theorem says this
   digital slice cannot be eliminated for free, only bought via (1) or avoided via (2)/(3).

**Demo-M reframe (upgrade):** the OTA board's local contrastive rule on non-reciprocal coupling *is* a
PAR process — position the demo as "PAR learning + our spectral/residual stability control on
non-reciprocal analog hardware": complementary to (and citing) their paper, covering exactly the half
they don't (forward stability), on their home turf.

---

## 2026-07-10 — ANALOG SOFTMAX / ATTENTION DOSSIER (deep-research, 12 claims 3-0 verified; synthesis truncated by API limit)

**Verdict: the whole cascade-analog hypothesis chain is CONFIRMED, and the training gap is REAL.**
Every non-weight op of attention has a demonstrated analog primitive; nobody has trained attention in analog.

### (a) softmax = subthreshold-exp + common-node KCL normalizer — CONFIRMED, and reciprocal
- **Elfadel & Wyatt, NIPS 1993** (`proceedings.neurips.cc/paper/1993/hash/352407221afb776e3143e8a1a0577885`):
  explicit subthreshold-CMOS softmax, i_m = I_c·exp(κv_m/V_0) / Σ exp(κv_p/V_0). exp from subthreshold
  MOS drain current; normalization from KCL at a shared node fed by control current I_c. **Exactly our
  hypothesis.** Temperature/process enter via V_0=kT/q and κ (calibration target).
- **★ Reciprocity (KEY for EP):** their Theorem 1 — the *diode-connected* variant ("entropic resistor")
  is **reciprocal, passive, locally passive**, with log-sum-exp co-content Φ(v)=(1/κ)I_c V_0 ln Σexp(κv_m/V_0).
  The naive gate-in/drain-out softmax is NOT reciprocal and can't compose into energy-based networks.
  **⟹ there exists a softmax circuit that composes into reciprocal EP optimization networks.** This is
  the physical realization of "softmax Jacobian diag(p)−ppᵀ is symmetric ⟹ VJP needs no adjoint" — our
  PAR-wall escape for attention is not just algebraic, it has a named 1993 circuit.
- **Softmax Jacobian symmetry** confirmed from the same paper (DF = diag(F)−FFᵀ, symmetric, singular,
  eigenvalues < 1, softmax = ∇ of a convex potential).
- **Gilbert 1984** 16-channel analog array normalizer (IEEE JSSC SC-19(6):956): divisive normalization
  in silicon since 1984.
- **Lazzaro WTA 1988** (the reference I'd assumed) is actually a HARD max (argmax + log(max)), NOT
  softmax — cite Elfadel&Wyatt for soft, Lazzaro only for the common-node inhibition mechanism.
- **Modern (2023, arXiv:2305.13649):** differential-pair softmax, tail-current-normalized. Measured
  ±0.9% error (subthreshold NMOS, SPICE, N=4); breadboard ±4.2%. Only N=4 ever built — **large N is
  open**, our first tolerance question.
- **★ Tolerance transfer coefficient (plug into E-tier sim):** ±1% device (group-constant Δc) mismatch
  → **≤±1% softmax output error, LINEAR.** (arXiv:2305.13649.) Pelgrom σ(ΔVT)=A_VT/√(WL) is the
  underlying law (Kinget JSSC 2005).

### (b) act×act (QKᵀ, P·V) with transient operands = charge-domain gain cells — CONFIRMED (built, sim+device)
- **Leroux et al., Nature Comp. Sci. 2025** (`nature.com/articles/s43588-025-00854-1`, arXiv:2409.19315):
  gain-cell (capacitor/OSFET, volatile charge) crossbars **simultaneously store the KV cache and compute
  QKᵀ and score·V in analog.** K,V as capacitor voltages in 64×64 arrays; Q & scores as PWM; products
  summed as bitline currents. Precision: 4-bit Q, 3-bit stored K/V, 5-bit out. **This is our hypothesis
  (b) already built.** BUT they **replaced softmax with HardSigmoid** (charge-to-pulse), citing
  sigmoid≈softmax attention up to ~7B — i.e. they sidestepped analog softmax, didn't solve it.
- Memristor route (Bettayeb, Sci.Rep. 2024) hits **write-endurance** limits writing transient operands
  into RRAM → confirms crossbars are wrong for act×act; **charge/capacitor is the right primitive** (KV
  is volatile anyway).

### (c) LayerNorm / divisive normalization = Gilbert normalizer / RMS circuits — prior art exists (as above).

### THE GAP — CONFIRMED unclaimed
Every analog/photonic attention accelerator surveyed **offloads softmax to digital or replaces it**, and
**all are inference-only** (weights trained offline in software, mapped to conductances):
Leroux 2025 (HardSigmoid + software adapt), 65nm hybrid chip arXiv:2409.04940 (softmax+PV digital),
Bettayeb 2024 (exp/log via digital LUT), photonic Lightening-Transformer HPCA2024 & arXiv:2603.12934
(GEMMs optical, softmax electronic). **No one has TRAINED softmax attention in analog with any learning
rule.** Pre-transformer analog neuro had no softmax demand; post-transformer analog work is inference-only.
⟹ "standard-form transformer, softmax intact, trained on analog by a local two-phase rule" is open territory.

### EP-in-crossbars precedent (RQ6): Kendall/Scellier/Bengio 2020 (arXiv:2006.01981) — nonlinear
resistive networks (memristor + diodes) are EBMs by Kirchhoff, trainable by EP with local rules; reverse
crossbar drive = Wᵀ. (Kendall → Rain AI.) Our reciprocal "entropic resistor" softmax slots into exactly
this framework.

**Open for the deferred optimizer-state dossier (wceo0z7fy, hit API limit — rerun later):** analog
momentum (leaky integrator / Loihi eligibility trace), Adam-as-AGC (translinear squarer + RC + divider),
IBM Tiki-Taka aux-array momentum, and the Muon-spectral gap. Hypothesis: Adafactor (row/col shared 2nd
moment = per-line AGC) is the analog-native adaptive optimizer.

---

## 2026-07-11 — OPTIMIZER BoM AUDIT: what each cascade block needs in analog, Adam vs Muon (vs Lion baseline)

Per-block inventory for one standard transformer block, cascade-EP training. Weight count per block
= 12C² (QKV 3C², out C², FFN 8C²). Numbers at C=512 (D1 scale, 3.1M weights/block) and C=1536
(sub-B scale, 28M weights/block). [dossier] = literature-anchored; [est] = engineering estimate.

### Common substrate (optimizer-independent)
| item | count per block | primitive |
|---|---|---|
| weight crossbars | 12C² cells | memristor/PCM, reciprocal → transpose free [dossier: Kendall 2020] |
| KV charge arrays | 2·T·C gain cells (volatile) | Leroux 2025 gain-cell tiles [dossier] |
| softmax normalizers | H per-row common-node circuits, T branches each (time-muxed) | Elfadel-Wyatt entropic resistor [dossier] |
| LayerNorm | 2 divisive-norm/RMS circuits, C channels | Gilbert 1984 normalizer family [dossier] |
| bidirectional periphery | 2× crossbar drivers/sense (reverse drive for Jᵀ) | [dossier: analog-BP/EP practice] |
| outer-product write | pulse-coincidence programming per crossbar | standard analog-training periphery |

### ADAM per block
| ingredient | analog form | added components | verdict |
|---|---|---|---|
| momentum m | per-synapse leaky integrator (cap+leak) OR Tiki-Taka fast aux array | **+12C² cells (+100% analog memory)** — 3.1M @C512, 28M @C1536 | feasible; aux-array form is proven practice [Tiki-Taka; pending re-research] |
| second moment v | per-synapse squarer = economic suicide; **factored (Adafactor-style) row/col statistics** | ~16C AGC channels (translinear squarer + RC + divider, ~10–30 transistors + 1 cap each) ≈ 8k channels @C512 | feasible [est]; "Adafactor is the analog-native Adam" |
| m/√v division | per-LINE programmable driver gain (factored) | C-channel gain DACs per matrix | feasible [est] |
| bias correction / schedules | digital global scalars | negligible | — |
| **total Adam tax** | | **~2× analog memory + ~16C AGC + per-line gain** ; ZERO gradient digitization | **the analog-compatible adaptive optimizer** |
| precision risk | AGC mismatch = per-line lr noise (linear transfer [dossier: ±1%→±1%]) | needs E-tier arm | |

### MUON per block
| ingredient | analog form | added components | verdict |
|---|---|---|---|
| momentum | same as Adam | +12C² cells | feasible |
| Newton-Schulz (X↦aX+b(XXᵀ)X+…, 5 iters) | matrix-MATRIX on a transient matrix = ~10–15·C vector-passes per matrix per step; needs a second C×C act×act fabric per matrix or steals the KV arrays | ~1000× the training step's own MVM duty [est, matches GPU-side analysis] | **analog: economically dead** |
| mixed-signal Muon | digitize FULL gradient/momentum matrix every step: 12C² conversions @ 8–16 bit per block per step (3.1M @C512, 37M/step for L12) + digital NS matmul sidecar + full-precision DAC write-back | ADC energy dominates; this is the exact cost Lion's 1-bit updates avoid | **negates the analog-training story; Muon stays a GPU/digital-era tool** |

### LION baseline (for contrast)
momentum array (+12C²) + 1-bit sign comparators per line + fixed-amplitude pulse writes.
**Minimal periphery, zero ADC traffic, native to pulse programming.** Known quality tax vs AdamW
~15–20% PPL-relative [OmniOpt @1B]; MARS-Lion recovers part.

### Cost ladder (per block, relative)
```
SGD/Lion:   1.0× memory-cells, ~0 ADC/step, trivial periphery      ← analog-native
Adam(fact): 2.0× memory-cells, ~0 ADC/step, +16C AGC channels      ← analog-compatible
Muon:       2.0× memory-cells, +12C² ADC conversions/step + digital NS sidecar ← mixed-signal heavy
```
**CORRECTION (2026-07-11, project empirical record): Lion/SGD are RULED OUT for EP training** —
tested and failed (looped line: abl_lion2 diverged at step 600, val 20.1 / res 0.34; earlier tests
concur). Per-parameter adaptivity (Adam-class) is empirically LOAD-BEARING for EP gradients here.
⟹ Revised chip narrative: **factored-Adam (AGC/Tiki-Taka route) is not the premium tier — it is
the MANDATORY baseline**. The +2× analog memory and ~16C AGC channels per block are the price of
admission for analog EP training, and the OmniOpt "Lion is cheap"路线 does not apply to us.
Muon stays priced out of FULLY-analog implementations (GPU/simulation-era tool).

### CORRECTED ECONOMICS (2026-07-11 #2) — priced in the PROJECT DOCTRINE currency (reuse, NOT fab)

**Retraction:** any tapeout/"$M program" framing in this session is VOID. The standing build
doctrine (this file, 2026-06-21 BOTTOM LINE + UIUC map) is: **COTS-stitch** (memristor CIM eval
module + FPGA carrying softmax/LN/optimizer/EP control) → Phase-2 **Mythic/HERMES-class COTS** for
bulk FIXED MVM → the un-buyable in-situ substrate via the **UIUC collaboration** (Zhu device fab +
Shanbhag DIMA-line CIM + Hanumolu converters), grant-funded. No startup silicon, ever.

**The doctrine already settles the optimizer hardware question:** the verified update path is
**digital-accumulate → threshold-program** (bulk-switching template, 3-0) — every gradient
increment passes a DIGITAL accumulator by design (that IS the endurance fix). Hence optimizer
state lives digital-side (FPGA BRAM/DDR), not in analog:
- **Adam ≈ free**: m,v = 2 extra digital words/weight in memory the doctrine already requires.
  Zero new analog components. (The per-synapse-cap / AGC-channel analysis above = appendix
  curiosity for a fully-analog optimizer nobody asked for.)
- **Muon: not hardware-blocked at demo scale**: NS runs FPGA/host-side on the digitized gradient
  stream (~24 GFLOP/step at C512×12 — trivial at Hz step rates). Its wall re-emerges only in
  fully-analog ambitions. ⟹ **Adam-vs-Muon remains an ALGORITHM question (D1a's EP-Muon arm is
  the referee), not a hardware question.** Lion/SGD stay algorithmically banned.

**Demo-1 cost in doctrine currency:** CIM eval module + FPGA dev board + converter glue ≈
**$5–20k board-level BOM + engineering time**; beyond-eval-kit in-situ arrays come from the UIUC
device/circuit line (their fab, grant money). GAP-2 procurement pass (TetraMem / Anadigm FPAA /
Aspinity SKU + prices) remains the open to-do it always was.

### CASCADE RE-PRICING under the doctrine (2026-07-11, user point: "block叠加只会更便宜")

The 2026-06-21 plan was priced for the LOOPED block — the hardest possible customer (in-situ
recurrent fabric, 150–300-step settling duty, on-chip spectral leash instrumentation). Cascade
strictly reduces every hardware demand:

1. **Free phase = plain feedforward** ⟹ the bulk of compute is exactly what LARGE-but-FIXED COTS
   (Mythic-class) sells. What the old plan could only use "cleverly" in Phase 2 is now natively fit.
2. **One physical block, L logical layers (time-multiplex).** Cascade blocks are structurally
   identical — ONE trainable CIM module + weight banks emulates the whole stack sequentially.
   Trainable-fabric area NO LONGER SCALES WITH DEPTH: L6→L12→L24 costs DDR/SRAM for weight sets
   (negligible) + wall-clock, not more analog hardware. "叠 block" is a memory line-item.
3. **Nudged phase = 3 fb rounds, not 150–300 settling steps** ⟹ analog duty cycle drops ~50×;
   retention/drift budgets relax accordingly.
4. **Leash/governor instrumentation DELETED** — free phase is unconditionally exact; the on-chip
   spectral-monitoring layer the looped plan required simply does not exist here.
5. **★ SRAM-CIM suffices for Demo-1.** Time-muxing means weights are (re)loaded per layer —
   endurance-free, fast-write SRAM-CIM (Shanbhag DIMA line: analog MVM + demonstrated on-chip
   write-back, JSSC 2018) handles it outright. **The NVM/memristor device fab (Zhu) becomes
   OPTIONAL for the demo** (kept for the non-volatile scale story) — the hardest UIUC dependency
   drops out of the critical path. Demo-1 = Shanbhag-line module (or equivalent SRAM-CIM eval
   fabric) + FPGA + converters, same **$5–20k board-level BOM**, now covering an L-layer
   tinystories-class trainer instead of one looped block.

Net: cascade turned depth from the cost driver into a memory line-item, and reduced the demo's
critical-path collaboration from three groups to one.

## OLMo2-standard block: analog audit (2026-07-10)
Verdict: **net MORE analog-friendly than the 2019 GPT block** — OLMo2's digital-stability changes
align 1:1 with analog dynamic-range needs. Per component:
- **RMSNorm (vs LN): net win.** Pure divisive normalization (square-law + KCL sum + divider), NO
  mean-subtraction path; reuses the softmax KCL-normalization periphery class. Jacobian
  (g/r)(I - xx^T/r^2) is SYMMETRIC -> no new PAR/adjoint burden.
- **Norm-after-sublayer (reordered): net win.** Every injection into the residual current-summing bus
  is bounded at the branch exit — exactly what a physical summing line wants (pre-norm leaves branch
  outputs unbounded).
- **Full-width QK-norm: win.** One square-sum bus per q/k (cheaper periphery than per-head), bounds
  the analog-softmax exp device input (Elfadel-Wyatt range), symmetric Jacobian.
- **RoPE (vs learned pos-emb): the biggest win.** Fixed per-position 2x2 rotations = I/Q quadrature
  mixing — a century-old mixed-signal standard block (cos/sin DDS + multiplying DACs). REMOVES a
  trainable TxC analog memory (write cycles + drift surface). Feedback path J^T = rotation by -theta
  = same mixer reused. Orthogonal Jacobian (condition 1).
- **SwiGLU: neutral.** Gate = elementwise Gilbert/translinear multiply (textbook cell, reuse-doctrine
  compliant); GELU was already x*Phi(x) = same squasher+multiplier class. 3 narrower MVMs, crossbar
  area param-matched.
- **No biases: win.** Every bias was a programmable, trainable, drifting current source/DAC. All gone.
- **Untied head + final RMSNorm: neutral/win.** LM head + CE live digital-side anyway (no 50k-vocab
  analog softmax); final RMSNorm bounds the signal crossing the ADC boundary.
- **Grouped wd / AdamW: neutral.** Optimizer state digital-side per standing doctrine.
- **z-loss: win, free.** Zero analog hardware (only changes the top nudge signal); its FUNCTION —
  keeping logits bounded — trains the network into analog range compliance.
**EP two-phase check: NO new non-reciprocal element.** All four new pieces have symmetric-or-orthogonal
Jacobians whose transposes reuse the same hardware (RMS/QK-norm self-transpose; RoPE = negative-angle
same-mixer; gate-multiply transpose = multiply by the partner signal). Non-reciprocity remains confined
to attention QKV coupling + crossbar transpose reads. PAR wall unchanged.
**Honest new E-tier tolerance line-items:** (1) ~2x normalizer count (4/block + final): divider
mismatch/offset sensitivity; (2) SwiGLU's 1408-wide Gilbert gate: per-multiply mismatch/noise;
(3) RoPE phase error (expect very tolerant — phase error ~ position blur).
Narrative gift: "the architecture the digital world adopted for training stability is the architecture
analog needs for dynamic range."

## 2026-08-05 — 优化器审计修订: OLion 改变了 Muon-模拟判决的前提; Hyperball = 最便宜的自适应
**OLion (arXiv:2602.01105, PKU+MSRA, Wang/Shen/Zhang)**: Lion 动量 → Newton-Schulz 正交化 →
**逐元素取号** → RMS 定标(γ=0.2√(d1d2)/‖S‖_F) → 内建解耦 weight decay。GPT-2 124/355/770M +
Llama-2-7B 全线快于 AdamW/Lion/Muon/AdaMuon, LR 3e-4..5e-3 全区间稳。单篇预印本, 待复现。
**对模拟实现的关键改变**: 7-11 审计判 Muon 死刑的前提是 NS 输出须以模拟精度写入; OLion 的最终
更新是 sign(正交化(M)) —— **正交化器的输出精度要求塌缩到 1 bit**(误差只在过零点附近才算数)。
模拟计算弱在精度、强在廉价的近似动力学 ⟹ "粗糙的模拟正交化器 + 比较器 + 定幅脉冲写"成为可行
栈, 后两级正是 Lion 行已定价 analog-native 的单元。附带: ±1/√d 平坦更新分布对受限电导范围友好
(论文自己为数字低精度论证过同一点); 内建 wd 恰好补上我方 Muon 无衰减的洞(RESULT: 08-05 发现)。
**Muon"神奇算法"三条未探路线(用户 08-05 问题触发)**:
(a) **正交化作为平衡态**: Ẋ=X(I−XᵀX) 类流以正交因子为吸引子 — 物理弛豫, 与 EP 同范式; 审计的
1000× 是"每步从头 NS"假设。
(b) **跟踪摊销**: 动量 EMA(0.95) 时间尺度 ~20 步, 极因子漂移缓慢 ⟹ 常驻旁路只需每步纠 O(lr)
漂移, 成本从 10-15·C 次向量通过塌到 ~C 级。
(c) **局部规则白化**: 右因子 (MᵀM)^{-1/2} 即白化变换, Pehlevan–Chklovskii 相似性匹配线用**局部
反 Hebbian 规则**在线计算白化/PCA — 正是物理学习圈会造的电路类。polar(M)=M·(MᵀM)^{-1/2}。
组合栈: 动量电容(漏积分) → 模拟白化旁路(局部规则, 跟踪) → 比较器(取号) → 定幅脉冲写。
每级都是已定价或经典的模拟原语。判决升级: "Muon 经济死刑"→"OLion 类谱优化器存在可信模拟路径
(纸面设计, 白化旁路 C×C fabric/矩阵 + 自身失配未定价, 比较器过零精度由 imperfection 表回答)"。
**Hyperball (arXiv:2606.16899, Wen/Dang/Lyu/Ma/Liang)**: wrapper, 把权重矩阵与更新的 Frobenius
范数钉常数(权重活在球面上, 只转方向, 显式角速度取代 weight decay)。Muon+Hyperball 在 ≤1.2B
Qwen3 类模型上 **20-30% token 等效加速**, 且**跨宽深的 LR 迁移更好**(直接利好我方迁移条令)。
反方 (2607.22444): 恒定角速度不豁免调度。模拟角度: 每矩阵一个标量范数约束 = 经典 AGC(一个功率
传感器+一个增益旋钮/矩阵), **最便宜的自适应形态**; 权重定范数与受限电导范围天然契合。
**价目表实验扩臂**: EP × {SGDM, Lion, AdamW, Adafactor, OLion, Muon+Hyperball} vs Muon 基线。