From 761ac0bae654333490df7de4c6dc50e24bd863f6 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Thu, 9 Jul 2026 12:54:08 -0500 Subject: =?UTF-8?q?HW=20dossier:=20analog=20softmax/attention=20(12=20veri?= =?UTF-8?q?fied=20claims)=20=E2=80=94=20Elfadel-Wyatt=201993=20reciprocal?= =?UTF-8?q?=20'entropic=20resistor'=20softmax=20(EP-composable,=20PAR=20es?= =?UTF-8?q?cape=20realized),=20charge-domain=20KV=20gain-cells=20(Leroux?= =?UTF-8?q?=202025),=20the=20analog-training=20gap=20confirmed=20unclaimed?= =?UTF-8?q?;=20+-1%=20mismatch->+-1%=20softmax=20linear=20tolerance=20coef?= =?UTF-8?q?f?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/hardware/HW_RESEARCH_FINDINGS.md | 63 +++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) (limited to 'docs') diff --git a/docs/hardware/HW_RESEARCH_FINDINGS.md b/docs/hardware/HW_RESEARCH_FINDINGS.md index c63106a..1e040c9 100644 --- a/docs/hardware/HW_RESEARCH_FINDINGS.md +++ b/docs/hardware/HW_RESEARCH_FINDINGS.md @@ -132,3 +132,66 @@ parameter at LM scale. PAR process — position the demo as "PAR learning + our spectral/residual stability control on non-reciprocal analog hardware": complementary to (and citing) their paper, covering exactly the half they don't (forward stability), on their home turf. + +--- + +## 2026-07-10 — ANALOG SOFTMAX / ATTENTION DOSSIER (deep-research, 12 claims 3-0 verified; synthesis truncated by API limit) + +**Verdict: the whole cascade-analog hypothesis chain is CONFIRMED, and the training gap is REAL.** +Every non-weight op of attention has a demonstrated analog primitive; nobody has trained attention in analog. + +### (a) softmax = subthreshold-exp + common-node KCL normalizer — CONFIRMED, and reciprocal +- **Elfadel & Wyatt, NIPS 1993** (`proceedings.neurips.cc/paper/1993/hash/352407221afb776e3143e8a1a0577885`): + explicit subthreshold-CMOS softmax, i_m = I_c·exp(κv_m/V_0) / Σ exp(κv_p/V_0). exp from subthreshold + MOS drain current; normalization from KCL at a shared node fed by control current I_c. **Exactly our + hypothesis.** Temperature/process enter via V_0=kT/q and κ (calibration target). +- **★ Reciprocity (KEY for EP):** their Theorem 1 — the *diode-connected* variant ("entropic resistor") + is **reciprocal, passive, locally passive**, with log-sum-exp co-content Φ(v)=(1/κ)I_c V_0 ln Σexp(κv_m/V_0). + The naive gate-in/drain-out softmax is NOT reciprocal and can't compose into energy-based networks. + **⟹ there exists a softmax circuit that composes into reciprocal EP optimization networks.** This is + the physical realization of "softmax Jacobian diag(p)−ppᵀ is symmetric ⟹ VJP needs no adjoint" — our + PAR-wall escape for attention is not just algebraic, it has a named 1993 circuit. +- **Softmax Jacobian symmetry** confirmed from the same paper (DF = diag(F)−FFᵀ, symmetric, singular, + eigenvalues < 1, softmax = ∇ of a convex potential). +- **Gilbert 1984** 16-channel analog array normalizer (IEEE JSSC SC-19(6):956): divisive normalization + in silicon since 1984. +- **Lazzaro WTA 1988** (the reference I'd assumed) is actually a HARD max (argmax + log(max)), NOT + softmax — cite Elfadel&Wyatt for soft, Lazzaro only for the common-node inhibition mechanism. +- **Modern (2023, arXiv:2305.13649):** differential-pair softmax, tail-current-normalized. Measured + ±0.9% error (subthreshold NMOS, SPICE, N=4); breadboard ±4.2%. Only N=4 ever built — **large N is + open**, our first tolerance question. +- **★ Tolerance transfer coefficient (plug into E-tier sim):** ±1% device (group-constant Δc) mismatch + → **≤±1% softmax output error, LINEAR.** (arXiv:2305.13649.) Pelgrom σ(ΔVT)=A_VT/√(WL) is the + underlying law (Kinget JSSC 2005). + +### (b) act×act (QKᵀ, P·V) with transient operands = charge-domain gain cells — CONFIRMED (built, sim+device) +- **Leroux et al., Nature Comp. Sci. 2025** (`nature.com/articles/s43588-025-00854-1`, arXiv:2409.19315): + gain-cell (capacitor/OSFET, volatile charge) crossbars **simultaneously store the KV cache and compute + QKᵀ and score·V in analog.** K,V as capacitor voltages in 64×64 arrays; Q & scores as PWM; products + summed as bitline currents. Precision: 4-bit Q, 3-bit stored K/V, 5-bit out. **This is our hypothesis + (b) already built.** BUT they **replaced softmax with HardSigmoid** (charge-to-pulse), citing + sigmoid≈softmax attention up to ~7B — i.e. they sidestepped analog softmax, didn't solve it. +- Memristor route (Bettayeb, Sci.Rep. 2024) hits **write-endurance** limits writing transient operands + into RRAM → confirms crossbars are wrong for act×act; **charge/capacitor is the right primitive** (KV + is volatile anyway). + +### (c) LayerNorm / divisive normalization = Gilbert normalizer / RMS circuits — prior art exists (as above). + +### THE GAP — CONFIRMED unclaimed +Every analog/photonic attention accelerator surveyed **offloads softmax to digital or replaces it**, and +**all are inference-only** (weights trained offline in software, mapped to conductances): +Leroux 2025 (HardSigmoid + software adapt), 65nm hybrid chip arXiv:2409.04940 (softmax+PV digital), +Bettayeb 2024 (exp/log via digital LUT), photonic Lightening-Transformer HPCA2024 & arXiv:2603.12934 +(GEMMs optical, softmax electronic). **No one has TRAINED softmax attention in analog with any learning +rule.** Pre-transformer analog neuro had no softmax demand; post-transformer analog work is inference-only. +⟹ "standard-form transformer, softmax intact, trained on analog by a local two-phase rule" is open territory. + +### EP-in-crossbars precedent (RQ6): Kendall/Scellier/Bengio 2020 (arXiv:2006.01981) — nonlinear +resistive networks (memristor + diodes) are EBMs by Kirchhoff, trainable by EP with local rules; reverse +crossbar drive = Wᵀ. (Kendall → Rain AI.) Our reciprocal "entropic resistor" softmax slots into exactly +this framework. + +**Open for the deferred optimizer-state dossier (wceo0z7fy, hit API limit — rerun later):** analog +momentum (leaky integrator / Loihi eligibility trace), Adam-as-AGC (translinear squarer + RC + divider), +IBM Tiki-Taka aux-array momentum, and the Muon-spectral gap. Hypothesis: Adafactor (row/col shared 2nd +moment = per-line AGC) is the analog-native adaptive optimizer. -- cgit v1.2.3