From d227a9ef87ab75e839ebc773a680c52196006f7c Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Thu, 9 Jul 2026 23:03:55 -0500 Subject: =?UTF-8?q?HW:=20optimizer=20BoM=20audit=20per=20cascade=20block?= =?UTF-8?q?=20=E2=80=94=20Adam=20analog-compatible=20via=20factored-AGC=20?= =?UTF-8?q?(+2x=20memory,=20zero=20ADC),=20Muon=20priced=20out=20of=20anal?= =?UTF-8?q?og=20(matrix-matrix=20NS=20+=20full-gradient=20ADC=20tax),=20Li?= =?UTF-8?q?on=20native=20baseline?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/hardware/HW_RESEARCH_FINDINGS.md | 51 +++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) (limited to 'docs/hardware/HW_RESEARCH_FINDINGS.md') diff --git a/docs/hardware/HW_RESEARCH_FINDINGS.md b/docs/hardware/HW_RESEARCH_FINDINGS.md index 1e040c9..d380b39 100644 --- a/docs/hardware/HW_RESEARCH_FINDINGS.md +++ b/docs/hardware/HW_RESEARCH_FINDINGS.md @@ -195,3 +195,54 @@ this framework. momentum (leaky integrator / Loihi eligibility trace), Adam-as-AGC (translinear squarer + RC + divider), IBM Tiki-Taka aux-array momentum, and the Muon-spectral gap. Hypothesis: Adafactor (row/col shared 2nd moment = per-line AGC) is the analog-native adaptive optimizer. + +--- + +## 2026-07-11 — OPTIMIZER BoM AUDIT: what each cascade block needs in analog, Adam vs Muon (vs Lion baseline) + +Per-block inventory for one standard transformer block, cascade-EP training. Weight count per block += 12C² (QKV 3C², out C², FFN 8C²). Numbers at C=512 (D1 scale, 3.1M weights/block) and C=1536 +(sub-B scale, 28M weights/block). [dossier] = literature-anchored; [est] = engineering estimate. + +### Common substrate (optimizer-independent) +| item | count per block | primitive | +|---|---|---| +| weight crossbars | 12C² cells | memristor/PCM, reciprocal → transpose free [dossier: Kendall 2020] | +| KV charge arrays | 2·T·C gain cells (volatile) | Leroux 2025 gain-cell tiles [dossier] | +| softmax normalizers | H per-row common-node circuits, T branches each (time-muxed) | Elfadel-Wyatt entropic resistor [dossier] | +| LayerNorm | 2 divisive-norm/RMS circuits, C channels | Gilbert 1984 normalizer family [dossier] | +| bidirectional periphery | 2× crossbar drivers/sense (reverse drive for Jᵀ) | [dossier: analog-BP/EP practice] | +| outer-product write | pulse-coincidence programming per crossbar | standard analog-training periphery | + +### ADAM per block +| ingredient | analog form | added components | verdict | +|---|---|---|---| +| momentum m | per-synapse leaky integrator (cap+leak) OR Tiki-Taka fast aux array | **+12C² cells (+100% analog memory)** — 3.1M @C512, 28M @C1536 | feasible; aux-array form is proven practice [Tiki-Taka; pending re-research] | +| second moment v | per-synapse squarer = economic suicide; **factored (Adafactor-style) row/col statistics** | ~16C AGC channels (translinear squarer + RC + divider, ~10–30 transistors + 1 cap each) ≈ 8k channels @C512 | feasible [est]; "Adafactor is the analog-native Adam" | +| m/√v division | per-LINE programmable driver gain (factored) | C-channel gain DACs per matrix | feasible [est] | +| bias correction / schedules | digital global scalars | negligible | — | +| **total Adam tax** | | **~2× analog memory + ~16C AGC + per-line gain** ; ZERO gradient digitization | **the analog-compatible adaptive optimizer** | +| precision risk | AGC mismatch = per-line lr noise (linear transfer [dossier: ±1%→±1%]) | needs E-tier arm | | + +### MUON per block +| ingredient | analog form | added components | verdict | +|---|---|---|---| +| momentum | same as Adam | +12C² cells | feasible | +| Newton-Schulz (X↦aX+b(XXᵀ)X+…, 5 iters) | matrix-MATRIX on a transient matrix = ~10–15·C vector-passes per matrix per step; needs a second C×C act×act fabric per matrix or steals the KV arrays | ~1000× the training step's own MVM duty [est, matches GPU-side analysis] | **analog: economically dead** | +| mixed-signal Muon | digitize FULL gradient/momentum matrix every step: 12C² conversions @ 8–16 bit per block per step (3.1M @C512, 37M/step for L12) + digital NS matmul sidecar + full-precision DAC write-back | ADC energy dominates; this is the exact cost Lion's 1-bit updates avoid | **negates the analog-training story; Muon stays a GPU/digital-era tool** | + +### LION baseline (for contrast) +momentum array (+12C²) + 1-bit sign comparators per line + fixed-amplitude pulse writes. +**Minimal periphery, zero ADC traffic, native to pulse programming.** Known quality tax vs AdamW +~15–20% PPL-relative [OmniOpt @1B]; MARS-Lion recovers part. + +### Cost ladder (per block, relative) +``` +SGD/Lion: 1.0× memory-cells, ~0 ADC/step, trivial periphery ← analog-native +Adam(fact): 2.0× memory-cells, ~0 ADC/step, +16C AGC channels ← analog-compatible +Muon: 2.0× memory-cells, +12C² ADC conversions/step + digital NS sidecar ← mixed-signal heavy +``` +Recommendation for the chip narrative: **Lion (or SGDM) as the on-chip rule; factored-Adam as the +premium tier; Muon explicitly priced out** (its value is in the GPU/simulation era, where it is +nearly free). Pending verification: Tiki-Taka transfer constants, AGC channel specs, Loihi-trace +leakage numbers — the deferred optimizer-state deep-research (rerun when API budget recovers). -- cgit v1.2.3