1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
|
# Component → analog-hardware map (cascade-EP, OLMo2-standard block)
**2026-07-11.** The full stack of the current recipe (the stage1b/Stage-2 architecture), component by
component: what it computes, the analog primitive it maps to, the reuse source (doctrine: COTS/eval
modules/FPGA — never tapeout), what the EP feedback path needs from it, and tolerance status.
Companion docs: `HW_RESEARCH_FINDINGS.md` (softmax dossier, OLMo2 analog audit, optimizer BoM),
`CLAIM_LARGEST_NOBP.md` (the headline this hardware underwrites).
## A. Forward path (free phase = the inference pass)
| # | Component | Computes | Analog primitive | Reuse source (no tapeout) | Feedback path (nudged phase) | Tolerance status |
|---|---|---|---|---|---|---|
| 1 | Token embedding | table lookup | **digital** (SRAM lookup + DAC line drivers) | any MCU/FPGA + DAC array | none (input boundary) | n/a |
| 2 | RoPE | fixed per-position 2×2 rotations on q,k | **I/Q quadrature mixer**: cos/sin DDS + multiplying DACs | century-old RF practice; COTS mixer/DDS chips | transpose = rotation by −θ — **same mixer, negated sin** | **E-tier w1: 0.03 rad FREE; 0.1 rad ΔCE +0.012, cos_clean 0.92** — ~2° I/Q phase accuracy suffices (easy) |
| 3 | qkv / attn-proj / SwiGLU w1,w3,w2 / head | matrix–vector products | **crossbar arrays** (the bulk of all compute) | Demo-0/1: **SRAM-CIM (Shanbhag DIMA class)**; alternatives: Mythic flash CIM, IBM HERMES PCM eval | J^T reads = **bidirectional crossbar access** (the known PAR price; standard in every analog-EP proposal) | **E-tier w1 (cascade): wq8 FREE (ΔCE +0.004, cos_clean 0.96); wq6 MARGINAL (+0.051, 0.81); wq4 DEAD (+2.23)** — ≥7b effective weight precision is the binding spec; wave-2 lever = quant-aware co-training |
| 4 | QK-norm (full-width RMS) | q,k ← q/rms(q)·g | **divisive normalization**: square-law devices + KCL current sum + divider/AGC | same primitive class as softmax normalization periphery | Jacobian symmetric — self-transpose, no extra hardware | **E-tier w1: 3% mismatch FREE (ΔCE +0.003, cos_clean 0.95); 10% MARGINAL (+0.042, 0.78)** — 3% device matching is routine |
| 5 | Softmax (causal) | exp + row-normalize | **Elfadel–Wyatt "entropic resistor"**: subthreshold-exp devices + KCL normalization (log-sum-exp co-content) | classic analog-VLSI circuit family; small arrays demonstrated | softmax Jacobian diag(p)−pp^T is symmetric; the QKV coupling around it is the non-reciprocal part | ±1% device mismatch → ±1% softmax error (linear); QK-norm bounds its input range. **E-tier w1: forward additive state noise σ=1e-2 FREE (ΔCE +0.001, cos 0.97) — the looped-era 1e-3 cliff does NOT transfer to cascade** |
| 6 | SwiGLU gate | silu(w1x) ⊙ (w3x) | sigmoid = differential pair (native); ⊙ = **Gilbert/translinear multiplier** per unit | textbook standard cells | transpose of ⊙ = multiply by the partner signal — cheap | **E-tier w1: 10% gain error ΔCE +0.006, cos_clean 0.945 — FREE** at translinear-practice tolerances |
| 7 | RMSNorm ×2 per block (norm-after-sublayer) | bound each sublayer output | divisive normalization (as #4), **no mean-subtraction path** (cheaper than LayerNorm) | same as #4 | symmetric Jacobian | as #4; ~4L+1 normalizers total = the largest NEW periphery count |
| 8 | Residual stream | z + branch outputs | **current-summing bus (KCL node)** | wires | pass-through | norm-after bounds every injection — the arch change is itself the tolerance fix |
| 9 | Final RMSNorm | bound pre-readout state | as #4 | as #4 | symmetric | bounds the **ADC** dynamic range at the digital boundary |
| 10 | LM head + CE + sampling | logits, loss | **digital** (ADC → digital head) | GPU/FPGA at the boundary | the top nudge −β·∇CE is computed digitally and injected as a **force via DACs** | ADC bits vs late-training SNR: couples to the β-floor story (see C.3) |
## B. Training machinery (EP two-phase; nothing here touches inference)
| # | Component | Computes | Analog/digital split | Notes |
|---|---|---|---|---|
| 11 | Free phase | plain forward | pure analog settle (A.1–A.9) | = the inference pass; zero relaxation by construction (cascade) |
| 12 | Nudged relaxation (fb, K=3) | per-block feedback d_l = J_{l+1}^T d_{l+1}; state rebuild | bidirectional crossbar reads (#3) + diagonal activation-derivative gating (local sample-and-multiply) + mixer/norm transposes (#2,#4 — free) | K=3 settle cycles; kretry = "settle longer on flagged batches" = a longer integration window, natively analog |
| 13 | Two-phase θ-readout | ∇θ = per-block energy contrast between phases | **local outer-product reads at each crossbar** (activity × co-activity), differenced across phases | the EP core; single-sided legal (free phase ∂E/∂θ ≡ 0) — one nudged settle per step |
| 14 | β_t + floor-ramp controller | scalar nudge amplitude schedule | trivial digital scalar | the late-SNR fix (dose-response 3e-4→1e-3 measured); on hardware the analog r-indifference bonus applies: big nudges are free against multiplicative noise |
| 15 | Drift/gn guards + kretry | reject/retry non-settled batches | residual monitor = analog comparator on ‖Δz‖; retry logic digital | wall-2 is architecture-cured (zero events over 58.8k steps); guards are belt-and-suspenders |
| 16 | Optimizer (Adam/Muon) | parameter updates | **digital-side co-processor** (standing doctrine): digital accumulate → threshold-program crossbar writes | Muon = GPU-era choice; analog story stays factored-Adam. Write pricing (pulse count/endurance) = **E-tier item** |
| 17 | cos telemetry gate | EP-vs-autograd check | GPU-only diagnostic; `--gate_every 0` for the audited fully-BP-free mode | does not exist on hardware |
## C. The three cross-cutting physics items
1. **Non-reciprocity budget.** Exactly two non-reciprocal elements in the whole stack: the QKV
coupling through attention, and crossbar transpose reads. Everything OLMo2 added (RoPE, RMS/QK
norms, gate) is symmetric-or-orthogonal — transposes reuse the same hardware. PAR wall unchanged.
2. **Dynamic range.** Norm-after bounds every bus injection; QK-norm bounds the softmax input; final
RMSNorm bounds the ADC. The architecture the digital world adopted for training stability is the
architecture analog needs for range compliance.
3. **Estimator SNR = the hardware constraint rehearsed in fp32.** The measured late-training SNR
decline (cos 0.994 at floor 1e-3; dose-response table) is the same physics as device/readout noise
vs nudge amplitude; the β-floor/ramp schedule developed on GPU is the chip's operating manual, and
ADC bits at the readout (#10) buy the same margin as β.
## D. Demo scoping (reuse-doctrine currency)
- **⚡ 2026-07-12: Demo-0 SUPERSEDED as the first rung by CLOCKLESS_ANALOG_MVP_PLAN.md**
(user-authored): a $170–300 clockless twin-network tile (Dillavou-lineage; EP current-nudge vs CL
voltage-clamp; exact vs sign local update; no processor/converter/clock in the learning loop)
validates the physical learning primitive first. The CIM block below becomes the PARTNER rung
(rung 5 of the ladder in COLLABORATOR_BRIEF.md).
- **Demo-0 (now rung 5, partner phase):** single cascade block, SRAM-CIM eval module + FPGA harness +
COTS mixer/DAC/ADC periphery; goal = demonstrate the two-phase θ-read on one block with measured
tolerances. Board-BOM class: $5–20k.
- **Demo-1:** time-multiplex ONE physical trainable block through L logical positions (cascade's
per-block locality makes depth a **memory line-item**, not a silicon line-item); digital-side Adam.
- **⚡ 2026-07-13: 18-agent first-principles substrate down-select COMPLETE → HW_FIRST_PRINCIPLES_SEARCH.md** (verdict: incumbent CONFIRMED-MODIFIED at $150-260; step-0 = 3 new $0 fault-injection channels GATE all purchases: replica-asymmetry 2/5/10%, Wᵀ-mismatch 0.3/1.5%, 1-LSB-quantized updates @7/8/10b; null-offset engineering = the real critical path; refined Dillavou/UIUC ask lists in §5).
- **E-tier queue feeding this map:** softmax dynamic-noise cliff (cascade re-test), wq8/wq6 static
quantization (cascade re-test), divider mismatch (#4/#7), Gilbert-gate mismatch (#6), RoPE phase
error (#2), write pricing (#16), QK-norm vs tanh-logit-softcap A/B, error-channel SNR ledger.
|
