diff options
Diffstat (limited to 'TINYSTORIES_EQPROP_ANALOG_TRAINER_MVP.md')
| -rw-r--r-- | TINYSTORIES_EQPROP_ANALOG_TRAINER_MVP.md | 287 |
1 files changed, 287 insertions, 0 deletions
diff --git a/TINYSTORIES_EQPROP_ANALOG_TRAINER_MVP.md b/TINYSTORIES_EQPROP_ANALOG_TRAINER_MVP.md new file mode 100644 index 0000000..715fdcb --- /dev/null +++ b/TINYSTORIES_EQPROP_ANALOG_TRAINER_MVP.md @@ -0,0 +1,287 @@ +# TinyStories EqProp Analogue Trainer MVP + +**Corrected scope — 2026-07-13** + +This document supersedes and deprecates both earlier proposals: + +- `CLOCKLESS_ANALOG_MVP_PLAN.md` — too small and not a language model. +- `TINYSTORIES_ANALOG_LM_MVP.md` — inference-only and therefore not the requested system. + +The target is an **end-to-end TinyStories language model trained from random initialization by Equilibrium Propagation in physical analogue circuitry**. The embedding, attention/memory core, output head, and optimizer state are trainable on the board. There is no frozen backbone, no offline-distilled model, and no backpropagation engine. + +The practical clock claim is: + +> No processor, FPGA, ADC, DAC, numerical digital memory, digital optimizer, or periodic global clock participates in the learning loop. Voltages and capacitor charges represent states, weights, optimizer statistics, and errors. A small asynchronous handshake controller changes phases and addresses only after analogue settling events. + +A literal system with no switches or Boolean events at all requires a fully spatial circuit and is substantially more expensive. The recommended MVP is therefore **self-timed switched analogue**, not digitally clocked. + +--- + +## 1. The central design decision + +Do not implement the OLMo2-standard block in the current component map for the first board. That architecture creates the expensive hardware requirements: + +- independent Q/K/V/O matrices and a non-reciprocal attention Jacobian; +- RoPE mixers; +- QK normalization and multiple RMSNorm banks; +- SwiGLU multipliers; +- transposed-Jacobian circuitry for AsymEP/AEP; +- a vocabulary head behind ADC/DAC boundaries; +- FPGA phase sequencing; +- a digital Adam/Muon sidecar. + +Instead, use an **energy-native attention language model** in which the same physical couplings are used in both directions. This permits classic EqProp with a local contrast rule and removes the adjoint network from the MVP. + +The model is still attention-based and autoregressive. It is not a reservoir, a frozen student, or an inference-only device. + +--- + +## 2. Recommended model: `TinyEP-256-R32` + +### 2.1 Task + +- Dataset: TinyStories, streamed as character-level next-token examples. +- Alphabet: 127 symbols, matching the existing TinyStories character-level software rung. +- Context length: 256 characters. +- Objective: next-character cross-entropy. +- Training: from random analogue weights. +- Generation: autoregressive sampling or argmax after training. + +### 2.2 State and architecture + +For each example, one 256-dimensional visible/query state `z` relaxes against: + +1. **Energy attention over the 256-character context.** Context token embeddings act as dynamically instantiated memories. Attention is a 256-way softmax in each of eight 32-dimensional heads. Keys and values are tied by construction, so the interaction is reciprocal. +2. **A Hopfield/DenseAM memory stage.** A 512-unit rectified memory replaces the Transformer FFN. +3. **A tied token embedding/output decoder.** The same factorized parameter bank embeds context characters and decodes the converged state into 127 logits. +4. **A leak/confinement term.** This ensures a bounded energy and supplies the contraction margin required by EqProp. + +No RoPE is used. Causality is physical: only the 256 preceding positions are connected. A fixed distance-dependent bias voltage can be added to the attention scores. + +### 2.3 Low-rank physical parameterization + +Use rank 32 factors as actual reciprocal intermediate layers, not as a software-only matrix compression. + +| Trainable bank | Shape | Physical weights | +|---|---:|---:| +| token factor A | 127 × 32 | 4,064 | +| embedding/decoder factor B | 32 × 256 | 8,192 | +| state-to-memory factor U | 256 × 32 | 8,192 | +| memory factor V | 32 × 512 | 16,384 | +| gains and biases | — | <1,000 | +| **Total** | — | **about 37,000** | + +The factors represent approximately 164,000 effective dense embedding and memory coefficients while requiring only about 37,000 analogue weight variables. This is roughly 25 times fewer trainable physical parameters than the existing 0.92M-parameter TinyStories software rung, while preserving its 256-dimensional state and 256-character context. + +The board should expose a rank-64 expansion footprint. Populating rank 64 approximately doubles the physical weights if rank 32 misses the language-quality gate. + +--- + +## 3. Exact EqProp training loop + +Let `s` contain the query, attention, memory, and output-node voltages. Let `E_theta(s, x)` be the circuit energy for context `x`, and let `C(s, y)` be next-character cross-entropy for target `y`. + +Two state replicas use the same stored weight voltages: + +``` +τ ds+/dt = -∂E/∂s(s+, x) - β ∂C/∂s(s+, y) +τ ds-/dt = -∂E/∂s(s-, x) + β ∂C/∂s(s-, y) +``` + +After both replicas have entered the valid equilibrium region, every weight capacitor receives the centred local EqProp update current + +``` +Cw dwi/dt = -(η / 2β) [∂E/∂wi(s+, x) - ∂E/∂wi(s-, x)]. +``` + +Because every trainable term is a reciprocal bilinear coupling, this reduces to a local difference of pre/post correlations. There is no reverse-layer schedule, stored activation tape, or software gradient. + +The 127-way output softmax directly produces probability currents. A one-hot target line subtracts the target current, so the physical nudge is proportional to `p - y`; the board does not need to calculate or digitize the scalar cross-entropy. + +### 3.1 Phase operation without a clock + +EqProp still requires different boundary conditions. “No clock” therefore means no periodic timing source, not no events. + +1. Apply a context and target with `β = 0` and let both replicas settle. +2. A residual envelope detector asserts `valid_free` when state-current or state-slew falls below threshold. +3. An asynchronous latch applies `+β` and `-β` target currents. +4. Both replicas relax. The local correlation-difference currents are integrated into the weight capacitors only while both residual detectors are valid. +5. A completion C-element requests the next training example. + +The duration of every phase is determined by physical settling, not by a counter. A continual-EP mode can update weights throughout the nudged relaxation and removes the need to retain a free-phase correlation snapshot. + +--- + +## 4. Recommended hardware implementation + +### 4.1 Analogue weight memory + +Each scalar signed weight is a bipolar voltage around a common-mode level stored on one capacitor. A MOS access device connects the cell to a shared analogue compute/update lane. Because sign is represented by voltage rather than by two positive conductances, one storage cell is sufficient per scalar weight. + +Recommended first implementation: + +- 37k 1T1C analogue weight cells across four to eight plug-in matrix cards; +- 10–100 nF storage capacitors, selected after leakage and dielectric-absorption tests; +- guarded high-impedance traces and driven shields around the weight banks; +- analogue rails that impose hard weight limits; +- optional slow common-mode restoration that does not alter the differential weight value. + +The same cells are addressed in forward and transpose order, so reciprocal coupling uses exactly the same physical weight voltage. + +### 4.2 Self-timed analogue matrix engine + +A fully spatial 37k-crosspoint system with a local multiplier and update cell at every synapse is not the low-cost route. The MVP should use 16 or 32 shared analogue lanes: + +- sample selected weight voltages; +- multiply them by selected state voltages with four-quadrant transconductance cells; +- sum currents into state capacitors; +- compute the `+β` and `-β` products in matched lanes; +- subtract the two correlation currents and return the result to the selected weight capacitor. + +An asynchronous one-hot scanner advances only after lane and integrator completion comparators agree. It has no oscillator and no programmable controller. This is a switched-analogue coordinate-descent realization of the same energy dynamics. + +Populate four training examples in parallel. The four context/query state sets share the same weight bank, and their local gradient currents sum by KCL before reaching each weight capacitor. With the centred `+β/-β` pair this requires eight state replicas but does not duplicate any trainable weight. The small increase in state memory buys physical mini-batch averaging and substantially improves update SNR. + +The scientific claim is analogue EqProp training, not constant-time inference. A later fully spatial revision can recover the speed and energy advantages after the learning rule is validated. + +### 4.3 Softmax and nonlinearities + +Use one reusable 256-channel exponential-normalization bank: + +- subthreshold BJT/MOS exponential cells; +- KCL denominator/current normalization; +- 256 channels for context attention; +- 127 active channels for the output distribution; +- ReLU or smooth rectifier cells for the 512 Hopfield memory units. + +No general multiplier array is needed for SwiGLU because there is no SwiGLU. No RMSNorm bank is required; bounded state swings, leak, matrix-gain rails, and the homeostatic controller provide the operating range. + +### 4.4 Analogue optimizer + +Plain matrix-wise SGD is sufficient for electrical bring-up but is likely too weak for the final TinyStories run. The board should include a bypassable **analogue Adafactor-like preconditioner**: + +- one leaky second-moment capacitor per matrix row; +- one leaky second-moment capacitor per matrix column; +- square-law envelope detectors driven by local gradient currents; +- transconductance gain proportional to the inverse row/column RMS estimate; +- current clipping; +- parameter-scale normalization; +- no momentum bank. + +This adds `O(rows + columns)` analogue state rather than one second-moment cell per weight. Weight decay is a controlled leakage toward the common-mode reference. Learning-rate warmup is an RC ramp, not a software schedule. + +### 4.5 Stability and validity control + +The controller is part of the training rule, not optional instrumentation: + +- residual/slew envelope from state capacitors; +- RC moving average; +- global contraction/leak control voltage with diode-set floor and ceiling; +- update gate that blocks learning outside the valid equilibrium region; +- saturation fuse that disconnects the weight-update bus after a large excursion; +- manual and analogue-programmed nudge-amplitude rails. + +The controller should be independently observable and bypassable so that its effect is a publishable ablation. + +--- + +## 5. What remains outside the analogue learning core + +TinyStories is a large discrete corpus. Storing the entire corpus as analogue voltages is not a meaningful contribution and would dominate the apparatus. + +The permitted boundary is a corpus reader or ROM that only selects 127 one-hot context/target lines and waits for the board’s `next_example` handshake. It performs no matrix arithmetic, no loss computation, no gradient calculation, no optimization, and no model-state storage. + +For the strictest demonstration, the reader can be a simple asynchronous address/ROM board rather than a computer. Replacing the corpus store with magnetic or optical tape would make the system more literally analogue but would not strengthen the EqProp claim. + +--- + +## 6. Budget + +These are research-build estimates excluding oscilloscope, bench supplies, and corpus-storage hardware. They assume automated SMT assembly for the dense storage cards and in-house design/debug. + +### 6.1 Rank-32 recommended build + +| Subsystem | Estimated cost | +|---|---:| +| 37k analogue weight cells, matrix-card PCBs and assembly | $650–1,250 | +| 16–32 dual-sign MAC/update lanes | $180–420 | +| state banks, buffers, integrators and residual detectors | $180–400 | +| 256-channel reusable softmax/rectifier periphery | $80–220 | +| row/column RMS optimizer and homeostatic controller | $100–250 | +| power, backplane, connectors, spares and rework | $200–450 | +| **Total** | **$1,390–2,990** | + +At current distributor scale, inexpensive dual MOSFET packages are roughly three cents and small ceramic capacitors are fractions of a cent to about one cent in reel quantities. The raw switch-plus-capacitor storage devices are therefore not the main obstacle; PCB area, assembly, guarding, analogue-lane matching, and rework dominate. + +### 6.2 Cheaper and larger variants + +| Configuration | Physical weights | Estimated build | Use | +|---|---:|---:|---| +| rank 16 | ~18.5k | $850–1,650 | electrical/algorithmic minimum; high language-quality risk | +| **rank 32** | **~37k** | **$1.4–3.0k** | recommended academic MVP | +| rank 64 | ~74k | $2.4–4.8k | expansion if rank 32 fails preflight | +| fully spatial, no scanner | ~37k plus per-synapse update circuitry | $5–10k+ | later speed/energy demonstrator, not first board | + +The defensible claim is therefore a roughly **2–14× reduction** from the current `$5–20k` map. A reliable full TinyStories trainer below `$500` is not credible with COTS analogue storage and tens of thousands of trainable weights. Sub-$1k is possible only by accepting rank 16, aggressive hand assembly, or a substantially weaker model. + +--- + +## 7. Delete/replace map + +| Current map item | MVP replacement | +|---|---| +| SRAM-CIM evaluation module | custom 1T1C analogue weight cards + shared analogue lanes | +| FPGA sequencer | asynchronous completion chain and C-elements | +| DAC/ADC boundaries | direct voltage/current state and target clamps | +| independent Q/K/V/O | reciprocal energy attention with tied key/value coupling | +| AsymEP/AEP `J^T` correction | removed in MVP; classic EqProp on conservative dynamics | +| RoPE DDS/mixers | fixed causal distance-bias voltages | +| QK-norm and RMSNorm banks | bounded activations, leak, gain rails and homeostasis | +| SwiGLU multiplier bank | Hopfield/DenseAM rectified memory | +| digital LM head and CE | analogue tied decoder + 127-way softmax + `p-y` nudge current | +| Adam/Muon sidecar | analogue row/column RMS preconditioner | +| digital settle/retry logic | residual comparators, latches and analogue fuse | +| gradient telemetry in loop | external measurement only; never used to update weights | + +--- + +## 8. Fabrication gates + +Do not lay out the 37k-cell board until the exact hardware-constrained digital twin passes all of these: + +1. The rank-32 energy model trains from random initialization on TinyStories with EqProp, not BPTT. +2. It beats a character n-gram baseline on held-out cross-entropy and produces nontrivial short completions. +3. Row/column RMS preconditioning materially improves over matrix-wise SGD. +4. Training remains stable with the planned asynchronous coordinate-update order. +5. Quantized device gains, capacitor leakage, switch charge injection, lane offsets, and dynamic noise do not destroy the EqProp update direction. +6. A centred `+β/-β` macromodel retains useful gradient alignment at the intended nudge amplitude and bandwidth. +7. Rank 32 is adequate; otherwise populate the rank-64 footprint before changing the architecture. + +The physical MVP acceptance test is: + +- start from randomized on-board weight voltages; +- train every model parameter in situ; +- show a sustained held-out-loss reduction on streamed TinyStories data; +- generate held-out text with the trained board; +- demonstrate that reversing the nudge produces ascent and disabling the contrast path stops learning; +- compare hardware, SPICE/macromodel, and constrained-software trajectories; +- report joules per accepted update, update throughput, drift, and temperature sensitivity. + +--- + +## 9. Recommended paper claim + +> We demonstrate end-to-end in-situ training of a character-level TinyStories attention language model by Equilibrium Propagation in a self-timed analogue circuit. Model states, trainable weights, loss nudges, local parameter updates, optimizer statistics, and stability control are represented by physical voltages and currents. The learning core uses no processor, ADC, DAC, digital optimizer, or periodic clock. + +Do not claim OLMo2 equivalence, full-scale LLM quality, or a literally switch-free circuit. The contribution is stronger when stated accurately: a complete analogue EqProp language-model trainer, not an inference accelerator and not a one-edge physics demonstration. + +--- + +## Primary references + +- Eldan and Li, *TinyStories: How Small Can Language Models Be and Still Speak Coherent English?*, arXiv:2305.07759. +- Ernoult et al., *Equilibrium Propagation with Continual Weight Updates*, arXiv:2005.04168. +- Kendall et al., *Training End-to-End Analog Neural Networks with Equilibrium Propagation*, arXiv:2006.01981. +- Bacvanski et al., *Dense Associative Memories with Analog Circuits*, arXiv:2512.15002. +- Scurria et al., *Equilibrium Propagation for Non-Conservative Systems*, arXiv:2602.03670. +- Shazeer and Stern, *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost*, arXiv:1804.04235. |