diff options
Diffstat (limited to 'TINYSTORIES_ANALOG_LM_MVP.md')
| -rw-r--r-- | TINYSTORIES_ANALOG_LM_MVP.md | 345 |
1 files changed, 345 insertions, 0 deletions
diff --git a/TINYSTORIES_ANALOG_LM_MVP.md b/TINYSTORIES_ANALOG_LM_MVP.md new file mode 100644 index 0000000..243df53 --- /dev/null +++ b/TINYSTORIES_ANALOG_LM_MVP.md @@ -0,0 +1,345 @@ +# TinyStories analogue language-model MVP + +**Revised scope, 2026-07-13** + +This document replaces the earlier one-edge learning-tile proposal. The target here is an actual, end-to-end autoregressive language model trained on TinyStories: it accepts a text prompt, maintains context, chooses successive characters, and generates a story. The proposed runtime contains no processor, FPGA, ADC, DAC, digital weight memory, or free-running clock. + +The lowest-cost credible route is **not** to reproduce an OLMo2/GPT block. It is to train a compact recurrent student in software, freeze and physically encode its weights, and execute the complete autoregressive loop in analogue hardware. + +## 1. Exact claim + +Recommended paper/demo claim: + +> A self-timed analogue recurrent language model trained on TinyStories, using continuous-valued physical state and passive weight storage, which generates text autoregressively without a processor, data converters, or a periodic clock in the inference loop. + +This is a full language model, but the base model is trained offline. It is not a claim of fully analogue pretraining. + +A second, higher-risk configuration can train a low-rank readout in situ. That makes all *adaptive* parameters analogue, but the recurrent reservoir/backbone remains fixed. + +## 2. Why the original hardware map is the wrong MVP + +The existing map keeps an OLMo2-standard Transformer block and then purchases hardware to reproduce every consequence of that choice: + +- token lookup and a vocabulary-sized head; +- Q/K/V projections and causal attention memory; +- RoPE; +- QK normalization and RMSNorm; +- exponential normalization/softmax; +- gated MLP/SwiGLU; +- transpose reads for feedback; +- FPGA sequencing; +- DAC and ADC boundaries; +- digitally managed weight programming and optimization. + +That is an accelerator integration programme. It is not the minimum hardware needed to demonstrate a TinyStories LM. + +The TinyStories paper used GPT-Neo, a context length of 512, a local-attention window of 256, and the 10,000 most common GPT-Neo tokens. Even at hidden width 64, the input embedding alone therefore contains 640,000 weights. The paper also reports that the 1M/width-64 model is weak and that story consistency begins to emerge when width rises from 64 to 128. Preserving the Transformer architecture therefore commits the hardware to approximately one million or more physical weights before analogue attention is addressed. + +For analogue hardware, the decisive reductions are: + +1. use a small character alphabet instead of a 10,000-token BPE vocabulary; +2. compress the past into a fixed-size recurrent state instead of retaining a token-by-token attention context; +3. tie the transition weights across every generated character; +4. factor dense matrices into low-rank matrices; +5. use passive, fixed, low-precision weights rather than a programmable CIM evaluation platform; +6. use winner-take-all selection instead of implementing softmax at inference; +7. use completion-triggered handshaking instead of a periodic clock. + +## 3. Recommended model: `A-Tiny-256` + +### 3.1 Tokenization + +Normalize TinyStories to a 64-symbol alphabet: + +- 26 lowercase letters; +- 10 digits; +- space, newline, apostrophe and quotation marks; +- common punctuation; +- beginning/end/unknown/control symbols. + +Case folding is a deliberate hardware trade. A 96-symbol case-preserving version is possible with a modest increase in the output and input matrices. + +Recent recurrent-model research has already used ASCII character-level TinyStories with a 256-symbol vocabulary, so character-level TinyStories is a defensible benchmark rather than a different task in disguise. + +### 3.2 State and transition + +Use a 256-dimensional continuous state and rank 48 factors: + +\[ + u_t = \tanh(Bh_t + Ex_t + b_u), +\] + +\[ + c_t = \tanh(Au_t + d\odot h_t + b_c), +\] + +\[ + h_{t+1} = (1-\alpha)h_t + \alpha c_t, +\] + +\[ + q_t = Rh_{t+1}, \qquad \ell_t = Sq_t+b_o. +\] + +Here: + +- \(x_t\) is a one-of-64 character signal; +- \(h_t\in\mathbb{R}^{256}\) is the compressed story state; +- \(u_t,q_t\in\mathbb{R}^{48}\); +- \(\ell_t\in\mathbb{R}^{64}\) are character scores; +- \(\alpha\) is a fixed per-group or per-channel leak implemented by conductance ratios, not a multiplier. + +Approximate parameter count, excluding biases: + +| Matrix | Shape | Weights | +|---|---:|---:| +| State compression `B` | 48 × 256 | 12,288 | +| Character input `E` | 48 × 64 | 3,072 | +| State expansion `A` | 256 × 48 | 12,288 | +| Diagonal recurrence `d` | 256 | 256 | +| Readout compression `R` | 48 × 256 | 12,288 | +| Character head `S` | 64 × 48 | 3,072 | +| **Total** | | **43,264** | + +Quantization-aware training should target at least 45–60% exact zeros, leaving roughly 17,000–24,000 populated passive connections. The recurrent core can use ternary weights; input and output matrices should initially use five levels `{-2,-1,0,+1,+2}` because boundary matrices are generally more sensitive. + +### 3.3 Why this model rather than a GRU + +A full GRU requires three large recurrent projections plus sigmoid gates and hundreds of analogue multipliers. The proposed transition needs one low-rank nonlinear update and obtains its memory from residual/leaky state dynamics. + +A 2026 recurrent-pretraining study evaluated character-level TinyStories and found that nonlinear fixed-memory RNN performance improved smoothly with memory size and model scale. It also showed that teacher-derived memory targets can avoid long unrolled credit paths. This supports training a hardware-friendly recurrent student rather than forcing attention into the board. + +The model still requires digital pre-validation. No published result establishes that exactly 256 states and rank 48 will meet the desired story-quality threshold. The board should not be fabricated until the hardware-aware digital twin produces acceptable free-running completions. + +## 4. Training and compression pipeline + +Training is done off-board. Runtime is analogue. + +### Stage 1 — character-level teacher + +Train or reuse a competent character-level TinyStories Transformer/sequence teacher. Use a teacher with clearly better generation than the TinyStories 1M baseline; the original paper’s examples indicate that 8.3M-class models are a safer quality source than the 1M model. + +### Stage 2 — recurrent state distillation + +Train `A-Tiny-256` using: + +- next-character cross-entropy; +- soft-logit distillation from the teacher; +- one-step memory/state targets so the recurrent transition learns a Markovian compressed state; +- free-running rollout fine-tuning to reduce accumulated drift. + +### Stage 3 — hardware-aware quantization + +Train with the actual physical constraints: + +- core weights ternary; +- input/output weights five-level; +- grouped matrix gains rather than individual row calibration; +- activation saturation; +- resistor mismatch; +- amplifier offset and finite gain; +- switch charge injection; +- state-capacitor leakage; +- additive and multiplicative noise; +- asynchronous settling tolerance. + +Do not quantize a completed floating-point model as an afterthought. Train with the discrete weight set in the forward pass. + +### Stage 4 — fabrication export + +The export script produces: + +- a sparse crosspoint list for each passive matrix; +- resistor value and polarity at every nonzero crosspoint; +- matrix-level/group-level gain resistors; +- a SPICE netlist with measured component distributions; +- PCB pick-and-place files; +- a golden teacher-forced voltage trace for bring-up. + +## 5. Analogue hardware map + +### 5.1 Passive weight ROM + +Each matrix is a signed passive summing network. State and latent signals are differential. At each nonzero crosspoint: + +- sign chooses the positive or negative source rail; +- magnitude chooses one of two resistor values; +- zero omits the component. + +This is a physical analogue ROM. It is not programmable during runtime, but it avoids the cost and peripheral complexity of memristor/SRAM-CIM evaluation hardware. + +Use plug-in matrix cards so a new trained model requires replacing only passive cards, not the state and control boards. + +### 5.2 State + +Use two 256-channel capacitor banks: + +- bank A holds `h_t` while bank B acquires `h_{t+1}`; +- after analogue completion detection, the banks exchange roles; +- low-input-bias buffers isolate the capacitors; +- a manual freeze input stops generation without refreshing or digitizing the state. + +Two banks make the physical update correspond closely to the trained discrete recurrence. A later continuous-time single-bank version can be investigated after the model works. + +### 5.3 Nonlinearities + +Implement the two tanh-like banks with saturating differential or diode-feedback amplifiers: + +- 48 channels for `u_t`; +- 256 channels for `c_t`. + +Exact mathematical tanh is unnecessary. The measured transfer curve is inserted into hardware-aware training. + +### 5.4 Residual/leak update + +Implement + +\[ + h_{t+1}=(1-\alpha)h_t+\alpha c_t +\] + +with resistor ratios and capacitor charging. No Gilbert multiplier is needed. Use 8–16 shared leak groups initially; per-channel leak values can be added only if digital ablations show a material gain. + +### 5.5 Character selection + +A 64-way current-mode winner-take-all network selects the next character directly from the logits. Softmax is unnecessary for deterministic generation. The TinyStories paper itself reports many sample completions at zero temperature. + +For diverse generation, add a controllable analogue noise voltage to the logits before winner-take-all. This is not exact categorical softmax sampling, but it is adequate for a research comparison between deterministic and noisy generation. + +### 5.6 Prompt and output + +- Prompt entry: a passive keyboard/diode matrix produces one-of-64 character lines. Each key event advances the state through the same self-timed transition used during generation. +- Generation: the winner line is fed directly back as the next one-of-64 input. +- Visible output: 64 labelled LEDs are sufficient for a strict no-digital demo. A logic analyser or computer may record the winner lines for plots, provided it is electrically outside the feedback loop. + +## 6. No-clock operation + +A language model necessarily emits discrete characters, so some event boundary is unavoidable. “No clock” should mean **no oscillator and no periodic global timing reference**, not “no bistable state anywhere.” + +The controller is completion-driven: + +1. source state and current character are held; +2. the destination state capacitors move toward their new values; +3. an aggregate settling circuit monitors total state-update current or slew; +4. the winner-take-all margin must remain stable for an RC qualification interval; +5. a comparator/latch emits one transition event; +6. the token latch and state-bank roles exchange; +7. the next analogue computation begins immediately. + +The character rate therefore changes with physical settling time. There is no free-running clock, phase counter, or firmware scheduler. + +The most accurate wording is **self-timed analogue datapath with asynchronous event control**. Calling it a purely continuous analogue system would be misleading because characters are discrete symbols. + +## 7. Estimated bill of materials + +These are engineering estimates for one assembled research system, excluding ordinary laboratory instruments and software-training compute. PCB assembly quotes will dominate the uncertainty. + +### Recommended `A-Tiny-256` + +| Category | Indicative implementation | Estimate | +|---|---|---:| +| Passive matrix cards | 4–5 PCBs; approximately 17k–24k populated 0402/0201 resistors after pruning | $220–450 | +| Summing, buffering and saturation | approximately 200–350 low-cost quad op-amp packages, mixed LM324-class and TLV9004-class | $120–260 | +| State banks and switches | 512 capacitors, low-leakage buffers, approximately 100–150 quad analogue-switch packages | $70–160 | +| WTA and completion detector | comparator/WTA cells, latches, RC qualification, noise option | $30–90 | +| Power, connectors, backplane and prompt panel | split or virtual-ground supply, headers, LEDs, keyboard matrix | $70–160 | +| Rework and component tolerance margin | spare boards/components | $100–220 | +| **Expected research build** | | **$610–1,340** | + +A conservative grant/BOM line should be **$1,500**, not $5,000–$20,000. + +### Scale options + +| Version | State/rank | Approx. weights | Purpose | Estimated build | +|---|---:|---:|---|---:| +| `A-Tiny-128` | 128 / 24 | 12,416 | full-loop electrical prototype; language quality may be weak | $280–600 | +| **`A-Tiny-256`** | **256 / 48** | **43,264** | recommended end-to-end LM MVP | **$610–1,340** | +| `A-Tiny-512` | 512 / 64 | 107,008 | quality contingency after digital validation | $1,400–2,800 | + +Relative to the realistic $10,000–$20,000 part of the original estimate, the recommended build is roughly an order of magnitude cheaper while including the entire autoregressive loop rather than one Transformer block. + +## 8. What to delete from the existing component map + +| Existing item | TinyStories analogue MVP replacement | +|---|---| +| 10k-token embedding lookup + DAC | 64-line character input and passive `E` matrix | +| RoPE mixers/DDS | delete | +| Q/K/V and attention projection crossbars | fixed-memory recurrent transition | +| QK normalization | delete | +| causal softmax | delete; 64-way WTA at output only | +| causal KV/context memory | 256 analogue state voltages | +| SwiGLU | two saturating-amplifier banks | +| RMSNorm banks | bounded recurrent state and matrix-level gains | +| bidirectional transpose access | delete for inference MVP | +| ADC + digital LM head | passive `R/S` matrices + analogue WTA | +| FPGA/phase sequencer | completion detector + asynchronous latch | +| programmable CIM evaluation board | plug-in passive resistor ROM cards | +| digital optimizer/write-pulse system | offline training and fixed weight export | + +## 9. Optional in-situ analogue-learning variant + +If learning in hardware is required for the first paper, use a reservoir/readout configuration rather than trying to adapt all 43,000 weights. + +Recommended limit: + +- fixed recurrent backbone and fixed `R` projection; +- 48 latent readout features; +- 64 outputs; +- 3,072 adaptive readout weights; +- local sign-LMS or analogue delta updates under teacher forcing. + +This is still an actual autoregressive TinyStories language model. All trainable parameters are analogue, but the base recurrent dynamics are fixed. Expected additional cost is approximately $350–800, with substantial layout and drift risk. + +A full 43,000-weight adaptive board is not a low-cost COTS project. Commercial memristor teaching hardware illustrates the mismatch: a current $800 discovery board exposes only a 1×16 device array, and a $2,000 crossbar product exposes at most a 32×32 array and is listed as sold out. Scaling such kits to a language model is economically and architecturally irrational. + +## 10. Fabrication gates + +Do not order the full board until all of the following pass in the digital twin: + +1. `A-Tiny-256` generates multi-sentence, 200–500-character TinyStories completions from held-out prompts. +2. It maintains at least basic entity and topic continuity better than a character n-gram baseline. +3. Ternary/five-level quantization causes an acceptable degradation in held-out loss and human/GPT story grading. +4. Free-running rollouts do not collapse into short loops under the planned noise and mismatch distributions. +5. A SPICE/macromodel simulation reproduces the quantized model’s top-character choice on representative teacher-forced traces. +6. The asynchronous controller produces exactly one state/token transition per settled computation over process, voltage and temperature sweeps. + +If the 256/48 model misses the language-quality gate, increase state/rank in software first. Do not compensate by adding transformer peripherals to the hardware. + +## 11. Experimental programme + +The first publishable evaluation should include: + +- digital floating-point teacher; +- quantized digital recurrent student; +- circuit-macromodel student; +- physical analogue board; +- teacher-forced logit correlation and top-1 agreement; +- free-running divergence length; +- story grammar, consistency and repetition scores; +- characters per second and joules per character; +- supply, temperature and resistor-mismatch sweeps; +- deterministic WTA versus noisy WTA; +- state dimension/rank ablation; +- 128-state electrical prototype versus 256-state final board. + +## 12. Claims not to make + +Do not claim: + +- OLMo2 equivalence; +- a Transformer implementation; +- exact softmax sampling; +- fully analogue pretraining of all weights; +- a mathematically clock-free continuous system; +- TinyStories-2.5M-equivalent quality before measurements. + +The defensible contribution is stronger and cleaner: **a complete, self-timed analogue language-model inference loop on TinyStories**, with a cost and component count appropriate to an academic laboratory. + +## References + +- Ronen Eldan and Yuanzhi Li, “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”, 2023. https://arxiv.org/abs/2305.07759 +- Akarsh Kumar et al., “Pretraining Recurrent Networks without Recurrence”, 2026. https://arxiv.org/abs/2606.06479 +- Felix Köster and Atsushi Uchida, “Reservoir Computing as a Language Model”, 2026. https://arxiv.org/abs/2507.15779 +- Sam Dillavou et al., “Machine Learning Without a Processor: Emergent Learning in a Nonlinear Electronic Metamaterial”, 2024. https://arxiv.org/abs/2311.00537 +- Knowm, Memristor Discovery and crossbar product pages. https://knowm.com/ +- Texas Instruments, TLV9004 and CD4066B product documentation. https://www.ti.com/ |