From 15abb3ee664955bf9bb93c46825a4ad8947998fd Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Fri, 31 Jul 2026 04:40:51 -0500 Subject: =?UTF-8?q?Dillavou=E9=80=9A=E8=AF=9D=E6=9D=90=E6=96=99v3:=20?= =?UTF-8?q?=E6=9C=BA=E5=88=B6=E6=AE=B5=E6=8C=89RESULT=2077-79=E6=94=B9?= =?UTF-8?q?=E5=88=A4(=E8=AF=BB=E5=87=BA=E5=9C=B0=E6=9D=BF+=E9=9B=B6?= =?UTF-8?q?=E6=88=90=E6=9C=AC=E4=BF=AE=E5=A4=8D+=E5=B0=8F=E5=B0=BA?= =?UTF-8?q?=E5=AF=B8EP=E8=83=9CBP),=20Q2=E6=94=B9=E4=B8=BA=E6=9D=BF?= =?UTF-8?q?=E4=B8=8A=E5=AF=B9=E6=AF=94=E8=AF=BB=E5=87=BA/physical=20epsilo?= =?UTF-8?q?n,=20cheat=20sheet=E5=8A=A0=E7=A0=B4=E6=A1=88=E6=9B=B4=E6=96=B0?= =?UTF-8?q?=E8=8A=82?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/hardware/CALL_SHEET_DILLAVOU.tex | 82 +++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/hardware/CALL_SHEET_DILLAVOU.tex (limited to 'docs/hardware/CALL_SHEET_DILLAVOU.tex') diff --git a/docs/hardware/CALL_SHEET_DILLAVOU.tex b/docs/hardware/CALL_SHEET_DILLAVOU.tex new file mode 100644 index 0000000..5bb9ce9 --- /dev/null +++ b/docs/hardware/CALL_SHEET_DILLAVOU.tex @@ -0,0 +1,82 @@ +\documentclass[10pt]{article} +\usepackage[letterpaper,margin=0.72in,top=0.6in,bottom=0.6in]{geometry} +\usepackage[T1]{fontenc} +\usepackage{newpxtext,newpxmath} +\usepackage{booktabs,array,enumitem,xcolor,titlesec} +\definecolor{accent}{RGB}{138,31,31} +\titleformat{\section}{\normalsize\bfseries\color{accent}}{}{0pt}{} +\titlespacing{\section}{0pt}{9pt}{4pt} +\setlist[enumerate]{leftmargin=1.2em,itemsep=1.5pt,topsep=2pt,parsep=0pt} +\setlist[itemize]{leftmargin=1.1em,itemsep=1.5pt,topsep=2pt,parsep=0pt} +\pagestyle{empty} +\setlength{\parindent}{0pt} +\setlength{\parskip}{3pt} +\begin{document} + +{\large\bfseries EP transformer program $\rightarrow$ analog board: call sheet}\\ +{\small supplements the July 12 brief and build plan \,\textperiodcentered\, Yuren Hao (UIUC) \,\textperiodcentered\, August 2026} + +\section{Program state since the July brief (measured)} +\begin{itemize} +\item \textbf{72M-parameter transformer LM sealed:} EP final CE 3.364 (2 seeds) vs.\ a tuned +backprop twin at 3.288 (3 seeds, $\pm$0.006); gap $+$0.076 nats ($\approx$8\% perplexity). The +symmetric two-phase estimator narrows this to $+$0.044 ($\approx$4.5\%) at 1.67$\times$ cost. Zero +instability events in both EP seeds. To our knowledge the largest LM trained with no +backpropagation anywhere in the loop. +\item \textbf{135M trained end to end} (2.7B tokens), also with zero instability events. It surfaced +a width-dependent estimator loss localized to the upper layers --- which a per-layer probe has since +traced to the simulator's own \emph{readout floor}: at larger widths the nudge displacement falls +below single-precision machine epsilon relative to the activations and is rounded away in the +contrast read. +\item \textbf{The fix is free, and it is the software version of differential readout:} read the +contrast from the stored displacement itself, never as the difference of two large signals. Verified +against a double-precision control and a pre-registered amplitude-shift test; the loss closes 100\% +at zero cost. At the smaller rungs of our scaling ladder, EP now matches or beats the matched +backprop twin (2 seeds per arm). +\item \textbf{Random-sign nudging} (nudge polarity flipped randomly per batch) is the stability +winner at both scales: it removes the systematic single-sided bias at zero extra cost. On a board it +is a polarity switch. +\end{itemize} + +\section{The four primitive measurements (recipe-independent)} +\begin{small} +\begin{tabular}{@{}p{0.28\textwidth}p{0.40\textwidth}p{0.24\textwidth}@{}} +\toprule +\textbf{measurement} & \textbf{simulation anchor (measured)} & \textbf{board question}\\ +\midrule +1. contrast-update fidelity (exact difference-of-squares channel) & update tracks the \emph{faulted} +network at cos $\approx$ 0.97 under 8-bit weights, 1\% forward noise, 10\% nudge noise & does the +same co-adaptation law hold in physics\\[2pt] +2. nudge operating window and its floor & ceiling tracks loop gain and sinks over training +($\sim 1/\sigma^2$ of the output map); error channel is amplitude-indifferent across $20\times$ +(multiplicative), so the floor is set by \emph{additive} readout noise & measure the physical floor; +track window drift over training\\[2pt] +3. sign-only vs.\ exact vs.\ random-sign updates & sign works; random sign removes the bias at +scale; exact channel is the reference & continuous three-way comparison on one board\\[2pt] +4. EP current nudge vs.\ CL voltage clamp & the distinction formalized by McGinnis, Li, and Mori & +same board, same task, same metrology\\ +\bottomrule +\end{tabular} +\end{small} + +\section{Where we most want your judgment} +\begin{enumerate} +\item Is this the right primitive-level list? What would you add or cut, from your +imperfection-characterization experience? +\item \textbf{Reading the contrast:} our width-scaling loss was ultimately a readout-floor artifact, +and the lesson is that the contrast must be measured differentially, never as a difference of two +large signals. How do your update cells read the contrast, and what sets the smallest resolvable +contrast (the physical epsilon) on the board? +\item \textbf{Additive vs.\ multiplicative separation:} in simulation we sweep nudge amplitude at a +fixed task. What is the right bench protocol for the same separation on the board? +\item \textbf{Settling as a gate:} in simulation the relaxation residual predicts every blowup and +gates the weight update. What settling observable did your boards expose, and did you gate on it? +\item \textbf{Twin replicas vs.\ a time-multiplexed single network} at 8--32 edges: matching cost +against memory cost, your call. +\end{enumerate} + +\vspace{2pt} +{\small We are not committing the board before this conversation; rungs 1--3 remain under \$1k in +parts, funded on our side. Results are intended for open publication.} + +\end{document} -- cgit v1.2.3