1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
|
\documentclass[10pt]{article}
\usepackage[letterpaper,margin=0.72in,top=0.6in,bottom=0.6in]{geometry}
\usepackage[T1]{fontenc}
\usepackage{newpxtext,newpxmath}
\usepackage{booktabs,array,enumitem,xcolor,titlesec}
\definecolor{accent}{RGB}{138,31,31}
\titleformat{\section}{\normalsize\bfseries\color{accent}}{}{0pt}{}
\titlespacing{\section}{0pt}{9pt}{4pt}
\setlist[enumerate]{leftmargin=1.2em,itemsep=1.5pt,topsep=2pt,parsep=0pt}
\setlist[itemize]{leftmargin=1.1em,itemsep=1.5pt,topsep=2pt,parsep=0pt}
\pagestyle{empty}
\setlength{\parindent}{0pt}
\setlength{\parskip}{3pt}
\begin{document}
{\large\bfseries EP transformer program $\rightarrow$ analog board: call sheet}\\
{\small supplements the July 12 brief and build plan \,\textperiodcentered\, Yuren Hao (UIUC) \,\textperiodcentered\, August 2026}
\section{Program state since the July brief (measured)}
\begin{itemize}
\item \textbf{72M-parameter transformer LM sealed:} EP final CE 3.364 (2 seeds) vs.\ a tuned
backprop twin at 3.288 (3 seeds, $\pm$0.006); gap $+$0.076 nats ($\approx$8\% perplexity). The
symmetric two-phase estimator narrows this to $+$0.044 ($\approx$4.5\%) at 1.67$\times$ cost. Zero
instability events in both EP seeds. To our knowledge the largest LM trained with no
backpropagation anywhere in the loop.
\item \textbf{135M trained end to end} (2.7B tokens), also with zero instability events. It surfaced
a width-dependent estimator loss localized to the upper layers --- which a per-layer probe has since
traced to the simulator's own \emph{readout floor}: at larger widths the nudge displacement falls
below single-precision machine epsilon relative to the activations and is rounded away in the
contrast read.
\item \textbf{The fix is free, and it is the software version of differential readout:} read the
contrast from the stored displacement itself, never as the difference of two large signals. Verified
against a double-precision control and a pre-registered amplitude-shift test; the loss closes 100\%
at zero cost. At the smaller rungs of our scaling ladder, EP now matches or beats the matched
backprop twin (2 seeds per arm).
\item \textbf{Random-sign nudging} (nudge polarity flipped randomly per batch) is the stability
winner at both scales: it removes the systematic single-sided bias at zero extra cost. On a board it
is a polarity switch.
\end{itemize}
\section{The four primitive measurements (recipe-independent)}
\begin{small}
\begin{tabular}{@{}p{0.28\textwidth}p{0.40\textwidth}p{0.24\textwidth}@{}}
\toprule
\textbf{measurement} & \textbf{simulation anchor (measured)} & \textbf{board question}\\
\midrule
1. contrast-update fidelity (exact difference-of-squares channel) & update tracks the \emph{faulted}
network at cos $\approx$ 0.97 under 8-bit weights, 1\% forward noise, 10\% nudge noise & does the
same co-adaptation law hold in physics\\[2pt]
2. nudge operating window and its floor & ceiling tracks loop gain and sinks over training
($\sim 1/\sigma^2$ of the output map); error channel is amplitude-indifferent across $20\times$
(multiplicative), so the floor is set by \emph{additive} readout noise & measure the physical floor;
track window drift over training\\[2pt]
3. sign-only vs.\ exact vs.\ random-sign updates & sign works; random sign removes the bias at
scale; exact channel is the reference & continuous three-way comparison on one board\\[2pt]
4. EP current nudge vs.\ CL voltage clamp & the distinction formalized by McGinnis, Li, and Mori &
same board, same task, same metrology\\
\bottomrule
\end{tabular}
\end{small}
\section{Where we most want your judgment}
\begin{enumerate}
\item Is this the right primitive-level list? What would you add or cut, from your
imperfection-characterization experience?
\item \textbf{Reading the contrast:} our width-scaling loss was ultimately a readout-floor artifact,
and the lesson is that the contrast must be measured differentially, never as a difference of two
large signals. How do your update cells read the contrast, and what sets the smallest resolvable
contrast (the physical epsilon) on the board?
\item \textbf{Additive vs.\ multiplicative separation:} in simulation we sweep nudge amplitude at a
fixed task. What is the right bench protocol for the same separation on the board?
\item \textbf{Settling as a gate:} in simulation the relaxation residual predicts every blowup and
gates the weight update. What settling observable did your boards expose, and did you gate on it?
\item \textbf{Twin replicas vs.\ a time-multiplexed single network} at 8--32 edges: matching cost
against memory cost, your call.
\end{enumerate}
\vspace{2pt}
{\small We are not committing the board before this conversation; rungs 1--3 remain under \$1k in
parts, funded on our side. Results are intended for open publication.}
\end{document}
|
