From 54dd6c5d7571e59bf5bbc389fb6bbf3ad43ded6a Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Sun, 2 Aug 2026 13:01:43 -0500 Subject: =?UTF-8?q?=E6=8A=80=E6=9C=AF=E6=91=98=E8=A6=81=E6=9B=B4=E6=96=B0:?= =?UTF-8?q?=2072M=E7=9C=9F=E6=95=B0=E5=AD=973.400=20vs=203.397(+0.4%),=201?= =?UTF-8?q?35M=E6=94=B9=E4=B8=BA=E8=BF=9B=E8=A1=8C=E4=B8=AD(=E5=90=8C?= =?UTF-8?q?=E8=BF=9B=E5=BA=A6=E6=8C=81=E5=B9=B3);=20=E5=85=A8=E8=A1=A8?= =?UTF-8?q?=E7=BB=9F=E4=B8=80=E5=B0=BE=E7=AA=97=E5=8F=A3=E5=BE=84=E5=B9=B6?= =?UTF-8?q?=E6=B3=A8=E6=98=8E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) (limited to 'docs/outreach/EPT_TECHNICAL_SUMMARY.tex') diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex index 4ac89fc..ebf4102 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -45,14 +45,17 @@ backprop seeds at 72M. Corpus is FineWeb-Edu with a 32k vocabulary. 11M & 4.194 & 4.197 & $-0.4\%$ (EP ahead)\\ 18M & 3.945 & 3.933 & $+1.2\%$\\ 36M & 3.649 & 3.648 & $+0.1\%$\\ -72M & rerun in progress & 3.288 & tracks the twins at matched progress\\ -135M & rerun queued & 3.090 & \\ +72M & 3.400 & 3.397 & $+0.4\%$\\ +135M & rerun in progress & 3.469 & level at matched progress\\ \bottomrule \end{tabular} \end{center} -Differences at the first three sizes are smaller than the spread between seeds, so EP and -backpropagation are statistically indistinguishable there. Wall clock cost of EP training in +All values are tail window means over the last tenth of training, which is the statistic we quote +everywhere. Every difference above is smaller than or comparable to the spread between backprop +seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically +indistinguishable at these sizes. The 135M entry compares both arms at the same point in an +unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in simulation is about twice backpropagation. To our knowledge the 135M model is the largest language model trained from scratch with no backpropagation at any layer. -- cgit v1.2.3