From 0918c65cc07b3d4a06be56a46f80a7c5f143471a Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Mon, 3 Aug 2026 09:20:07 -0500 Subject: =?UTF-8?q?RESULT=2087:=20KHS=E8=AE=BA=E6=96=87=E7=B2=BE=E8=AF=BB?= =?UTF-8?q?=20=E2=80=94=20=E5=90=8C=E6=80=A7=E8=B4=A8=E8=8E=B7=E5=8F=91?= =?UTF-8?q?=E6=98=8E=E4=BA=BA=E8=83=8C=E4=B9=A6,=20=E4=BD=86=E8=BF=AD?= =?UTF-8?q?=E4=BB=A3=E8=AE=A1=E6=95=B0=E5=AF=B9=E6=AF=94=E5=8F=A3=E5=BE=84?= =?UTF-8?q?=E4=B8=8D=E5=BD=93=E5=B7=B2=E5=81=9C=E7=94=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 关键: 他们自由相亦为一次前向; BP等价性由他们主动写出(cos≈1是正确性依据非弱点); 他们未测余弦 (此轴我方更严格)。不利: 其扫描结论K≥4且"最优K≈层数", 我方K=3在线下一格(K电池落点极佳)。 口径修正: 其迭代为异步局部更新(一次约走一层), 我方一次走完整链 ⟹ "3次vs20次"是不同物相比且 偏向我方, 停用; 改用实测墙钟2.7× vs其6.7-12×。项目页proof卡与技术摘要已同步。 Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- docs/outreach/EPT_TECHNICAL_SUMMARY.pdf | Bin 24634 -> 25126 bytes docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 12 +++++++++--- docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf | Bin 29813 -> 30437 bytes 3 files changed, 9 insertions(+), 3 deletions(-) (limited to 'docs/outreach') diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf index 90c86d1..8a40aa3 100644 Binary files a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf and b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf differ diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex index ebf4102..8552afb 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -55,9 +55,15 @@ All values are tail window means over the last tenth of training, which is the s everywhere. Every difference above is smaller than or comparable to the spread between backprop seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically indistinguishable at these sizes. The 135M entry compares both arms at the same point in an -unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in -simulation is about twice backpropagation. To our knowledge the 135M model is the largest language -model trained from scratch with no backpropagation at any layer. +unfinished run and carries one seed per arm, so it is provisional. To our knowledge the 135M model is +the largest language model trained from scratch with no backpropagation at any layer. + +Training cost is 2.7 times backpropagation in wall clock, measured on the same GPU with the same +model and batch and with all other load suspended, and 3 times in analytic FLOPs. We quote the +measured ratio against backprop rather than a relaxation-iteration count, because iteration counts in +this literature measure different operations: the closest comparable work updates layers +asynchronously, so one of its iterations moves information about one layer, while one of ours +transmits through the whole stack. That work reports 6.7 to 12 times backprop in wall clock. \section{A result from this week, and why it matters for how we work} Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf index d764699..ee69847 100644 Binary files a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf and b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf differ -- cgit v1.2.3