diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-08-03 09:20:07 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-08-03 09:20:07 -0500 |
| commit | 0918c65cc07b3d4a06be56a46f80a7c5f143471a (patch) | |
| tree | a2f36bcebbc2c5b3de2a3bf277d2925854a1f79d /docs/outreach | |
| parent | f0552201cfea2521052bdd7faa6227e73bd1118b (diff) | |
RESULT 87: KHS论文精读 — 同性质获发明人背书, 但迭代计数对比口径不当已停用
关键: 他们自由相亦为一次前向; BP等价性由他们主动写出(cos≈1是正确性依据非弱点); 他们未测余弦
(此轴我方更严格)。不利: 其扫描结论K≥4且"最优K≈层数", 我方K=3在线下一格(K电池落点极佳)。
口径修正: 其迭代为异步局部更新(一次约走一层), 我方一次走完整链 ⟹ "3次vs20次"是不同物相比且
偏向我方, 停用; 改用实测墙钟2.7× vs其6.7-12×。项目页proof卡与技术摘要已同步。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'docs/outreach')
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY.pdf | bin | 24634 -> 25126 bytes | |||
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY.tex | 12 | ||||
| -rw-r--r-- | docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf | bin | 29813 -> 30437 bytes |
3 files changed, 9 insertions, 3 deletions
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf Binary files differindex 90c86d1..8a40aa3 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex index ebf4102..8552afb 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex @@ -55,9 +55,15 @@ All values are tail window means over the last tenth of training, which is the s everywhere. Every difference above is smaller than or comparable to the spread between backprop seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically indistinguishable at these sizes. The 135M entry compares both arms at the same point in an -unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in -simulation is about twice backpropagation. To our knowledge the 135M model is the largest language -model trained from scratch with no backpropagation at any layer. +unfinished run and carries one seed per arm, so it is provisional. To our knowledge the 135M model is +the largest language model trained from scratch with no backpropagation at any layer. + +Training cost is 2.7 times backpropagation in wall clock, measured on the same GPU with the same +model and batch and with all other load suspended, and 3 times in analytic FLOPs. We quote the +measured ratio against backprop rather than a relaxation-iteration count, because iteration counts in +this literature measure different operations: the closest comparable work updates layers +asynchronously, so one of its iterations moves information about one layer, while one of ours +transmits through the whole stack. That work reports 6.7 to 12 times backprop in wall clock. \section{A result from this week, and why it matters for how we work} Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf Binary files differindex d764699..ee69847 100644 --- a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf +++ b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf |
