summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorYuren Hao <yurenh2@illinois.edu>2026-08-03 09:20:07 -0500
committerYuren Hao <yurenh2@illinois.edu>2026-08-03 09:20:07 -0500
commit0918c65cc07b3d4a06be56a46f80a7c5f143471a (patch)
treea2f36bcebbc2c5b3de2a3bf277d2925854a1f79d /docs
parentf0552201cfea2521052bdd7faa6227e73bd1118b (diff)
RESULT 87: KHS论文精读 — 同性质获发明人背书, 但迭代计数对比口径不当已停用
关键: 他们自由相亦为一次前向; BP等价性由他们主动写出(cos≈1是正确性依据非弱点); 他们未测余弦 (此轴我方更严格)。不利: 其扫描结论K≥4且"最优K≈层数", 我方K=3在线下一格(K电池落点极佳)。 口径修正: 其迭代为异步局部更新(一次约走一层), 我方一次走完整链 ⟹ "3次vs20次"是不同物相比且 偏向我方, 停用; 改用实测墙钟2.7× vs其6.7-12×。项目页proof卡与技术摘要已同步。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'docs')
-rw-r--r--docs/campaign/CASCADE_ABLATION_PLAN.md19
-rw-r--r--docs/outreach/EPT_TECHNICAL_SUMMARY.pdfbin24634 -> 25126 bytes
-rw-r--r--docs/outreach/EPT_TECHNICAL_SUMMARY.tex12
-rw-r--r--docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdfbin29813 -> 30437 bytes
4 files changed, 28 insertions, 3 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index 7315cb9..8bb231b 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -2100,3 +2100,22 @@ BP seed2 在跑(定散布尺度)。
- **对外口径调整(即刻生效)**: 不得再用"我们只做 3 次弛豫所以比同类的 20 次便宜"作为卖点;
改为"级联架构使自由相按构造收敛, 仅 nudged 平衡需迭代, 且 K 的边际收益实测在 3 以内饱和",
并**主动公布 K=1 的余弦**——审稿人自己会算, 先说优于被抓。
+
+## RESULT 87 (2026-08-03): KHS 论文精读 — Ben 自己的方法同性质, 但我们的成本对比口径不当须改
+读 arXiv:2606.03584 (Kerjan/Høier/Scellier) 全文, 逐句核对:
+- **自由相亦为一次前向**(原文 "In the free phase (β=0), finding the global minimum of F_PCN with
+ respect to h reduces to a single forward pass.")⟹ "自由相按构造收敛"是发明人自己的设计, 非我方捷径。
+- **BP 等价性由他们主动写出**("in the limit when β tends to 0, the EP gradients are equal to the BP
+ gradients.")⟹ cos≈1 是正确性依据, 不是弱点。RESULT 86 的担忧就此降级。
+- **他们未测梯度余弦**(全文无 angle/cosine 对照, 仅 "In every situation, EP is competitive with BP")
+ ⟹ 此轴上我方更严格, 可主张。
+- ⚠️ **对我方不利的一条**: 敏感性扫描结论为 "as long as K≥4 and 0.0002≤β≤0.1, the final error rate is
+ mostly insensitive"; 且 "empirically, the optimal value for K is the number of layers"。我们的 K=3
+ 恰在其 K≥4 线下一格 ⟹ 正在跑的 K=1/2/3 CE 电池落点极佳, 结果须如实报。
+- **口径错误(已修正)**: 他们的迭代是**异步局部更新**(先偶层后奇层), 一次迭代信息约走一层(故最优
+ K≈层数); 我方 forces() 一次调用走完整条链。**"我们 1 相×3 次 vs 他们 2 相×10 次"是拿不同物在比,
+ 且偏向我方** ⟹ 停用该对比, 一律改用实测墙钟(我方 2.7× BP vs 其 6.7–12× BP)。
+ 已修: 项目页 proof 卡("0.4× vs EP-family baseline" → "2.7× vs backprop, measured on matched
+ hardware")、技术摘要(新增成本段并说明为何不用迭代计数)。
+- 他们主实验用**两相 centered**(β 与 −β, 中间重置回自由平衡); 论文亦指出 backward/random 方案只需
+ 一相且 "particularly appealing for PCNs" ⟹ 我方单相随机符号(bsign)与其判断同向, 可引为佐证。
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf
index 90c86d1..8a40aa3 100644
--- a/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf
+++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.pdf
Binary files differ
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
index ebf4102..8552afb 100644
--- a/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
+++ b/docs/outreach/EPT_TECHNICAL_SUMMARY.tex
@@ -55,9 +55,15 @@ All values are tail window means over the last tenth of training, which is the s
everywhere. Every difference above is smaller than or comparable to the spread between backprop
seeds, which is 0.003 in cross entropy at 72M, so EP and backpropagation are statistically
indistinguishable at these sizes. The 135M entry compares both arms at the same point in an
-unfinished run and carries one seed per arm, so it is provisional. Wall clock cost of EP training in
-simulation is about twice backpropagation. To our knowledge the 135M model is the largest language
-model trained from scratch with no backpropagation at any layer.
+unfinished run and carries one seed per arm, so it is provisional. To our knowledge the 135M model is
+the largest language model trained from scratch with no backpropagation at any layer.
+
+Training cost is 2.7 times backpropagation in wall clock, measured on the same GPU with the same
+model and batch and with all other load suspended, and 3 times in analytic FLOPs. We quote the
+measured ratio against backprop rather than a relaxation-iteration count, because iteration counts in
+this literature measure different operations: the closest comparable work updates layers
+asynchronously, so one of its iterations moves information about one layer, while one of ours
+transmits through the whole stack. That work reports 6.7 to 12 times backprop in wall clock.
\section{A result from this week, and why it matters for how we work}
Between 72M and 135M we had been measuring a quality loss that grew with model width, resisted every
diff --git a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf
index d764699..ee69847 100644
--- a/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf
+++ b/docs/outreach/EPT_TECHNICAL_SUMMARY_final.pdf
Binary files differ