diff options
Diffstat (limited to 'docs/audit/EP_READOUT_FLOOR_AUDIT.md')
| -rw-r--r-- | docs/audit/EP_READOUT_FLOOR_AUDIT.md | 38 |
1 files changed, 38 insertions, 0 deletions
diff --git a/docs/audit/EP_READOUT_FLOOR_AUDIT.md b/docs/audit/EP_READOUT_FLOOR_AUDIT.md index e925be3..7786b87 100644 --- a/docs/audit/EP_READOUT_FLOOR_AUDIT.md +++ b/docs/audit/EP_READOUT_FLOOR_AUDIT.md @@ -85,3 +85,41 @@ Hughes et al. Optica 2018、Kendall 2006.01981、2503.22810、2602.03670、Stoch **同一个算术,两种处方**。位移以独立张量存在时(dualprop 与我们),修复免费且彻底;两个状态来自 独立弛豫时(其余所有人),任何重写都救不回来(已验证),只剩 fp64 累加对比量或给 β 设下限两条路。 这一点必须写清楚,否则读者会望文生义地按"灾难性抵消"去找数值稳定的重写形式。 + +--- + +## 关键人际连接(审计本身未点出,本人核对) +**`Rasmuskh` = Rasmus Kjær Høier**(仓库 commit 署名 `Rasmus Kjær Høier`,README 指向 ICML 2024 +"Two Tales of Single-Phase Contrastive Hebbian Learning", arXiv:2402.08573),**即 KHS ImageNet PCN +论文 (arXiv:2606.03584, Kerjan/Høier/Scellier, 2026-06-02) 的第二作者**。 +论文已公开发表;**代码未发布**(arXiv 页与检索均无仓库链接,2026-08-02 复核)。 + +⟹ **策略推论**:一封发给 Ben Scellier 的信即可同时覆盖三个目标,且走已开启的暖渠道,无需三封冷邮件: +(1) 他自己 Rain 的 `energy-based-learning/training/sgd.py:357`;(2) 合著的 ImageNet PCN(β 标称 0.05, +敏感性扫描下探 2e-4, 余量薄 250×);(3) 合作者 Høier 的 dualprop(唯一出厂即危险者, 输出层可 1 行修复)。 +时机:等 135M 修复版完赛后一并发出(结果本身是这封信的由头)。 + +## 给 Ben 的信中技术段(预写, 不依赖 135M 结果) +> While tracing a width dependent quality loss in our own language model runs, we found the cause was +> arithmetic rather than the learning rule. We form the nudged state as z = o + d and then read the +> contrast as the squared norm of z minus o. Those agree algebraically and not in float32: forming z +> rounds away every component of d below machine epsilon times the local activation, and the later +> subtraction returns the damaged copy with no error signal. Wider models push more layers below that +> threshold, so the loss grew with width and imitated a scaling wall. Reading the contrast from the +> stored d instead, which has the same derivative, costs nothing and removed the effect completely. +> +> We then checked whether the pattern appears elsewhere, and it is common. The relative error of a +> float32 contrast estimator follows roughly epsilon over r, where r is the displacement to activation +> ratio, so r near 1e-5 costs a few percent and r near 1e-7 leaves noise. Most published work is +> protected by a large nudge or by float64 in numpy, and the margin is real but undocumented: none of +> the implementations we read assert on r or accumulate the contrast in double precision. Two specific +> notes, offered privately and only because they are close to you. The estimator in +> energy-based-learning at training/sgd.py:357 is unguarded at small nudging, and the beta sweep in the +> ImageNet PCN paper reaches 2e-4, where the margin is a couple of hundred times thinner than at the +> headline setting. Separately, Rasmus's dualprop code forms the states as act(ff plus or minus fb) and +> then recovers the difference by subtraction, in float32 at beta 0.01 on VGG16; at the output layer +> there is no activation in between, so the difference is exactly fb and the retained tensor can be +> substituted directly. In the hidden layers the difference passes through the activation, so the +> remedy there is to form it elementwise in higher precision rather than to substitute. +> +> A standalone checker is at [tools/contrast_floor_check.py], and it runs against any implementation. |
