From ec20a776e0c3e026236942fe99f3840a39e211fd Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Tue, 7 Apr 2026 23:48:09 -0500 Subject: =?UTF-8?q?CHECKLIST:=20add=20pitfall=20#6.5=20=E2=80=94=20silent?= =?UTF-8?q?=20self-cosine=20fallback=20when=20aux=20nets=20not=20saved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Discovered in our own cnn_baseline.py: when the random feedback Bs (for DFA) or bridge predictor (for SB/CB) are not persisted alongside the model checkpoint, post-hoc Gamma computation cannot reconstruct the local credit signal. Instead of erroring, the script falls back to cos(BP_grad, BP_grad) = 1.0 and records that as Gamma. Reader who doesn't notice the small 'Gamma_note' field interprets 1.0 as perfect alignment. Recommendation: always save aux nets alongside checkpoints; if they're missing, report Gamma as N/A, not 1.0. --- protocol/CHECKLIST.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) (limited to 'protocol') diff --git a/protocol/CHECKLIST.md b/protocol/CHECKLIST.md index 3227642..6302b32 100644 --- a/protocol/CHECKLIST.md +++ b/protocol/CHECKLIST.md @@ -69,6 +69,27 @@ bridge it's 5× (Γ ∈ [0.057, 0.285]). and let the reader pick. Do not collapse to a single number without showing the layer breakdown. +## 6.5 Auxiliary networks (Bs, bridge models) not persisted → silent self-cosine fallback + +We discovered this in our own dogfood codebase: the `cnn_baseline.py` script +records State Bridge Γ = 1.0 and Credit Bridge Γ = 1.0 in its JSON output. +Reading the source: when the auxiliary networks (the bridge predictor for +SB/CB, the random feedback `Bs` for DFA) are not saved alongside the model +checkpoint, the post-hoc Γ-computation script *cannot* reconstruct the +local credit signal. Instead of erroring out, the script falls back to +`cos(BP_grad, BP_grad) = 1.0` and silently records that. The JSON has a +`Gamma_note` field saying "BP grad self-cosine (feedback nets not saved)" +but a reader who doesn't read that field would interpret 1.0 as perfect +alignment. + +**Check**: when you save a model checkpoint for FA evaluation, ALSO save: +- the random feedback `Bs` for DFA +- the bridge / value network state_dict for SB / CB +- the EP feedback weights for EP + +If any of these are missing, your post-hoc Γ measurement is undefined. +Report `Γ = N/A` in your tables, NOT a fallback value of 1.0. + ## 6. Layer-0 dominates the headline Γ; deeper layers are ~0 For DFA on a 4-block ResMLP, the headline Γ ≈ 0.10 is driven almost entirely -- cgit v1.2.3