diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-04-07 23:48:09 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-04-07 23:48:09 -0500 |
| commit | ec20a776e0c3e026236942fe99f3840a39e211fd (patch) | |
| tree | 05138a794e7161bd26c9e96c7ec9c95e62993a8f /protocol | |
| parent | 05cd478cb45f78ccf89ab42918df9010cd534ede (diff) | |
CHECKLIST: add pitfall #6.5 — silent self-cosine fallback when aux nets not saved
Discovered in our own cnn_baseline.py: when the random feedback Bs (for
DFA) or bridge predictor (for SB/CB) are not persisted alongside the
model checkpoint, post-hoc Gamma computation cannot reconstruct the
local credit signal. Instead of erroring, the script falls back to
cos(BP_grad, BP_grad) = 1.0 and records that as Gamma. Reader who
doesn't notice the small 'Gamma_note' field interprets 1.0 as perfect
alignment.
Recommendation: always save aux nets alongside checkpoints; if they're
missing, report Gamma as N/A, not 1.0.
Diffstat (limited to 'protocol')
| -rw-r--r-- | protocol/CHECKLIST.md | 21 |
1 files changed, 21 insertions, 0 deletions
diff --git a/protocol/CHECKLIST.md b/protocol/CHECKLIST.md index 3227642..6302b32 100644 --- a/protocol/CHECKLIST.md +++ b/protocol/CHECKLIST.md @@ -69,6 +69,27 @@ bridge it's 5× (Γ ∈ [0.057, 0.285]). and let the reader pick. Do not collapse to a single number without showing the layer breakdown. +## 6.5 Auxiliary networks (Bs, bridge models) not persisted → silent self-cosine fallback + +We discovered this in our own dogfood codebase: the `cnn_baseline.py` script +records State Bridge Γ = 1.0 and Credit Bridge Γ = 1.0 in its JSON output. +Reading the source: when the auxiliary networks (the bridge predictor for +SB/CB, the random feedback `Bs` for DFA) are not saved alongside the model +checkpoint, the post-hoc Γ-computation script *cannot* reconstruct the +local credit signal. Instead of erroring out, the script falls back to +`cos(BP_grad, BP_grad) = 1.0` and silently records that. The JSON has a +`Gamma_note` field saying "BP grad self-cosine (feedback nets not saved)" +but a reader who doesn't read that field would interpret 1.0 as perfect +alignment. + +**Check**: when you save a model checkpoint for FA evaluation, ALSO save: +- the random feedback `Bs` for DFA +- the bridge / value network state_dict for SB / CB +- the EP feedback weights for EP + +If any of these are missing, your post-hoc Γ measurement is undefined. +Report `Γ = N/A` in your tables, NOT a fallback value of 1.0. + ## 6. Layer-0 dominates the headline Γ; deeper layers are ~0 For DFA on a 4-block ResMLP, the headline Γ ≈ 0.10 is driven almost entirely |
