summaryrefslogtreecommitdiff
path: root/protocol
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-04-07 23:48:09 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-04-07 23:48:09 -0500
commitec20a776e0c3e026236942fe99f3840a39e211fd (patch)
tree05138a794e7161bd26c9e96c7ec9c95e62993a8f /protocol
parent05cd478cb45f78ccf89ab42918df9010cd534ede (diff)
CHECKLIST: add pitfall #6.5 — silent self-cosine fallback when aux nets not saved
Discovered in our own cnn_baseline.py: when the random feedback Bs (for DFA) or bridge predictor (for SB/CB) are not persisted alongside the model checkpoint, post-hoc Gamma computation cannot reconstruct the local credit signal. Instead of erroring, the script falls back to cos(BP_grad, BP_grad) = 1.0 and records that as Gamma. Reader who doesn't notice the small 'Gamma_note' field interprets 1.0 as perfect alignment. Recommendation: always save aux nets alongside checkpoints; if they're missing, report Gamma as N/A, not 1.0.
Diffstat (limited to 'protocol')
-rw-r--r--protocol/CHECKLIST.md21
1 files changed, 21 insertions, 0 deletions
diff --git a/protocol/CHECKLIST.md b/protocol/CHECKLIST.md
index 3227642..6302b32 100644
--- a/protocol/CHECKLIST.md
+++ b/protocol/CHECKLIST.md
@@ -69,6 +69,27 @@ bridge it's 5× (Γ ∈ [0.057, 0.285]).
and let the reader pick. Do not collapse to a single number without showing
the layer breakdown.
+## 6.5 Auxiliary networks (Bs, bridge models) not persisted → silent self-cosine fallback
+
+We discovered this in our own dogfood codebase: the `cnn_baseline.py` script
+records State Bridge Γ = 1.0 and Credit Bridge Γ = 1.0 in its JSON output.
+Reading the source: when the auxiliary networks (the bridge predictor for
+SB/CB, the random feedback `Bs` for DFA) are not saved alongside the model
+checkpoint, the post-hoc Γ-computation script *cannot* reconstruct the
+local credit signal. Instead of erroring out, the script falls back to
+`cos(BP_grad, BP_grad) = 1.0` and silently records that. The JSON has a
+`Gamma_note` field saying "BP grad self-cosine (feedback nets not saved)"
+but a reader who doesn't read that field would interpret 1.0 as perfect
+alignment.
+
+**Check**: when you save a model checkpoint for FA evaluation, ALSO save:
+- the random feedback `Bs` for DFA
+- the bridge / value network state_dict for SB / CB
+- the EP feedback weights for EP
+
+If any of these are missing, your post-hoc Γ measurement is undefined.
+Report `Γ = N/A` in your tables, NOT a fallback value of 1.0.
+
## 6. Layer-0 dominates the headline Γ; deeper layers are ~0
For DFA on a 4-block ResMLP, the headline Γ ≈ 0.10 is driven almost entirely