|
3-seed result on the existing dfa_s{42,123,456}.pt checkpoints from
results/confirmatory/checkpoints_A2/, computing per-layer cosine of
DFA's local credit signal e_T@B_l^T vs the true BP gradient at h_l.
Key findings:
per-layer cos (3-seed mean):
l0: +0.42 (high — embedding alignment)
l1: +0.006 (essentially zero)
l2: -0.015 (essentially zero)
l3: -0.004 (essentially zero)
l4: -0.004 (essentially zero)
layer-mean across all 5: +0.07-0.10
The deep blocks (l1-l4) have essentially zero alignment with BP grad in
the vanilla scale-failure regime. Layer 0 dominates the headline.
The script reconstructs the training-time random Bs by replaying the RNG
sequence (torch.manual_seed + ResidualMLP construction + randn draws),
since the existing checkpoints don't save Bs. For the still-running
direction-quality experiment which DOES save Bs, the script auto-detects
the dict format and uses the saved Bs directly.
|
