|
Direct deep-block credit measurement on penalized DFA s42 checkpoint
(lam=1e-2, 30 epochs, just trained):
per-layer cos(e_T B^T, BP grad) — TRAINING Bs, no eps clamp:
l0: +0.316 (±0.188) ||g||=9.18e-7 ||a||=4.53
l1: +0.169 (±0.087) ||g||=8.87e-7 ||a||=4.57
l2: +0.151 (±0.084) ||g||=8.77e-7 ||a||=4.50
l3: +0.165 (±0.099) ||g||=8.73e-7 ||a||=4.64
l4: +0.166 (±0.098) ||g||=8.69e-7 ||a||=4.64
layer-mean: +0.193
Compare to vanilla DFA (existing measurement, scale-broken regime):
l0: +0.42 l1-4: ~0 (essentially zero)
CRITICAL INTERPRETATION: The penalty doesn't just fix scale, it ALSO
restores deep-layer direction quality from ~0 to ~0.17. This contradicts
the prior 'two failure modes' framing where I assumed direction would
remain broken even after scale fix. The honest story is:
- vanilla DFA: scale catastrophic, BP grad at floor, cosine measurement
DEGENERATE (cos ~0 is noise dominance, not 'no alignment')
- penalized DFA: scale fixed, BP grad healthy, cosine measurement
INTERPRETABLE — and the value is +0.17 on deep layers (partially
aligned, much less than BP's self-cosine of 1.0)
- the +0.17 alignment explains why penalized DFA gets 0.36 (60% of
BP's 0.61) — partial credit gives partial training, not zero training
The 'second failure mode' claim is wrong. There's ONE unified failure
mode (scale + measurement degeneracy), and the penalty rescues BOTH.
The remaining gap to BP is 'partial credit quality', not a separate
failure mode.
|
