|
Round 19's #3 critical experiment. Trained vanilla DFA s42 for 5 epochs,
saved checkpoint at each, then measured per-layer cos(e_T B^T, BP grad).
Key trajectory of ||g_l|| during vanilla DFA training:
ep 0: ~1e-3 (random init, healthy)
ep 1: ~1.4e-6 (3 OOM drop, STILL above 1e-7 floor)
ep 2: ~3e-7 (above floor)
ep 3: ~1.3e-7 (above floor, barely)
ep 4: ~7e-8 (BELOW floor)
ep 5: ~4e-8 (well below floor)
So ep 1, 2, 3 vanilla checkpoints are in the MEANINGFUL ||g|| regime.
Cos measurement on those:
ep 1: l0=+0.42, l1=+0.005, l2=-0.028, l3=-0.039, l4=-0.038
ep 2: l0=+0.44, l1=-0.002, l2=-0.040, l3=-0.055, l4=-0.054
ep 3: l0=+0.43, l1=+0.007, l2=-0.039, l3=-0.054, l4=-0.054
DEEP-LAYER COSINES ARE ESSENTIALLY ZERO AT EVERY VANILLA EPOCH, even when
||g|| is in the meaningful regime (ep 1: ||g||=6.7e-7).
Compare to penalized DFA s42 at 30 ep: deep cos = +0.17.
Hypothesis B confirmed: the penalty CREATED the deep-layer alignment.
It is a training outcome of the regularization, not a measurement-regime
revelation.
Paper implications: there are two distinct failure modes after all, but
they are not 'scale + direction'. They are:
(1) Measurement degeneracy via terminal LN gradient cancellation
(caught by diagnostic (b))
(2) Low intrinsic credit quality of random feedback even in the
meaningful regime (caught by direct cos measurement)
The penalty partially alleviates BOTH (residual stream contained AND
deep alignment improved from ~0 to +0.17), but neither fully.
|
