From df9f69bc9172b3473be144ff8a17370bc7a68e64 Mon Sep 17 00:00:00 2001
From: YurenHao0426 <Blackhao0426@gmail.com>
Date: Wed, 8 Apr 2026 00:47:38 -0500
Subject: MAJOR: penalized DFA deep-layer cosine is +0.17, NOT zero
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Direct deep-block credit measurement on penalized DFA s42 checkpoint
(lam=1e-2, 30 epochs, just trained):

  per-layer cos(e_T B^T, BP grad) — TRAINING Bs, no eps clamp:
    l0: +0.316  (±0.188)  ||g||=9.18e-7  ||a||=4.53
    l1: +0.169  (±0.087)  ||g||=8.87e-7  ||a||=4.57
    l2: +0.151  (±0.084)  ||g||=8.77e-7  ||a||=4.50
    l3: +0.165  (±0.099)  ||g||=8.73e-7  ||a||=4.64
    l4: +0.166  (±0.098)  ||g||=8.69e-7  ||a||=4.64
  layer-mean: +0.193

Compare to vanilla DFA (existing measurement, scale-broken regime):
    l0: +0.42  l1-4: ~0 (essentially zero)

CRITICAL INTERPRETATION: The penalty doesn't just fix scale, it ALSO
restores deep-layer direction quality from ~0 to ~0.17. This contradicts
the prior 'two failure modes' framing where I assumed direction would
remain broken even after scale fix. The honest story is:

  - vanilla DFA: scale catastrophic, BP grad at floor, cosine measurement
    DEGENERATE (cos ~0 is noise dominance, not 'no alignment')
  - penalized DFA: scale fixed, BP grad healthy, cosine measurement
    INTERPRETABLE — and the value is +0.17 on deep layers (partially
    aligned, much less than BP's self-cosine of 1.0)
  - the +0.17 alignment explains why penalized DFA gets 0.36 (60% of
    BP's 0.61) — partial credit gives partial training, not zero training

The 'second failure mode' claim is wrong. There's ONE unified failure
mode (scale + measurement degeneracy), and the penalty rescues BOTH.
The remaining gap to BP is 'partial credit quality', not a separate
failure mode.
---
 results/dfa_pen_short/dfa_pen_lam0.01_s42.json | 43 ++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 results/dfa_pen_short/dfa_pen_lam0.01_s42.json

(limited to 'results/dfa_pen_short')

diff --git a/results/dfa_pen_short/dfa_pen_lam0.01_s42.json b/results/dfa_pen_short/dfa_pen_lam0.01_s42.json
new file mode 100644
index 0000000..1be1de6
--- /dev/null
+++ b/results/dfa_pen_short/dfa_pen_lam0.01_s42.json
@@ -0,0 +1,43 @@
+{
+  "config": {
+    "seed": 42,
+    "epochs": 30,
+    "lr": 0.001,
+    "wd": 0.01,
+    "lam": 0.01,
+    "output_dir": "results/dfa_pen_short"
+  },
+  "final_test_acc": 0.3593,
+  "log": [
+    {
+      "epoch": 0,
+      "h_L_norm": 8.893179893493652,
+      "g_2_norm": 0.0009934091940522194,
+      "acc_eval": 0.115234375
+    },
+    {
+      "epoch": 1,
+      "h_L_norm": 788.7547607421875,
+      "g_2_norm": 1.1413892934797332e-05,
+      "acc_eval": 0.3359375
+    },
+    {
+      "epoch": 10,
+      "h_L_norm": 7251.46533203125,
+      "g_2_norm": 2.3037373466650024e-06,
+      "acc_eval": 0.3662109375
+    },
+    {
+      "epoch": 20,
+      "h_L_norm": 11374.17578125,
+      "g_2_norm": 1.813692506402731e-06,
+      "acc_eval": 0.373046875
+    },
+    {
+      "epoch": 30,
+      "h_L_norm": 12158.3349609375,
+      "g_2_norm": 1.7568806924828095e-06,
+      "acc_eval": 0.375
+    }
+  ]
+}
\ No newline at end of file
-- 
cgit v1.2.3