summaryrefslogtreecommitdiff
path: root/experiments
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-04-08 06:11:25 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-04-08 06:11:25 -0500
commit02c3d2c80805daedb2b6c8e9d6e5f36c52d361a1 (patch)
treee1dcf1086721c1fd392d9bfc410fb179d42f3063 /experiments
parent47f833be0f2abacd0ce53bbe32c7ac7b60fd59d6 (diff)
Round 36: upgrade (b) wording + add EP random-target neg control to §3
Two changes from round 36: 1. §3 paragraph 3: replace 'observational association' with full causal claim based on existing April 7 no-out_ln data (3 seeds, ResMLP-d256+terminal-LN removed, residual skip kept): ||h_L||=1.21e7 (Mode 1 (a) still fires) but ||g_L||=7.4e-4 (HEALTHY, ~10000x above floor — (b) eliminated). Final acc 0.327±0.013 indistinguishable from vanilla DFA's 0.308±0.014. Wording upgraded to 'terminal LayerNorm is necessary for Mode 1(b) in the audited residual ResMLP and ViT-Mini setting'. 2. §3 paragraph after random-target ablation: add EP under random targets smoke result (||h_L||=586 at ep 5 vs DFA's 14510 at ep 3, 25x gap). Random-target assay now cleanly separates fixed-feedback methods (explode) from EP (bounded). Cross-method negative control complete. - experiments/ep_baseline.py: add --random_targets flag + train_ep parameter - v2.5 paper compiles to 15 pages, main content 1-9 (right at E&D limit) Combined picture (rounds 32-36): - Mode 1 (a) localized to 'fixed-feedback local-credit objectives without scale control on architectures absorbing scale at output'. Falsified: residual skip (round 33), task signal (round 34), DFA-specific (round 35). EP is the working negative control (round 36). - Mode 1 (b) localized to terminal LayerNorm via the 1/||h|| Jacobian. Causally established by April 7 no_outln 3-seed data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Diffstat (limited to 'experiments')
-rw-r--r--experiments/ep_baseline.py9
1 files changed, 7 insertions, 2 deletions
diff --git a/experiments/ep_baseline.py b/experiments/ep_baseline.py
index 7f3d004..36f97f6 100644
--- a/experiments/ep_baseline.py
+++ b/experiments/ep_baseline.py
@@ -90,7 +90,7 @@ def ep_nudged_phase(model, x, y, h_free, beta, T_nudge, alpha_nudge):
def train_ep(model, trl, tel, dev, epochs=100, lr=1e-3, wd=0.01,
- beta=0.5, T_nudge=20, alpha_nudge=0.1):
+ beta=0.5, T_nudge=20, alpha_nudge=0.1, random_targets: bool = False):
L = model.num_blocks
# Separate optimizers for different parts
@@ -104,6 +104,8 @@ def train_ep(model, trl, tel, dev, epochs=100, lr=1e-3, wd=0.01,
model.train()
for x, y in trl:
x = x.view(x.size(0), -1).to(dev); y = y.to(dev)
+ if random_targets:
+ y = torch.randint(0, 10, y.shape, device=dev)
# ---- FREE PHASE ----
# Standard forward pass to get free fixed point
@@ -281,6 +283,8 @@ def main():
p.add_argument('--lr', type=float, default=1e-3)
p.add_argument('--wd', type=float, default=0.01)
p.add_argument('--d_hidden', type=int, default=256)
+ p.add_argument('--random_targets', action='store_true',
+ help='Replace each minibatch label with i.i.d. random class targets (codex round 36 OPTION EP).')
args = p.parse_args()
os.makedirs(args.output_dir, exist_ok=True)
@@ -294,7 +298,8 @@ def main():
print(f"[{args.method} s={args.seed}] Training EP beta={args.beta} T={args.T_nudge} alpha={args.alpha_nudge}", flush=True)
model = train_ep(model, trl, tel, dev, epochs=args.epochs, lr=args.lr, wd=args.wd,
- beta=args.beta, T_nudge=args.T_nudge, alpha_nudge=args.alpha_nudge)
+ beta=args.beta, T_nudge=args.T_nudge, alpha_nudge=args.alpha_nudge,
+ random_targets=args.random_targets)
acc = evaluate(model, tel, dev)
diag = compute_diagnostics(model, tel, dev, args.beta, args.T_nudge, args.alpha_nudge)