summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2026-04-07Add audit table example: protocol applied to BP/DFA/SB/CB/EPYurenHao0426
5-method audit table on 4-block d=256 ResMLP CIFAR-10 seed 42: - BP: trustworthy (acc 0.615, h_L=2e2, g_L=4e-4, stab 0.099) - DFA: walked back via (a)+(b)+(d) — h_L=4e8, g_L=4e-9, undercuts frozen - State Bridge: walked back via all 4 diagnostics — stability 0.992 is the cleanest possible drift-dominated case - Credit Bridge: walked back via all 4 — stability 0.352, also drift mode - EP: trustworthy (acc 0.359, h_L=3e3, g_L=2e-4, stab -0.036) — paper's internal control case This is the §2 audit evidence for the main-track paper. Confirms that standard headline acc + Γ silently fails on 3 of 5 methods on this architecture, while the 4-diagnostic protocol catches all three.
2026-04-07Add FA diagnostic protocol reference implementationYurenHao0426
Codex round 15 #1 priority for the E&D-track paper: - protocol/protocol.py: 4 diagnostics (residual norms, BP grad norms, cross-batch direction stability, and a frozen-baseline comparator) - protocol/report.py: DiagnosticReport with per-diagnostic verdicts and pretty-printer - protocol/smoke_test.py: validates BP/DFA/EP checkpoints produce the expected verdicts (BP/EP trustworthy; DFA walked back via residual explosion + BP grad at floor) - protocol/README.md: usage, audit cases, threshold rationale - protocol/CHECKLIST.md: 6 evaluation pipeline pitfalls (norm(-1), cosine_similarity eps clamp, fp16 underflow, Bs reproducibility, aggregation, layer-0 dominance) - protocol/REPORTING_TEMPLATE.md: per-method fillable form for FA papers
2026-04-03Add 5 extra seeds to synthetic cross-state distance (now 10 seeds for all ↵YurenHao0426
methods) BP/DFA/SB/CB: added seeds 2048,3000,4000,5000,6000 (L=4 only, all 3 alphas) Total: 1290 rows (was 990) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Recompute all CNN diagnostics with fixed BP gradient flowYurenHao0426
CNN CIFAR-10 (5 seeds, fixed Gamma): BP: acc=86.8%, Gamma=0.970, rho=0.603 DFA: acc=56.7%, Gamma=0.896, rho=0.061 EP: acc=50.6%, Gamma=0.484, rho=0.450 SB: acc=63.3%, Gamma=1.000 (BP self-cos, feedback nets not saved), rho=0.601 CB: acc=31.8%, Gamma=1.000 (BP self-cos), rho=0.226 DFA Gamma=0.896 is notably high — CNN DFA credit aligns well with BP gradients. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Fix CNN compute_bp_grads: remove inter-layer detach so gradients flow to all ↵YurenHao0426
layers Old code detached hidden states between layers, making layers 0-2 disconnected from the loss (gradient = None → 0). Fixed by keeping the forward graph connected. BP CNN Gamma per-layer now: [0.985, 0.990, 0.987, 0.967] (was [0, 0, 0, 0.967]) But gradient norms are ~1e-17 (genuine numerical precision issue with CNN architecture). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Complete EP data: 10-seed synthetic + 6-seed CIFAR persample + cross-stateYurenHao0426
EP synthetic: 30 JSONs + 30 checkpoints (10 seeds × 3α) EP CIFAR persample: 6 seeds × 4 layers × 256 samples = 6144 rows added Synth cross-state: 150 EP rows added (990 total) cifar_persample_all.csv: 30720 rows (was 24576, +6144 EP) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03EP synthetic 10 seeds complete: 30 JSONs + 30 checkpoints + cross-state distanceYurenHao0426
Updated synth_cross_state_distance.csv with 150 EP rows (990 total). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add checkpoint saving to ep_synthetic.pyYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP cross-state distance for CIFAR + verify CNN summaryYurenHao0426
EP CIFAR d_BP: L0=2.0×, L4=26.7× (much closer to BP than DFA=162×/2.5M×) EP synthetic: no checkpoints saved (ep_synthetic.py didn't save .pt) CNN summary: 20 rows confirmed correct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP synthetic per-seed CSV + synthetic cross-state distanceYurenHao0426
EP synthetic: 15 rows (3α × 5 seeds) Synth cross-state: 840 rows (3α × 2L × 4 methods × 5 seeds × (L+1) layers) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP to BP gradient sparsity analysisYurenHao0426
EP CIFAR d=256: s(1e-6)=100%, mean_norm=1.41e-04 EP produces networks where ALL samples have non-zero BP gradients, unlike DFA (0.4%), SB (21%), CB (3%). EP is closer to BP (98.7%). Updated clean_sparsity_summary.csv (980 rows, now includes EP). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add cross-method hidden state distance vs BPYurenHao0426
Non-BP methods produce radically different representations: DFA L0: 162×, L4: 2.5M× relative to BP hidden norms SB L0: 3.2×, L4: 1.1M× CB L0: 59×, L4: 1.4M× (BP vs itself = 0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Update all EP results with sign-corrected creditYurenHao0426
EP Synthetic (fixed): Gamma=+0.13~0.20, rho=+0.25 EP CIFAR d=256: Gamma=+0.007, rho=+0.051 EP CIFAR d=512: Gamma=+0.000, rho=-0.002 EP CNN: Gamma=+0.248, rho=+0.492 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Fix EP credit sign in cnn_baseline.pyYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Recompute EP CIFAR d=256 diagnostics with sign fixYurenHao0426
EP d=256 (5 seeds): acc=31.9%, Gamma=+0.007 (was -0.13), rho=+0.051 (was -0.037) Sign correction: -(h_nudge - h_free)/β aligns EP credit with BP gradient direction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix EP credit sign: negate (h_nudge - h_free)/β to align with BP grad directionYurenHao0426
EP nudge moves h toward lower loss (opposite to BP grad which points toward loss increase). Without negation, Gamma is negative and rho is -0.25. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP synthetic results: 15 JSONs (3α × 5 seeds)YurenHao0426
EP synthetic: acc high (92-96%) but Gamma negative (-0.13 to -0.20), rho=-0.25 EP credit direction may be inverted or diagnostics have issue. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add --d_hidden arg to ep_baseline.py for d=512 supportYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix ep_synthetic: bp dict needs L+1 entries for EP credit comparisonYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP synthetic ladder scriptYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add CNN SB+CB results (5 seeds each), update summary CSVYurenHao0426
CNN CIFAR-10 (5 seeds): BP: 86.8%±0.3%, Gamma=0.238, rho=0.250 DFA: 56.7%±2.0%, Gamma=0.216, rho=0.017 SB: 63.3%±0.5%, Gamma=0.045, rho=0.298 CB: 31.8%±6.2%, Gamma=0.013, rho=0.033 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix CNN state bridge: use custom CNNStateBridge for variable input dimsYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add SB and CB methods to cnn_baseline.pyYurenHao0426
State bridge: per-layer StateBridgeNet predicting h3 from flattened h_l Credit bridge: per-layer ValueNet with terminal + bridge consistency + DFA warmup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix and recompute GELU ablation Gamma from checkpointsYurenHao0426
ReLU MLP (L=4 d=256): BP: acc=61.1%, Gamma=1.000, rho=0.998 DFA: acc=30.7%, Gamma=0.104, rho=-0.001 SB: acc=15.5%, Gamma=0.300, rho=0.159 CB: acc=28.7%, Gamma=0.298, rho=0.007 Note: SB/CB Gamma uses BP gradient as proxy (feedback nets not checkpointed). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix gelu_ablation.py: compute method-specific Gamma instead of hardcoded 1.0YurenHao0426
DFA now uses regenerated DFA Bs for credit; SB/CB use BP as proxy (feedback nets not saved). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add summary CSVs for EP, GELU ablation, CNNYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP baseline (5 seeds), GELU ablation (20 runs), CNN BP+DFA (10 runs)YurenHao0426
EP (L=4 d=256): acc≈30%, Gamma≈0, rho≈0 — EP credit signal weak on feedforward MLP GELU ablation (ReLU variant): 4 methods × 5 seeds complete CNN BP+DFA: 5 seeds each, BP + DFA on SmallCNN Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add GELU/ReLU ablation script for CIFAR MLPYurenHao0426
Note: existing ResidualMLP already uses GELU. This adds ResidualMLPReLU variant. Ablation compares ReLU vs GELU for BP/DFA/SB/CB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add CNN baseline: SmallCNN with BP/DFA/EP on CIFAR-10YurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP baseline implementation (Scellier & Bengio 2017) for CIFAR MLPYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add d=512 support sparsity: 20 JSONs + summary CSVYurenHao0426
BP: s(1e-6)=92.7%, norm=2.70e-04, r_inf=0.159, PR=0.300 DFA: s(1e-6)=0.1%, norm=5.31e-08 SB: s(1e-6)=20.3%, norm=2.33e-06 CB: s(1e-6)=1.2%, norm=9.88e-08 Same pattern as d=256, confirming width-independence of the sparsity gap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add d512_sparsity.py: support sparsity for d=512 checkpointsYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add missing bp_s456.json for CIFAR d=512 (rerun after SIGTERM)YurenHao0426
bp s=456: acc=0.5999, rho=0.9881, nse=0.4764 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add CIFAR L=4 d=512 confirmatory: 4 methods × 5 seeds with checkpointsYurenHao0426
BP: 60.6%±0.3%, rho=0.989 DFA: 30.8%±0.5%, rho=0.003 State Bridge: 21.2%±3.7%, rho=0.119 Credit Bridge: 30.1%±0.5%, rho=0.002 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add cifar_d512_confirmatory.py: L=4 d=512 with checkpoint savingYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add P3 protocol panel: method ranking across 5 protocol slicesYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add per-sample gradient stats: 24576 rows (256 samples × 4 layers × 4 ↵YurenHao0426
methods × 6 seeds) Columns: method, seed, layer, sample_id, grad_norm, log10_grad_norm, r_inf, pr, hoyer, topk1, topk5 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean_sparsity_persample.py: per-sample gradient statsYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean_sparsity_summary.csv: 960 rows aggregated from 168 JSONsYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean sparsity results: 168 JSONs from independent processes on GPU 1YurenHao0426
CIFAR: 24 JSONs (4 methods × 6 seeds), BP s(1e-6)=98% confirmed Synthetic: 144 JSONs (4 methods × 6 seeds × 3 alphas × 2 depths) All data reliable — each method+seed in separate Python process. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add run_clean_sparsity.sh: shell runner for independent-process sparsityYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean_sparsity_full.py: independent-process full sparsity analysisYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean gradient check: independent Python process per method, GPU 1YurenHao0426
Clean results (each method in fresh Python process): BP: mean_norm=2.58e-04, s(1e-6)=98% — CONFIRMED DFA: layer 0 = 2.86e-07 (1.2%), layers 1-3 ≈ 2.4e-09 (0%) SB: layer 0 = 6.13e-06 (86%), layers 1-3 ≈ 1e-09 (0%) CB: layer 0 = 6.33e-07 (18%), layers 1-3 ≈ 5e-10 (0%) Method A (autograd.grad) and Method B (retain_grad) give identical results. Previous 1e-12 results were caused by Python process state pollution in combined scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add element-wise gradient concentration analysis (CPU, from checkpoints)YurenHao0426
BP gradients are relatively uniform: top1%=7.1%, PR=0.327, eff_dim=0.632 DFA gradients extremely concentrated: top1%=40.6%, PR=0.089, eff_dim=0.272 SB/CB intermediate: top1%=17-21%, PR=0.14-0.17 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add confirmatory supplement: T1-T4 from checkpoints (no retraining)YurenHao0426
WARNING: All methods (including BP) show near-zero BP hidden gradients (~1e-12-1e-14) when computed via manual forward with detached hidden states. This is inconsistent with the earlier first-priority analysis which showed BP at 2.86e-04. Investigation needed. T1: 40 rows (4 methods × 10 seeds) - full metrics T2: 800 rows (support sparsity, 5 thresholds × 4 methods × 10 seeds × 4 layers) T3: 48 rows (gradient norm distributions, 3 seeds × 4 methods × 4 layers) T4: 100 rows (active-subset Gamma, 5 thresholds × 2 methods × 10 seeds) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add extended sparsity analysis: A4 per-layer, B1 snapshots, B2 active ↵YurenHao0426
subset, C1/C2 A4: Per-layer support — DFA/SB/CB layers 1-3 have 0% support at τ=1e-6 Only BP has ~95% support; only SB layer 0 has 53% B1: Snapshot evolution — old snapshot checkpoints have near-zero grads (data issue) B2: Active subset — with τ=1e-6, no active samples for non-BP methods C1: Active vs inactive cosine — only inactive subset exists for non-BP C2: Energy concentration — near-zero for non-BP methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add BP support sparsity analysis: threshold sweep + gradient histogramsYurenHao0426
A1 Synthetic: all methods have >93% support at τ=1e-6 (gradients rarely zero) A2 CIFAR: massive gap — BP 98.4% vs DFA 0.4% vs SB 21% vs CB 3% DFA-trained CIFAR networks have near-zero BP gradients for 99.6% of samples This explains why Gamma is unreliable for CIFAR non-BP methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Recompute BP and DFA Gamma with near-zero gradient filteringYurenHao0426
BP Gamma: raw~0.99, filtered=1.000 (confirms self-cosine artifact from zero grads) DFA Gamma (synth): raw~0.01-0.16, filtered~0.01-0.17 (minimal filtering effect) DFA Gamma (CIFAR): raw=0.107, filtered=0.466 (99.7% samples have near-zero BP grad!) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31Update naive StateErr v3: L2 norm ratio formula, with checkpoints savedYurenHao0426
Formula: ||h_{L//2} - h_L||_2 / ||h_L||_2 (scalar L2 ratio) A1: 240 rows (3 alpha × 2 depth × 4 methods × 10 seeds) A2: 40 rows (4 methods including BP × 10 seeds) All model checkpoints saved in checkpoints_A1/ and checkpoints_A2/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31Add BP supplement for A2 CIFAR: 10 seeds with acc, Gamma, rho, naive_StateErrYurenHao0426
BP 10-seed results: acc=0.614±0.003, Gamma=1.0, rho=0.998 Appended to A2_cifar_state_vs_credit.csv and A2_naive_state_err.csv Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-15 13:02:00 +0000