summaryrefslogtreecommitdiff
path: root/results
AgeCommit message (Collapse)Author
2026-04-07Protocol diagnostic (a): use max per-block growth, not max/min ratioYurenHao0426
Old metric: max(||h||) / max(||h_0||, eps). False-positives on ViT-style architectures because the cls token at layer 0 (right after patch_embed) has anomalously small magnitude (~0.3-1.5), inflating the ratio even on healthy BP-trained ViTs. New metric: max_l(||h_{l+1}|| / ||h_l||) — the largest single-block residual amplification. Architecture-invariant. Calibration: - BP-trained, late training: <5x per block - BP ViT, early epochs (cls token resolving): 13-25x max - DFA-trained ResMLP/ViT: 100-4000x per block Threshold raised from 10 to 50 to sit cleanly between healthy-early- training (max 25) and failure-regime (min 100). Re-verifications: - smoke test (BP/DFA/EP): all 3 verdicts unchanged - random init (3 seeds): trustworthy on all 3 - 5-method audit table single-seed: identical verdicts - decision-utility ablation: identical (still 0/5 by S1, 3/5 by S_full) - temporal evolution 3-seed: (b) now fires first at ep 3-4, (a) at ep 8-11. Both well before training ends. The 'protocol fires ~92 epochs early' story still holds. - ViT temporal evolution: BP no longer false-fires; DFA fires (a) ep 1, (b) ep 3 — protocol works on the second architecture.
2026-04-07Temporal evolution 3-seed: protocol fires at DFA epoch 3-4 on all seedsYurenHao0426
s42: (a)+(b) fire at epoch 4, DFA final acc 0.3076 s123: (a)+(b) fire at epoch 4, DFA final acc 0.3203 s456: (a)+(b) fire at epoch 3, DFA final acc 0.2998 BP never fires on any seed (final acc 0.61-0.63). The 'protocol catches it 96 epochs early' finding is fully reproducible across seeds.
2026-04-07Add temporal diagnostic evolution: protocol fires at epoch 4 of DFAYurenHao0426
Replays per-epoch logged data from results/snapshot_evolution_v2/ through the protocol thresholds. Result: diagnostics (a) ||h_l|| explosion AND (b) ||g_L|| at floor BOTH first fire at epoch 4 of DFA training. At that point, DFA test acc is 0.308 — its final value at epoch 100 is also 0.308. The protocol could have walked back the headline 96 epochs before training finished. DFA's gamma hovers at 0.087-0.107 for all 100 epochs. A reviewer looking at acc+gamma would conclude 'DFA is hovering at 31% acc with ~0.10 alignment, both reasonable'. Wrong on both counts. BP never fires any diagnostic at any epoch. Stays bounded at ||h_L||~200, ||g_L||~3-5e-5, accuracy climbs to 0.61. This is the temporal validation of decision utility: the protocol catches the pathology AS IT HAPPENS, not just retrospectively.
2026-04-07Audit table extension to 3 seeds (s42/s123/s456)YurenHao0426
3 seeds × 5 methods × 4 diagnostics = 60 measurements. Key reproducibility findings: - BP: trustworthy on all 3 seeds (acc 0.61-0.62, h_L ~200, g_L ~3-4e-4) - EP: trustworthy on all 3 seeds (acc 0.29-0.36, h_L 3-8e3, g_L ~1e-4) - DFA, SB, CB: walked back on all 3 seeds × all 3 of (a)/(b)/(d) Diagnostic (c) is bimodal across seeds — confirms the prior memory finding: - DFA s42=0.047 (noise), s123=0.436 (drift), s456=-0.005 (noise) - SB s42=0.992 (drift), s123=0.561 (drift), s456=0.035 (noise) - CB s42=0.352 (drift), s123=0.250 (~edge), s456=0.518 (drift) (c) catches different methods on different seeds. (a)/(b)/(d) catch all 3 failing methods on all 3 seeds — robust binary detection.
2026-04-07Add protocol decision-utility ablation tableYurenHao0426
Builds on the 5-method audit JSON. For each method, evaluates 7 reporting strategies (S0=acc only, S1=+Γ field standard, S2-S5=+single diagnostic, S_full=full protocol), and emits the verdict each strategy would have reached. Result: 3 of 5 methods (DFA/SB/CB) are walked back by S_full but NOT by S1. Each of (a)scale, (b)floor, (d)frozen is independently sufficient for binary detection of those 3 failures. Diagnostic (c)stability adds sub-mode discrimination (drift vs noise) but not new positive detections. This is the §3 protocol decision-utility evidence.
2026-04-07Add audit table example: protocol applied to BP/DFA/SB/CB/EPYurenHao0426
5-method audit table on 4-block d=256 ResMLP CIFAR-10 seed 42: - BP: trustworthy (acc 0.615, h_L=2e2, g_L=4e-4, stab 0.099) - DFA: walked back via (a)+(b)+(d) — h_L=4e8, g_L=4e-9, undercuts frozen - State Bridge: walked back via all 4 diagnostics — stability 0.992 is the cleanest possible drift-dominated case - Credit Bridge: walked back via all 4 — stability 0.352, also drift mode - EP: trustworthy (acc 0.359, h_L=3e3, g_L=2e-4, stab -0.036) — paper's internal control case This is the §2 audit evidence for the main-track paper. Confirms that standard headline acc + Γ silently fails on 3 of 5 methods on this architecture, while the 4-diagnostic protocol catches all three.
2026-04-03Add 5 extra seeds to synthetic cross-state distance (now 10 seeds for all ↵YurenHao0426
methods) BP/DFA/SB/CB: added seeds 2048,3000,4000,5000,6000 (L=4 only, all 3 alphas) Total: 1290 rows (was 990) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Recompute all CNN diagnostics with fixed BP gradient flowYurenHao0426
CNN CIFAR-10 (5 seeds, fixed Gamma): BP: acc=86.8%, Gamma=0.970, rho=0.603 DFA: acc=56.7%, Gamma=0.896, rho=0.061 EP: acc=50.6%, Gamma=0.484, rho=0.450 SB: acc=63.3%, Gamma=1.000 (BP self-cos, feedback nets not saved), rho=0.601 CB: acc=31.8%, Gamma=1.000 (BP self-cos), rho=0.226 DFA Gamma=0.896 is notably high — CNN DFA credit aligns well with BP gradients. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Complete EP data: 10-seed synthetic + 6-seed CIFAR persample + cross-stateYurenHao0426
EP synthetic: 30 JSONs + 30 checkpoints (10 seeds × 3α) EP CIFAR persample: 6 seeds × 4 layers × 256 samples = 6144 rows added Synth cross-state: 150 EP rows added (990 total) cifar_persample_all.csv: 30720 rows (was 24576, +6144 EP) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03EP synthetic 10 seeds complete: 30 JSONs + 30 checkpoints + cross-state distanceYurenHao0426
Updated synth_cross_state_distance.csv with 150 EP rows (990 total). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP cross-state distance for CIFAR + verify CNN summaryYurenHao0426
EP CIFAR d_BP: L0=2.0×, L4=26.7× (much closer to BP than DFA=162×/2.5M×) EP synthetic: no checkpoints saved (ep_synthetic.py didn't save .pt) CNN summary: 20 rows confirmed correct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP synthetic per-seed CSV + synthetic cross-state distanceYurenHao0426
EP synthetic: 15 rows (3α × 5 seeds) Synth cross-state: 840 rows (3α × 2L × 4 methods × 5 seeds × (L+1) layers) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add EP to BP gradient sparsity analysisYurenHao0426
EP CIFAR d=256: s(1e-6)=100%, mean_norm=1.41e-04 EP produces networks where ALL samples have non-zero BP gradients, unlike DFA (0.4%), SB (21%), CB (3%). EP is closer to BP (98.7%). Updated clean_sparsity_summary.csv (980 rows, now includes EP). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Add cross-method hidden state distance vs BPYurenHao0426
Non-BP methods produce radically different representations: DFA L0: 162×, L4: 2.5M× relative to BP hidden norms SB L0: 3.2×, L4: 1.1M× CB L0: 59×, L4: 1.4M× (BP vs itself = 0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03Update all EP results with sign-corrected creditYurenHao0426
EP Synthetic (fixed): Gamma=+0.13~0.20, rho=+0.25 EP CIFAR d=256: Gamma=+0.007, rho=+0.051 EP CIFAR d=512: Gamma=+0.000, rho=-0.002 EP CNN: Gamma=+0.248, rho=+0.492 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Recompute EP CIFAR d=256 diagnostics with sign fixYurenHao0426
EP d=256 (5 seeds): acc=31.9%, Gamma=+0.007 (was -0.13), rho=+0.051 (was -0.037) Sign correction: -(h_nudge - h_free)/β aligns EP credit with BP gradient direction. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP synthetic results: 15 JSONs (3α × 5 seeds)YurenHao0426
EP synthetic: acc high (92-96%) but Gamma negative (-0.13 to -0.20), rho=-0.25 EP credit direction may be inverted or diagnostics have issue. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add CNN SB+CB results (5 seeds each), update summary CSVYurenHao0426
CNN CIFAR-10 (5 seeds): BP: 86.8%±0.3%, Gamma=0.238, rho=0.250 DFA: 56.7%±2.0%, Gamma=0.216, rho=0.017 SB: 63.3%±0.5%, Gamma=0.045, rho=0.298 CB: 31.8%±6.2%, Gamma=0.013, rho=0.033 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Fix and recompute GELU ablation Gamma from checkpointsYurenHao0426
ReLU MLP (L=4 d=256): BP: acc=61.1%, Gamma=1.000, rho=0.998 DFA: acc=30.7%, Gamma=0.104, rho=-0.001 SB: acc=15.5%, Gamma=0.300, rho=0.159 CB: acc=28.7%, Gamma=0.298, rho=0.007 Note: SB/CB Gamma uses BP gradient as proxy (feedback nets not checkpointed). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add summary CSVs for EP, GELU ablation, CNNYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02Add EP baseline (5 seeds), GELU ablation (20 runs), CNN BP+DFA (10 runs)YurenHao0426
EP (L=4 d=256): acc≈30%, Gamma≈0, rho≈0 — EP credit signal weak on feedforward MLP GELU ablation (ReLU variant): 4 methods × 5 seeds complete CNN BP+DFA: 5 seeds each, BP + DFA on SmallCNN Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add d=512 support sparsity: 20 JSONs + summary CSVYurenHao0426
BP: s(1e-6)=92.7%, norm=2.70e-04, r_inf=0.159, PR=0.300 DFA: s(1e-6)=0.1%, norm=5.31e-08 SB: s(1e-6)=20.3%, norm=2.33e-06 CB: s(1e-6)=1.2%, norm=9.88e-08 Same pattern as d=256, confirming width-independence of the sparsity gap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add missing bp_s456.json for CIFAR d=512 (rerun after SIGTERM)YurenHao0426
bp s=456: acc=0.5999, rho=0.9881, nse=0.4764 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add CIFAR L=4 d=512 confirmatory: 4 methods × 5 seeds with checkpointsYurenHao0426
BP: 60.6%±0.3%, rho=0.989 DFA: 30.8%±0.5%, rho=0.003 State Bridge: 21.2%±3.7%, rho=0.119 Credit Bridge: 30.1%±0.5%, rho=0.002 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add P3 protocol panel: method ranking across 5 protocol slicesYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add per-sample gradient stats: 24576 rows (256 samples × 4 layers × 4 ↵YurenHao0426
methods × 6 seeds) Columns: method, seed, layer, sample_id, grad_norm, log10_grad_norm, r_inf, pr, hoyer, topk1, topk5 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean_sparsity_summary.csv: 960 rows aggregated from 168 JSONsYurenHao0426
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean sparsity results: 168 JSONs from independent processes on GPU 1YurenHao0426
CIFAR: 24 JSONs (4 methods × 6 seeds), BP s(1e-6)=98% confirmed Synthetic: 144 JSONs (4 methods × 6 seeds × 3 alphas × 2 depths) All data reliable — each method+seed in separate Python process. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add clean gradient check: independent Python process per method, GPU 1YurenHao0426
Clean results (each method in fresh Python process): BP: mean_norm=2.58e-04, s(1e-6)=98% — CONFIRMED DFA: layer 0 = 2.86e-07 (1.2%), layers 1-3 ≈ 2.4e-09 (0%) SB: layer 0 = 6.13e-06 (86%), layers 1-3 ≈ 1e-09 (0%) CB: layer 0 = 6.33e-07 (18%), layers 1-3 ≈ 5e-10 (0%) Method A (autograd.grad) and Method B (retain_grad) give identical results. Previous 1e-12 results were caused by Python process state pollution in combined scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add element-wise gradient concentration analysis (CPU, from checkpoints)YurenHao0426
BP gradients are relatively uniform: top1%=7.1%, PR=0.327, eff_dim=0.632 DFA gradients extremely concentrated: top1%=40.6%, PR=0.089, eff_dim=0.272 SB/CB intermediate: top1%=17-21%, PR=0.14-0.17 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add confirmatory supplement: T1-T4 from checkpoints (no retraining)YurenHao0426
WARNING: All methods (including BP) show near-zero BP hidden gradients (~1e-12-1e-14) when computed via manual forward with detached hidden states. This is inconsistent with the earlier first-priority analysis which showed BP at 2.86e-04. Investigation needed. T1: 40 rows (4 methods × 10 seeds) - full metrics T2: 800 rows (support sparsity, 5 thresholds × 4 methods × 10 seeds × 4 layers) T3: 48 rows (gradient norm distributions, 3 seeds × 4 methods × 4 layers) T4: 100 rows (active-subset Gamma, 5 thresholds × 2 methods × 10 seeds) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add extended sparsity analysis: A4 per-layer, B1 snapshots, B2 active ↵YurenHao0426
subset, C1/C2 A4: Per-layer support — DFA/SB/CB layers 1-3 have 0% support at τ=1e-6 Only BP has ~95% support; only SB layer 0 has 53% B1: Snapshot evolution — old snapshot checkpoints have near-zero grads (data issue) B2: Active subset — with τ=1e-6, no active samples for non-BP methods C1: Active vs inactive cosine — only inactive subset exists for non-BP C2: Energy concentration — near-zero for non-BP methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Add BP support sparsity analysis: threshold sweep + gradient histogramsYurenHao0426
A1 Synthetic: all methods have >93% support at τ=1e-6 (gradients rarely zero) A2 CIFAR: massive gap — BP 98.4% vs DFA 0.4% vs SB 21% vs CB 3% DFA-trained CIFAR networks have near-zero BP gradients for 99.6% of samples This explains why Gamma is unreliable for CIFAR non-BP methods Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01Recompute BP and DFA Gamma with near-zero gradient filteringYurenHao0426
BP Gamma: raw~0.99, filtered=1.000 (confirms self-cosine artifact from zero grads) DFA Gamma (synth): raw~0.01-0.16, filtered~0.01-0.17 (minimal filtering effect) DFA Gamma (CIFAR): raw=0.107, filtered=0.466 (99.7% samples have near-zero BP grad!) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31Update naive StateErr v3: L2 norm ratio formula, with checkpoints savedYurenHao0426
Formula: ||h_{L//2} - h_L||_2 / ||h_L||_2 (scalar L2 ratio) A1: 240 rows (3 alpha × 2 depth × 4 methods × 10 seeds) A2: 40 rows (4 methods including BP × 10 seeds) All model checkpoints saved in checkpoints_A1/ and checkpoints_A2/ Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31Add BP supplement for A2 CIFAR: 10 seeds with acc, Gamma, rho, naive_StateErrYurenHao0426
BP 10-seed results: acc=0.614±0.003, Gamma=1.0, rho=0.998 Appended to A2_cifar_state_vs_credit.csv and A2_naive_state_err.csv Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31Add naive state prediction baseline for A1 and A2YurenHao0426
A1: 240 rows (3 alpha × 2 depth × 4 methods × 10 seeds) A2: 30 rows (3 methods × 10 seeds) naive_StateErr = ||h_{L//2} - h_L|| / ||h_L|| Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30Add confirmatory paper experiments: A1-A4, all 10 seeds completeYurenHao0426
A1: Synthetic nonlinearity ladder (240 rows: 3 alpha × 2 depth × 4 methods × 10 seeds) A2: CIFAR state-vs-credit counterexample (30 rows: 3 methods × 10 seeds) A3: Frozen vs online dissociation (60 rows: 2 regimes × 3 methods × 10 seeds) A4: Protocol dependence panel (82 rows: assembled from existing results) All experiments ran on GPU 3. Total runtime: ~20 hours. CSVs in results/confirmatory/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23Add final report, plots, experiment guide, and complete NOTE.mdYurenHao0426
All experiments complete: - Toy LQ: credit bridge matches state bridge (~0.94 costate cosine) - CIFAR-10: credit bridge (29.6%) comparable to DFA (30.0%), both beat state bridge (18.5%) - State bridge confirms core hypothesis: perfect state prediction != useful credit - Terminal gradient matching is essential for credit bridge
2026-03-23Initial implementation: all models, methods, toy and CIFAR experimentsYurenHao0426
Debug phase. Toy LQ experiments (3 seeds) complete with terminal gradient matching. Credit bridge matches state bridge on linear system (~0.94 cosine). CIFAR experiments in progress.
generated by cgit v1.2.3 (git 2.25.1) at 2026-06-15 07:44:34 +0000