| Age | Commit message (Collapse) | Author |
|---|
|
3 seeds × 5 methods × 4 diagnostics = 60 measurements. Key reproducibility
findings:
- BP: trustworthy on all 3 seeds (acc 0.61-0.62, h_L ~200, g_L ~3-4e-4)
- EP: trustworthy on all 3 seeds (acc 0.29-0.36, h_L 3-8e3, g_L ~1e-4)
- DFA, SB, CB: walked back on all 3 seeds × all 3 of (a)/(b)/(d)
Diagnostic (c) is bimodal across seeds — confirms the prior memory finding:
- DFA s42=0.047 (noise), s123=0.436 (drift), s456=-0.005 (noise)
- SB s42=0.992 (drift), s123=0.561 (drift), s456=0.035 (noise)
- CB s42=0.352 (drift), s123=0.250 (~edge), s456=0.518 (drift)
(c) catches different methods on different seeds. (a)/(b)/(d) catch all 3
failing methods on all 3 seeds — robust binary detection.
|
|
Builds on the 5-method audit JSON. For each method, evaluates 7 reporting
strategies (S0=acc only, S1=+Γ field standard, S2-S5=+single diagnostic,
S_full=full protocol), and emits the verdict each strategy would have
reached.
Result: 3 of 5 methods (DFA/SB/CB) are walked back by S_full but NOT by S1.
Each of (a)scale, (b)floor, (d)frozen is independently sufficient for
binary detection of those 3 failures. Diagnostic (c)stability adds
sub-mode discrimination (drift vs noise) but not new positive detections.
This is the §3 protocol decision-utility evidence.
|
|
5-method audit table on 4-block d=256 ResMLP CIFAR-10 seed 42:
- BP: trustworthy (acc 0.615, h_L=2e2, g_L=4e-4, stab 0.099)
- DFA: walked back via (a)+(b)+(d) — h_L=4e8, g_L=4e-9, undercuts frozen
- State Bridge: walked back via all 4 diagnostics — stability 0.992 is the
cleanest possible drift-dominated case
- Credit Bridge: walked back via all 4 — stability 0.352, also drift mode
- EP: trustworthy (acc 0.359, h_L=3e3, g_L=2e-4, stab -0.036) — paper's
internal control case
This is the §2 audit evidence for the main-track paper. Confirms that
standard headline acc + Γ silently fails on 3 of 5 methods on this
architecture, while the 4-diagnostic protocol catches all three.
|
|
Codex round 15 #1 priority for the E&D-track paper:
- protocol/protocol.py: 4 diagnostics (residual norms, BP grad norms,
cross-batch direction stability, and a frozen-baseline comparator)
- protocol/report.py: DiagnosticReport with per-diagnostic verdicts and
pretty-printer
- protocol/smoke_test.py: validates BP/DFA/EP checkpoints produce the
expected verdicts (BP/EP trustworthy; DFA walked back via residual
explosion + BP grad at floor)
- protocol/README.md: usage, audit cases, threshold rationale
- protocol/CHECKLIST.md: 6 evaluation pipeline pitfalls (norm(-1),
cosine_similarity eps clamp, fp16 underflow, Bs reproducibility,
aggregation, layer-0 dominance)
- protocol/REPORTING_TEMPLATE.md: per-method fillable form for FA papers
|
