From 2304e83086804f78cf752b6144683c6bb5f33002 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Wed, 22 Jul 2026 06:36:36 -0500 Subject: docs: track reviewer score trajectory --- REVIEW_SCORECARD.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++ ROADMAP.md | 4 ++++ 2 files changed, 67 insertions(+) create mode 100644 REVIEW_SCORECARD.md diff --git a/REVIEW_SCORECARD.md b/REVIEW_SCORECARD.md new file mode 100644 index 0000000..4cae7a1 --- /dev/null +++ b/REVIEW_SCORECARD.md @@ -0,0 +1,63 @@ +# ICLR-style reviewer scorecard + +This is a deliberately adversarial paper-level assessment, not an average of the best runs. It is +updated only when a frozen experiment is complete and audited. Development pilots, partial jobs, +and post-hoc subsets cannot raise the score. Failed gates remain visible and narrow the claim. + +## Scoring convention + +The overall score uses a 1--10 ICLR-like scale: 1 strong reject, 3 reject, 5 borderline reject, +6 borderline accept, 8 accept, and 10 strong accept. Component scores use 1--4. The overall score +is a reviewer judgment rather than the arithmetic mean of the components. Confidence uses 1--5. + +Every formal result report records: + +- the previous and new overall score; +- soundness, novelty, significance, empirical support, and reproducibility; +- which frozen evidence changed the assessment; +- the strongest remaining reviewer objections; +- whether the evidence is development, validation, or untouched confirmation. + +## Current paper-level assessment (2026-07-22) + +| Dimension | Score | Strict reviewer assessment | +|:--|--:|:--| +| Soundness | 3/4 | Theory, local-gradient checks, causal diagnostics, cost accounting, and frozen stop rules are unusually careful. The learned apical vectorizer remains an unresolved failure mode. | +| Novelty | 2/4 | Learned node-perturbation feedback is prior art. The defensible novelty is the per-cell innovation operation under mixed apical traffic, together with its causal and scaling analysis. | +| Significance | 3/4 | Near-flat performance over 12x depth while DFA alignment collapses is potentially important, but the current flattened-CIFAR task does not benefit from depth. | +| Empirical support | 2/4 | Five-depth scaling and the residualization ablation are strong. Useful-depth C2, broad endogenous C1, and oral-B gates failed; standard ResNet-scale confirmation is still pending. | +| Reproducibility | 4/4 | Code, exact provenance, seed panels, costs, failed branches, frozen selectors, and staged test-access rules are retained in git. | +| **Overall** | **5/10** | **Borderline reject: a strong core result without a completed standard-scale or useful-depth demonstration.** | +| Confidence | 4/5 | High confidence in the assessment because the positive and negative branches are both extensively audited. | + +### Evidence already carrying the paper + +- On flattened CIFAR-10, SDIL changes by only `-0.214 +/- 0.349` accuracy points from depth 5 to + depth 60, while DFA's early-layer alignment falls from `0.514` to `0.047`. +- Under strong soma-predictable apical traffic, raw and norm-matched raw learning fall to about + chance (`10.38%` and `10.31%`) while innovation learning retains `97.35%` accuracy. +- The local update has a proved descent condition and an explicit query/MAC/memory audit; direct + node perturbation isolates the learned vectorizer as the useful-depth bottleneck. + +### Objections currently preventing acceptance confidence + +1. The depth result is preservation on a depth-flat task, not evidence that SDIL uses added depth. +2. The amortized apical vectorizer failed the frozen useful-depth gate; direct causal targets work + but cost `68.4x` ordinary forward-equivalent work. +3. Innovation was not uniformly beneficial for arbitrary endogenous top-down traffic, so the + supported mechanism is narrower than the initial claim. +4. The frozen oral-B screen falsified the desired-velocity and Harnett error-derivative claims. +5. Native BurstCCN/Dual Prop endpoints and the standard CIFAR ResNet panel are not yet complete. + +## Score trajectory and prospective gates + +| Checkpoint | Overall | What changed | Remaining ceiling | +|:--|--:|:--|:--| +| Current audited package | 5 | Strong depth-preservation and residual-necessity evidence; negative gates retained | Standard useful scale is absent | +| Native baselines complete | pending | Can close fairness/completeness objections, but cannot by itself establish the main claim | Usually no automatic score increase | +| Oral-A A3 passes | pending | A full-data ResNet20 SDIL run is within 5 points of BP, beats DFA by 2, stays aligned, finite, and locally cost-audited | Plausible 6 if the evidence is clean | +| Oral-A A4 passes | pending | Five untouched seeds across ResNet20/32/56 pass the frozen accuracy, scaling, alignment, and Pareto gates | Plausible 7--8; oral discussion becomes credible | + +These are conditional reviewer forecasts, not promised scores. A failed stage leaves its negative +result in the record and can lower the score if it invalidates a current claim. Oral-B does not get +retroactively reopened by success on standard vision benchmarks. diff --git a/ROADMAP.md b/ROADMAP.md index 6efa182..b2a7185 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,6 +4,10 @@ This document predeclares the evidence gates used to decide which claims survive It is intentionally stricter than a list of experiments: a failed gate narrows the claim rather than triggering selective seed removal or post-hoc protocol changes. +Paper-level progress is tracked independently in `REVIEW_SCORECARD.md`. Its ICLR-style score may +change only after a frozen result is complete and audited; partial jobs and development pilots do +not count. + ## Order of work 1. **Accept bar:** establish a load-bearing innovation mechanism, useful-depth scaling, fair cost, -- cgit v1.2.3