diff options
| -rw-r--r-- | RESULTS.md | 28 | ||||
| -rw-r--r-- | ROADMAP.md | 9 |
2 files changed, 37 insertions, 0 deletions
@@ -298,6 +298,34 @@ also informative: d2 can optimize worse than d1, and d6 is less stable than d3/d therefore about a verified useful-depth pair, not monotonic improvement from adding arbitrary blocks. +## C2 local-rule panel (validation only) + +The next frozen panel crossed BP, FA, DFA, and no-traffic SDIL at d1/d4 over three task seeds and +five student seeds (120 runs). SDIL used the C3-selected `K=1/every=4` calibration schedule. All +runs came from clean revision `51301f6`, evaluated validation only after training, and computed +alignment and lesion diagnostics on the fixed training prefix. The independent synthetic test +sets remain unevaluated. + +| method | depth | validation acc (%) | d1-to-d4 gain | d4 lesion drop | +|:-------|------:|-------------------:|--------------:|---------------:| +| BP | 1 | 62.757 ± 8.111 | — | — | +| BP | 4 | 92.453 ± 8.834 | +29.697 | +30.157 | +| FA | 1 | 58.600 ± 4.839 | — | — | +| FA | 4 | 77.147 ± 16.082 | +18.547 | +16.853 | +| DFA | 1 | 55.517 ± 11.161 | — | — | +| DFA | 4 | 73.193 ± 13.987 | +17.677 | +22.650 | +| SDIL | 1 | 65.347 ± 6.516 | — | — | +| SDIL | 4 | 71.587 ± 15.307 | +6.240 | +12.787 | + +SDIL recovers only `21.0%` of BP's depth gain, versus `62.5%` for FA and `59.5%` for DFA, and its +d4 endpoint trails the stronger comparator by `5.560` points. Its final-third lesion reduces +accuracy in 11/15 runs, so the model often uses the late block even though its learned credit is +poor. **The C2 local validation gate fails and no test confirmation is allowed.** Additional +validation scratch found that K=4 or K=16 perturbations, more frequent events, larger `eta_A`, and +normalized local deltas do not remove the basin variability. This points to a representational +limitation of the fixed linear `A_l c` vectorizer: on a compositional target, the correct hidden +direction depends on neural state as well as the two-dimensional output error. + ## Paper-faithful non-backprop baselines `PETITE` in the initial request is treated as **PEPITA**. Implementations and hyperparameters were @@ -116,6 +116,15 @@ may run only if the original C2 gate holds on validation and the d4 SDIL lesion 2 points on average with at least 10 of 15 paired runs harmed. Test confirmation retrains on the same 8,000-example training subset and evaluates each task seed's untouched 5,000 examples once. +**Local-rule validation status: failed on 2026-07-22; test remains unevaluated.** Across the 15 +task/model pairs, BP gained `+29.697` points from d1 to d4 while SDIL gained only `+6.240` +(`21.0%` recovery). FA and DFA recovered `62.5%` and `59.5%`; SDIL's d4 endpoint was `5.560` +points below the stronger comparator. The SDIL lesion criterion itself passed (`+12.787`-point +mean drop; 11/15 harmed), so the failure is credit/optimization rather than unused depth. The +fixed linear output-error vectorizer `A_l c` is now the leading limitation: increasing perturbation +directions/frequency or simple gain normalization did not stabilize scratch validation. No C2 +test panel may run unless a separately declared mechanism change passes validation. + ### C3. Causal calibration is genuinely amortized **Status: passed on 2026-07-22.** The frozen five-seed confirmation retained 112.9% of the K16/e4 |