docs: report audited baseline comparisons

1 files changed, 113 insertions, 56 deletions

diff --git a/RESULTS.md b/RESULTS.md
index d1a7c75..4e3b001 100644
--- a/RESULTS.md
+++ b/RESULTS.md
@@ -1,9 +1,9 @@
-# SDIL — results log (seed 0 unless noted; runs on timan107 GTX-1080 / ep_pascal)
+# SDIL — results log (audited sections report n; timan107 GTX-1080 / ep_pascal)
 
 ## 2026-07-21 audit and version boundary
 
 Git was initialized after inheriting the project. The original code/results are preserved at
-`215acff`. Four audit findings changed the canonical implementation:
+`215acff`. The following audit findings changed the canonical implementation:
 
 1. `49b86d6`: residual-block local updates now include the branch Jacobian
    `res_alpha=1/sqrt(depth)`. The omission preserved cosine direction but inflated update norms by
@@ -16,6 +16,11 @@ Git was initialized after inheriting the project. The original code/results are
 4. `0f107d1` / `e0bfdfd`: simultaneous hidden-layer perturbations reduce calibration from
    quadratic to linear depth work; batching directions makes the reduction visible in GPU wall
    time. `66e3034` fixes diagnostics so raw-apical ablations report the signal actually used.
+5. `2f3f44d`: the raw-apical control can be matched per sample to the innovation norm. This
+   separates subtraction/direction from a trivial update-magnitude explanation.
+6. `6093fe5` / `d6a00aa`: FA, PEPITA, Forward-Forward, and EP were reimplemented against their
+   papers and reference code. EP uses persistent free particles; deep PEPITA requires an explicit
+   learning rate; FA now logs the signal actually transported through its fixed feedback path.
 
 All older `deepres_*` files predate item 1. Their alignment directions remain informative, but
 their accuracy and stability should not be pooled with the corrected runs below.
@@ -46,43 +51,52 @@ magnitude-invariant so alignment looked fine while optimization blew up. Fixing
 "worse than DFA everywhere" to "matches DFA on MNIST, beats it on FMNIST/CIFAR". (Codex independently
 flagged the same σ-scale issue.)
 
-## Corrected narrow depth pilot (CIFAR-10, width 64, 5 epochs, seed 0)
-
-| depth | method | test acc | early-third alignment | wall time |
-|------:|--------|---------:|----------------------:|----------:|
-| 30 | BP | 41.31% | — | 29.4 s |
-| 30 | DFA | 37.31% | 0.084 | 19.4 s |
-| 30 | SDIL layerwise | 42.01% | 0.371 | 339.5 s |
-| 30 | SDIL multiplexed+batched | 41.46% | 0.370 | 47.3 s |
-| 60 | BP | 42.93% | — | 63.7 s |
-| 60 | DFA | 37.31% | 0.041 | 41.1 s |
-| 60 | SDIL layerwise | 40.79% | 0.328 | 1238.5 s |
-| 60 | SDIL multiplexed+batched | 41.10% | 0.332 | 90.3 s |
-
-Narrowing the model exposed the performance difference that width-256 flattened CIFAR had hidden.
-DFA alignment halves from depth 30 to 60, while both SDIL estimators remain near 0.33–0.37. Batched
-multiplexing preserves alignment while accelerating SDIL 7.2x at d30 and 13.7x at d60. Its wall
-growth from d30 to d60 is 1.91x, versus 3.65x for layerwise calibration. These are pilot, single-seed
-results; the ~0.5-point layerwise/multiplexed accuracy difference is not yet statistically resolved.
-
-## Harnett-specific residualization (MNIST, 15 epochs, seed 0)
-
-| nuisance rho | raw apical acc | residual acc | raw teaching alignment | residual alignment |
-|-------------:|---------------:|-------------:|-----------------------:|-------------------:|
-| 0.00 | 97.31% | 97.31% | 0.138 | 0.138 |
-| 0.02 | 96.90% | 97.27% | 0.096 | 0.156 |
-| 0.05 | 96.48% | 97.24% | 0.092 | 0.160 |
-| 0.10 | 95.87% | 97.32% | 0.068 | 0.140 |
-| 0.20 | 95.60% | 97.31% | 0.103 | 0.147 |
-| 0.50 | 8.92% | 97.18% | 0.000 | 0.149 |
-| 1.00 | 8.92% | 97.29% | 0.000 | 0.140 |
-| 2.00 | 9.82% | 97.31% | 0.000 | 0.121 |
-
-At zero nuisance, subtraction is a no-op. As normal soma-dendrite coupling grows, raw feedback
-degrades continuously and eventually enters a self-reinforcing Hebbian failure, while the learned
-innovation remains stable. The low-rho sweep is important: it shows a graded 0.37→1.71 point gap
-before the strong-contamination phase transition, rather than only an artificial chance-level
-contrast.
+## Verified narrow depth scaling (CIFAR-10, width 64, 5 epochs, five seeds)
+
+Mean ± sample SD. Alignment is the mean per-sample cosine over the earliest third of hidden layers
+at the final checkpoint. SDIL uses simultaneous perturbations with 16 batched directions.
+
+| depth | method | test acc (%) | early-third alignment | wall time (s) |
+|------:|:-------|-------------:|----------------------:|--------------:|
+| 30 | BP | 41.908 ± 0.697 | — | 29.8 ± 1.3 |
+| 30 | FA | 38.158 ± 0.517 | 0.593 ± 0.014 | 26.9 ± 1.9 |
+| 30 | DFA | 36.988 ± 0.524 | 0.087 ± 0.007 | 23.4 ± 0.8 |
+| 30 | SDIL | 41.198 ± 0.523 | 0.366 ± 0.007 | 47.8 ± 1.4 |
+| 60 | BP | 41.624 ± 0.781 | — | 58.4 ± 1.0 |
+| 60 | FA | 38.428 ± 0.387 | 0.577 ± 0.011 | 49.8 ± 1.8 |
+| 60 | DFA | 36.898 ± 0.444 | 0.047 ± 0.005 | 47.6 ± 5.6 |
+| 60 | SDIL | 41.246 ± 0.323 | 0.336 ± 0.013 | 96.4 ± 12.4 |
+
+The paired SDIL−DFA accuracy gap is `4.210 ± 0.426` points at d30 and
+`4.348 ± 0.554` at d60. SDIL wall time grows 2.02x when depth doubles, so the batched simultaneous
+calibration removes the former superlinear depth cost. Its accuracy is depth-stable and remains
+within 0.71 points of BP at d30 and 0.38 points at d60. Flattened CIFAR itself is depth-flat here, so
+these runs establish preservation rather than positive accuracy scaling.
+
+DFA's early-layer direction nearly halves with depth. Residual FA does **not** show that failure:
+its identity skips transport most of the state gradient exactly, leaving cosine near 0.58–0.59.
+Nevertheless FA plateaus around 38.3%, roughly three points below SDIL. Thus the supported claim is
+not “all FA signals lose directional alignment”; direction, signal scale, serial transport, and
+optimization efficiency must be separated.
+
+## Harnett-specific residualization (MNIST, d3/w256, 15 epochs, five seeds)
+
+The matched-raw control preserves the raw apical direction but rescales every sample to the
+innovation's norm.
+
+| nuisance rho | raw apical acc (%) | matched raw acc (%) | residual acc (%) |
+|-------------:|-------------------:|--------------------:|-----------------:|
+| 0.00 | 97.450 ± 0.046 | 97.450 ± 0.046 | 97.450 ± 0.046 |
+| 0.05 | 96.438 ± 0.077 | 96.980 ± 0.148 | 97.382 ± 0.154 |
+| 0.20 | 95.512 ± 0.150 | 95.974 ± 0.144 | 97.370 ± 0.083 |
+| 0.50 | 10.380 ± 0.548 | 10.310 ± 0.633 | 97.348 ± 0.133 |
+
+At zero nuisance all three signals are identical. As normal soma-dendrite coupling grows, raw
+feedback degrades and eventually reaches chance; residual learning stays at 97.35–97.45%. Matching
+the raw signal's norm recovers only part of the low-rho loss and does nothing at rho=0.5. The strong
+result therefore depends on subtracting the soma-predictable component, not merely clipping or
+normalizing an oversized update. This is the most direct evidence that the Harnett-specific
+innovation operation is algorithmically necessary.
 
 ## Legacy accuracy (pre-audit implementation)
 | task                         | BP        | DFA   | SDIL          |
@@ -108,26 +122,69 @@ DFA's teaching signal to early/deep layers collapses toward random (~0.05–0.19
 SDIL stays ~0.85 and is depth-invariant. This is the "does credit reach early layers" metric that
 DFA fails and SDIL passes — the core advantage.
 
-## Baselines (`sdil/local_baselines.py`)
-- FA (Lillicrap, sequential random feedback): WORKS — 94.7% MNIST (1 epoch). Included.
-- DFA: SDIL with fixed-random A. Included (primary comparison).
-- EP (Equilibrium Propagation, Scellier & Bengio 2017): WORKS — 87% MNIST on an 8k subset with
-  β=0.5, feedforward warm-start settling; ~95%+ expected on full data. Included.
-- PEPITA (forward-only error modulation): implemented, undertuned (~19%). Needs work.
-- Forward-Forward (Hinton): implemented; per-layer goodness separates but inference is off. Needs work.
+## Paper-faithful non-backprop baselines
+
+`PETITE` in the initial request is treated as **PEPITA**. Implementations and hyperparameters were
+checked against the [FA paper](https://www.nature.com/articles/ncomms13276),
+[PEPITA paper](https://proceedings.mlr.press/v162/dellaferrera22a.html) and
+[author code](https://github.com/GiorgiaD/PEPITA),
+[Forward-Forward paper](https://www.cs.toronto.edu/~hinton/absps/FFXfinal.pdf) and
+[reference implementation](https://github.com/mpezeshki/pytorch_forward_forward), and the
+[EP paper](https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2017.00024/full)
+and [author code](https://github.com/bscellier/Towards-a-Biologically-Plausible-Backprop).
+
+Native protocol results are not an equal-compute comparison:
+
+| method | architecture | budget | n | last test acc (%) | wall time (s) |
+|:-------|:-------------|:-------|--:|------------------:|--------------:|
+| PEPITA | d3/w256 | 60 epochs | 5 | 91.094 ± 0.214 | 89.2 ± 3.5 |
+| Forward-Forward | d3/w256 | 15 epochs/layer | 5 | 93.558 ± 0.232 | 68.2 ± 0.8 |
+| EP | d1/w500 | 25 epochs | 2 | 97.040 ± 0.269 | 839.8 ± 78.4 |
+
+PEPITA uses its two forward presentations, He-uniform initialization, shared 10% dropout mask, and
+an explicit deep-network learning rate of 0.001. Forward-Forward trains greedy local goodness layers
+and has no classifier readout. EP uses the original hard-sigmoid energy dynamics, random beta sign,
+20 free steps, 4 nudged steps, layer rates `[0.1, 0.05]`, and persistent free particles for the fixed
+first 50k examples. This corrected EP is the strongest baseline and is also by far the most expensive.
+
+### EP exact-architecture comparison
+
+All methods below use d1/w500, the same first 50k MNIST examples, and 25 epochs. EP retains its
+native raw-pixel energy protocol; BP/DFA/SDIL retain the shared z-score/tanh feedforward protocol.
+Forward parameter count is 397,510 for every row.
+
+| method | n | last test acc (%) | best test acc (%) | wall time (s) |
+|:-------|--:|------------------:|------------------:|--------------:|
+| BP | 2 | 98.190 ± 0.014 | 98.215 ± 0.007 | 15.1 ± 0.9 |
+| DFA | 2 | 97.620 ± 0.071 | 97.690 ± 0.099 | 9.9 ± 0.1 |
+| EP | 2 | 97.040 ± 0.269 | 97.220 ± 0.283 | 839.8 ± 78.4 |
+| SDIL | 2 | 97.510 ± 0.028 | 97.570 ± 0.028 | 15.4 ± 0.4 |
+
+SDIL is 0.47 points above EP on last-epoch mean and about 54.5x faster in wall time. This establishes
+that SDIL does not lose to the strongest baseline in the exact shallow architecture. It is **not**
+evidence of a unique scaling advantage: DFA is also strong on this easy d1 MNIST problem, and only
+two EP seeds have been run. At similar forward-parameter count but d3/w256, SDIL and EP are tied
+within noise (97.13% vs 97.04% last-epoch mean), again with a ~26x wall-time advantage for SDIL.
 
 ## How to run
 `experiments/run.py --mode {bp,dfa,sdil} --dataset {mnist,fmnist,cifar10} --depth D --residual {0,1} --act {tanh,gelu,silu,relu}`
 Batteries: `experiments/run_v2.sh <ds> "<depths>" <res> <act> "<seeds>" <ep> <pfx>`.
-Analysis: `experiments/analyze_depth.py {depth,deepres}` → tables + figures in `results/figs/`.
-Smoke/mechanism checks: `experiments/smoke.py`.
+Canonical local baselines: `experiments/baseline_sweep.sh`; original d1 EP:
+`experiments/ep_original_sweep.sh`; EP comparison: `experiments/ep_matched_sweep.sh`.
+Audited aggregate tables: `experiments/analyze_verified.py`.
+Smoke/mechanism checks: `experiments/smoke.py` and `experiments/baseline_smoke.py`.
 
 ## Open items
-- Repeat corrected width-64 d30/d60 and nuisance transition runs for seeds 1–4.
-- Compare multiplexed calibration at fixed loss-evaluation budgets and sweep directions (4/8/16/32).
-- Add a magnitude-matched or clipped raw-apical control to separate directional contamination from
-  simple update explosion at large nuisance.
-- Validate simultaneous perturbation on a genuinely depth-necessary compositional task and a CNN,
-  not only flattened CIFAR.
-- FA's residual-network implementation and the current EP prototype are not yet strong enough for a
-  publication-quality scaling comparison; do not overstate those baselines.
+
+- Compare simultaneous calibration at fixed loss-evaluation budgets and sweep directions
+  (4/8/16/32).
+- Validate SDIL on a genuinely depth-necessary compositional task and a CNN; flattened CIFAR does
+  not gain accuracy with depth.
+- Diagnose why residual FA has high directional cosine but lower accuracy: log update-norm ratios,
+  parameter-space cosine, and serial feedback latency/cost.
+- Increase EP and exact-architecture comparisons from two to five seeds. If compute permits, run the
+  author's d2 EP schedule (60 epochs, 150 free steps, 6 nudged steps) rather than inventing a cheap
+  deep-EP approximation.
+- Tune SDIL schedules using a held-out validation split, never the test trajectory, then freeze the
+  protocol before expanding the EP comparison.
+- Extend PEPITA and Forward-Forward beyond shallow MNIST before making any scaling claim about them.