1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
|
# SDIL — results log (seed 0 unless noted; runs on timan107 GTX-1080 / ep_pascal)
## Method
SDIL (Somato-Dendritic Innovation Learning), a local non-backprop rule inspired by Harnett 2026.
Per hidden layer l:
- forward: `u_l = W_l h_{l-1}`, `h_l = φ(u_l)`
- apical feedback `a_l = A_l c` (c = output error e = softmax−onehot, broadcast)
- predictor `ĥ_l = P_l h_l`; teaching signal (innovation) `r_l = a_l − ĥ_l`
- three-factor update `ΔW_l = η (r_l ⊙ φ'(u_l)) h_{l-1}^T`
- `A_l` LEARNED by amortized node perturbation (q_l ≈ −∇_{h_l}L, forward-only, no weight transport)
- `P_l` learned on neutral (c=0) periods only, so it strips the soma-predictable nuisance
without eating the teaching signal.
Distinct from the failed `~/sdrn` (fixed-random feedback + post-hoc residualization) and from
Dual Prop / DFA / FA: the feedback pathway is *learned by causal perturbation*, so the residual is
aligned by construction rather than being a random projection.
## Critical fix
Node-perturbation estimator must divide by σ, not σ² (with δ=σξ, ĝ = δ·ΔL/σ² = ξ·ΔL/σ ⇒ ‖q‖≈‖∇L‖).
The σ² bug made the teaching signal ~100× too large ⇒ effective LR ~100× ⇒ SDIL stalled. Cosine is
magnitude-invariant so alignment looked fine while optimization blew up. Fixing it turned SDIL from
"worse than DFA everywhere" to "matches DFA on MNIST, beats it on FMNIST/CIFAR". (Codex independently
flagged the same σ-scale issue.)
## Accuracy (test %, seed 0)
| task | BP | DFA | SDIL |
|------------------------------|-----------|-------|---------------|
| MNIST d3 / d5 / d7 / d10 | 98.3 | 97.3 / 97.2 / 97.1 / 96.8 | 97.4 / 97.3 / 97.1 / 97.1 |
| FashionMNIST d3 / d5 / d7 / d10 | 88–89 | 86.3 / 87.1 / 86.2 / 86.9 | 88.4 / 87.6 / 87.3 / 88.1 |
| CIFAR-10 (no-BN residual) d5/10/20 | 44.2 / 44.8 / 45.5 | 40.6 / 39.9 / 37.4 | 42.4 / 42.6 / 42.4 |
SDIL matches DFA on easy MNIST and **beats DFA on FashionMNIST (+1–2 pts) and CIFAR (+2 to +5 pts)**.
On CIFAR the SDIL−DFA gap GROWS with depth (+1.8 → +2.7 → +5.0 at d5/10/20): DFA degrades with depth,
SDIL holds.
## Credit assignment / depth utility — the mechanistic result
Per-layer alignment cos(r_l, −∇_{h_l}L), early-third average, CIFAR no-BN residual:
| depth | DFA early-align | SDIL early-align |
|-------|-----------------|------------------|
| 5 | 0.55 | 0.83 |
| 10 | 0.36 | 0.86 |
| 20 | 0.25 | 0.88 |
| 30 | 0.19 | (running) |
DFA's teaching signal to early/deep layers collapses toward random (~0.05–0.19) as depth grows;
SDIL stays ~0.85 and is depth-invariant. This is the "does credit reach early layers" metric that
DFA fails and SDIL passes — the core advantage.
## Baselines (`sdil/local_baselines.py`)
- FA (Lillicrap, sequential random feedback): WORKS — 94.7% MNIST (1 epoch). Included.
- DFA: SDIL with fixed-random A. Included (primary comparison).
- EP (Equilibrium Propagation, Scellier & Bengio 2017): WORKS — 87% MNIST on an 8k subset with
β=0.5, feedforward warm-start settling; ~95%+ expected on full data. Included.
- PEPITA (forward-only error modulation): implemented, undertuned (~19%). Needs work.
- Forward-Forward (Hinton): implemented; per-layer goodness separates but inference is off. Needs work.
## How to run
`experiments/run.py --mode {bp,dfa,sdil} --dataset {mnist,fmnist,cifar10} --depth D --residual {0,1} --act {tanh,gelu,silu,relu}`
Batteries: `experiments/run_v2.sh <ds> "<depths>" <res> <act> "<seeds>" <ep> <pfx>`.
Analysis: `experiments/analyze_depth.py {depth,deepres}` → tables + figures in `results/figs/`.
Smoke/mechanism checks: `experiments/smoke.py`.
## Open items
- Complete CIFAR SDIL d30 (running) + seeds 1,2 (running) for mean±std.
- Codex-suggested boosts (not yet applied): separate normalized feedback direction from a learned
per-layer scalar gain; Adam-like second-moment on local updates; forward-only gradient-norm gain
calibration. Current results already beat DFA without these.
- GRAPE (ICLR2026) read for DFA-failure regimes only; treated as unverified, not a baseline.
|
