path: root/THEORY.md

# SDIL theory and cost audit

This note states the minimum claims that can be proved for the implemented
estimators and innovation rule. It deliberately separates an unbiased neural
teaching direction from a useful parameter update: the completed experiments
show that a high hidden-state cosine can coexist with poor optimization.

## 1. Simultaneous multi-layer perturbation

Consider one example and collect the hidden-state gradients
`g_l = dL/dh_l` for hidden blocks `l=1,...,H`, where `g_l` has dimension
`d_l`. For one simultaneous Rademacher intervention, draw independent
`xi_l in {-1,+1}^{d_l}` and linearize the centered loss difference as

```text
D = sum_j <g_j, xi_j>.
```

The gradient and descent estimators for layer `l` are

```text
g_hat_l = xi_l D,        q_l = -g_hat_l.
```

Independence and `E[xi_l xi_l^T] = I` give

```text
E[g_hat_l] = g_l.
```

Because a Rademacher vector has deterministic squared norm `d_l`, the exact
linearized mean-squared error is

```text
E ||g_hat_l - g_l||^2
  = (d_l - 1) ||g_l||^2
    + d_l sum_{j != l} ||g_j||^2.
```

Averaging `K` independent directions divides this expression by `K`. The first
term is ordinary directional-estimator variance; the second is cross-layer
interference. Thus simultaneous perturbation is unbiased but its relative
variance grows with layer width and with the total gradient energy injected at
other layers. This predicts why K1 can work in an easy/depth-flat regime yet
fail to train a moving context vectorizer on a compositional task.

For comparison, layerwise Rademacher perturbation removes the cross-layer term.
The current layerwise implementation uses Gaussian directions, for which

```text
E ||g_hat_l - g_l||^2 = (d_l + 1) ||g_l||^2 / K.
```

Layerwise targets therefore buy variance reduction with depth-dependent replay
cost; they are not a free scalable replacement for simultaneous calibration.

### Finite perturbation bias

Suppose the third derivative of the loss has operator norm at most `M` in the
perturbation neighborhood, and let `D_total = sum_l d_l`. A centered antithetic
difference has directional error at most

```text
M sigma^2 ||xi||^3 / 6.
```

For simultaneous Rademacher directions this gives the conservative layerwise
bias bound

```text
||E[g_hat_l] - g_l||
  <= (M sigma^2 / 6) sqrt(d_l) D_total^(3/2).
```

The important dependence is `O(sigma^2)`, with a dimension-dependent constant.
Reducing K removes variance but not this bias; increasing width/depth without a
sigma audit is therefore unsafe.

## 2. What is sufficient for a descent step

Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let
`v` be the complete local update before multiplying by the learning rate:

```text
theta_next = theta + eta v.
```

If the loss is `beta`-smooth and

```text
c = <-p, v> / (||p|| ||v||) > 0,
```

then the descent lemma gives

```text
L(theta_next) - L(theta)
 <= -eta c ||p|| ||v|| + (beta eta^2 / 2) ||v||^2.
```

Consequently the update is guaranteed to descend only when

```text
0 < eta < 2 c ||p|| / (beta ||v||).
```

This separates three quantities that a neural-state cosine conflates:

1. direction (`c`);
2. gain (`||v|| / ||p||`);
3. curvature (`beta`).

For a local layer, multiplying a hidden teaching vector by the activation gain
and presynaptic activity maps it into parameter space. A positive
`cos(r_l, -g_l)` does not by itself imply a positive cosine after aggregating
examples, parameters, and layers, and even perfect parameter alignment can
increase loss if its gain makes the step exceed the bound. This is why future
diagnostics must include parameter-update cosine, norm ratio, and realized
single-step loss change rather than reporting hidden cosine alone.

## 3. Somato-dendritic innovation as conditional nuisance removal

Write apical activity during a task period as

```text
a = s + M h + epsilon,
```

where `s` is the instructional component, `M h` is ordinary traffic predictable
from soma, and `epsilon` is unpredictable noise. In neutral periods `s=0`.
Assume `Sigma_h = E[h h^T]` is nonsingular and `E[epsilon h^T]=0`. Population
least squares on neutral periods yields

```text
P_neutral = E[a h^T | neutral] Sigma_h^(-1) = M.
```

The task-period innovation is then

```text
r_neutral = a - P_neutral h = s + epsilon.
```

If `s`, `M h`, and `epsilon` are mutually uncorrelated, a squared-cosine/SNR
proxy improves from

```text
raw:       S / (S + T + E)
innovation:S / (S + E),
```

where `S`, `T`, and `E` are their expected squared norms. The gain is strict
whenever predictable traffic has nonzero power.

Task-period fitting has a different estimand. Let

```text
C = E[s h^T] Sigma_h^(-1).
```

Then `P_task = M + C` and

```text
r_task = s - C h + epsilon.
```

It removes the soma-predictable projection of the teaching signal itself. This
proves the identification role of neutral periods and also defines the failure
boundary: residualization cannot remove contextual traffic that remains
unpredictable from the chosen per-cell soma statistic, and it can erase credit
when the predictor is fit on task periods.

For the diagonal normalized-LMS predictor used in the code, an ideal noiseless
cell with neutral relation `a=m h` follows

```text
p_(t+1) = p_t + eta_P (m - p_t),
traffic residual power fraction = (1 - eta_P)^(2t).
```

The relevant timescale is therefore the product of learning rate and number of
neutral updates. Too few neutral updates leave traffic; task-period updates
converge to the wrong coefficient regardless of speed.

## 4. Hardware-independent complexity

Let `H` be the number of hidden layers, `K` the perturbation directions per
event, `e` the number of ordinary minibatches between events, `B` the batch
size, and `F_l` the affine work of layer `l`. Define `F=sum_l F_l` and suffix
work `S_l=sum_{j>l} F_j`.

| method | activity phases / passes | transport | scalar task-loss observations | extra affine work | main transient memory |
|:--|:--|:--|:--|:--|:--|
| BP | forward + reverse | exact transposed Jacobians | ordinary supervised loss | about one reverse pass | saved activations / graph |
| FA | forward + reverse-like serial feedback | fixed random feedback | ordinary loss | `O(F)` feedback work | activations + feedback states |
| DFA | forward + direct feedback | fixed random output maps | ordinary loss | `O(sum_l d_l d_out)` | activations + feedback states |
| learned NP, simultaneous | ordinary forward; one duplicate clean plus `2K` noisy full forwards every `e` batches | learned direct maps; no weight transport | `2KB/e` per ordinary batch | `(1+2K)F/e` | up to `2KB` noisy examples in the vectorized implementation |
| learned NP, layerwise | ordinary forward; clean baseline plus antithetic suffix replays | learned direct maps | `(1+2KH)B/e` | `(F + 2K sum_l S_l)/e`, quadratic in depth for balanced layers | serial suffix replay, no `2K` batch expansion |
| direct NP | layerwise or simultaneous estimator every update | none beyond causal intervention | same formulas with `e=1` | same formulas with `e=1` | estimator dependent |
| canonical EP | free relaxation then nudged relaxation | symmetric energy weights | label injected in nudged phase | `T_free + T_nudged` recurrent steps | persistent/free and nudged states |
| Dual Prop `DP^T` VGG | one inference phase, 17 forward-only layer passes in the author CIFAR protocol | dyadic forward/top-down states | ordinary target loss | 17 layer passes before update | two states per dyadic unit |
| BurstCCN | one burst/event-probability phase with plastic W/Y/Q pathways | separate plastic feedback pathways | ordinary supervised target | architecture-dependent local W/Y/Q updates | event and burst states plus feedback weights |

The project ledger reports the exact realized loss observations,
forward-equivalent examples, peak allocated memory, and wall time. Asymptotic
labels never replace those measurements. In particular, the frozen direct-NP
C2 diagnosis costs `68.4x` ordinary work, whereas the frozen CIFAR d20 K1/e4
protocol uses 16x fewer logical queries, 11x less calibration work, and 5.3x
less total work than K16/e4 while retaining its gain over DFA.

## 5. Executable checks

Run:

```bash
python experiments/verify_theory.py
```

The verifier checks the exact Rademacher MSE across depth, width, and K; the
quadratic finite-sigma bias law; the smooth-descent step threshold; neutral
predictor convergence across `eta_P` and update counts; and the task-fit
teaching-signal absorption predicted above.