diff options
Diffstat (limited to 'THEORY.md')
| -rw-r--r-- | THEORY.md | 222 |
1 files changed, 222 insertions, 0 deletions
diff --git a/THEORY.md b/THEORY.md new file mode 100644 index 0000000..25d27ef --- /dev/null +++ b/THEORY.md @@ -0,0 +1,222 @@ +# SDIL theory and cost audit + +This note states the minimum claims that can be proved for the implemented +estimators and innovation rule. It deliberately separates an unbiased neural +teaching direction from a useful parameter update: the completed experiments +show that a high hidden-state cosine can coexist with poor optimization. + +## 1. Simultaneous multi-layer perturbation + +Consider one example and collect the hidden-state gradients +`g_l = dL/dh_l` for hidden blocks `l=1,...,H`, where `g_l` has dimension +`d_l`. For one simultaneous Rademacher intervention, draw independent +`xi_l in {-1,+1}^{d_l}` and linearize the centered loss difference as + +```text +D = sum_j <g_j, xi_j>. +``` + +The gradient and descent estimators for layer `l` are + +```text +g_hat_l = xi_l D, q_l = -g_hat_l. +``` + +Independence and `E[xi_l xi_l^T] = I` give + +```text +E[g_hat_l] = g_l. +``` + +Because a Rademacher vector has deterministic squared norm `d_l`, the exact +linearized mean-squared error is + +```text +E ||g_hat_l - g_l||^2 + = (d_l - 1) ||g_l||^2 + + d_l sum_{j != l} ||g_j||^2. +``` + +Averaging `K` independent directions divides this expression by `K`. The first +term is ordinary directional-estimator variance; the second is cross-layer +interference. Thus simultaneous perturbation is unbiased but its relative +variance grows with layer width and with the total gradient energy injected at +other layers. This predicts why K1 can work in an easy/depth-flat regime yet +fail to train a moving context vectorizer on a compositional task. + +For comparison, layerwise Rademacher perturbation removes the cross-layer term. +The current layerwise implementation uses Gaussian directions, for which + +```text +E ||g_hat_l - g_l||^2 = (d_l + 1) ||g_l||^2 / K. +``` + +Layerwise targets therefore buy variance reduction with depth-dependent replay +cost; they are not a free scalable replacement for simultaneous calibration. + +### Finite perturbation bias + +Suppose the third derivative of the loss has operator norm at most `M` in the +perturbation neighborhood, and let `D_total = sum_l d_l`. A centered antithetic +difference has directional error at most + +```text +M sigma^2 ||xi||^3 / 6. +``` + +For simultaneous Rademacher directions this gives the conservative layerwise +bias bound + +```text +||E[g_hat_l] - g_l|| + <= (M sigma^2 / 6) sqrt(d_l) D_total^(3/2). +``` + +The important dependence is `O(sigma^2)`, with a dimension-dependent constant. +Reducing K removes variance but not this bias; increasing width/depth without a +sigma audit is therefore unsafe. + +## 2. What is sufficient for a descent step + +Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let +`v` be the complete local update before multiplying by the learning rate: + +```text +theta_next = theta + eta v. +``` + +If the loss is `beta`-smooth and + +```text +c = <-p, v> / (||p|| ||v||) > 0, +``` + +then the descent lemma gives + +```text +L(theta_next) - L(theta) + <= -eta c ||p|| ||v|| + (beta eta^2 / 2) ||v||^2. +``` + +Consequently the update is guaranteed to descend only when + +```text +0 < eta < 2 c ||p|| / (beta ||v||). +``` + +This separates three quantities that a neural-state cosine conflates: + +1. direction (`c`); +2. gain (`||v|| / ||p||`); +3. curvature (`beta`). + +For a local layer, multiplying a hidden teaching vector by the activation gain +and presynaptic activity maps it into parameter space. A positive +`cos(r_l, -g_l)` does not by itself imply a positive cosine after aggregating +examples, parameters, and layers, and even perfect parameter alignment can +increase loss if its gain makes the step exceed the bound. This is why future +diagnostics must include parameter-update cosine, norm ratio, and realized +single-step loss change rather than reporting hidden cosine alone. + +## 3. Somato-dendritic innovation as conditional nuisance removal + +Write apical activity during a task period as + +```text +a = s + M h + epsilon, +``` + +where `s` is the instructional component, `M h` is ordinary traffic predictable +from soma, and `epsilon` is unpredictable noise. In neutral periods `s=0`. +Assume `Sigma_h = E[h h^T]` is nonsingular and `E[epsilon h^T]=0`. Population +least squares on neutral periods yields + +```text +P_neutral = E[a h^T | neutral] Sigma_h^(-1) = M. +``` + +The task-period innovation is then + +```text +r_neutral = a - P_neutral h = s + epsilon. +``` + +If `s`, `M h`, and `epsilon` are mutually uncorrelated, a squared-cosine/SNR +proxy improves from + +```text +raw: S / (S + T + E) +innovation:S / (S + E), +``` + +where `S`, `T`, and `E` are their expected squared norms. The gain is strict +whenever predictable traffic has nonzero power. + +Task-period fitting has a different estimand. Let + +```text +C = E[s h^T] Sigma_h^(-1). +``` + +Then `P_task = M + C` and + +```text +r_task = s - C h + epsilon. +``` + +It removes the soma-predictable projection of the teaching signal itself. This +proves the identification role of neutral periods and also defines the failure +boundary: residualization cannot remove contextual traffic that remains +unpredictable from the chosen per-cell soma statistic, and it can erase credit +when the predictor is fit on task periods. + +For the diagonal normalized-LMS predictor used in the code, an ideal noiseless +cell with neutral relation `a=m h` follows + +```text +p_(t+1) = p_t + eta_P (m - p_t), +traffic residual power fraction = (1 - eta_P)^(2t). +``` + +The relevant timescale is therefore the product of learning rate and number of +neutral updates. Too few neutral updates leave traffic; task-period updates +converge to the wrong coefficient regardless of speed. + +## 4. Hardware-independent complexity + +Let `H` be the number of hidden layers, `K` the perturbation directions per +event, `e` the number of ordinary minibatches between events, `B` the batch +size, and `F_l` the affine work of layer `l`. Define `F=sum_l F_l` and suffix +work `S_l=sum_{j>l} F_j`. + +| method | activity phases / passes | transport | scalar task-loss observations | extra affine work | main transient memory | +|:--|:--|:--|:--|:--|:--| +| BP | forward + reverse | exact transposed Jacobians | ordinary supervised loss | about one reverse pass | saved activations / graph | +| FA | forward + reverse-like serial feedback | fixed random feedback | ordinary loss | `O(F)` feedback work | activations + feedback states | +| DFA | forward + direct feedback | fixed random output maps | ordinary loss | `O(sum_l d_l d_out)` | activations + feedback states | +| learned NP, simultaneous | ordinary forward; one duplicate clean plus `2K` noisy full forwards every `e` batches | learned direct maps; no weight transport | `2KB/e` per ordinary batch | `(1+2K)F/e` | up to `2KB` noisy examples in the vectorized implementation | +| learned NP, layerwise | ordinary forward; clean baseline plus antithetic suffix replays | learned direct maps | `(1+2KH)B/e` | `(F + 2K sum_l S_l)/e`, quadratic in depth for balanced layers | serial suffix replay, no `2K` batch expansion | +| direct NP | layerwise or simultaneous estimator every update | none beyond causal intervention | same formulas with `e=1` | same formulas with `e=1` | estimator dependent | +| canonical EP | free relaxation then nudged relaxation | symmetric energy weights | label injected in nudged phase | `T_free + T_nudged` recurrent steps | persistent/free and nudged states | +| Dual Prop `DP^T` VGG | one inference phase, 17 forward-only layer passes in the author CIFAR protocol | dyadic forward/top-down states | ordinary target loss | 17 layer passes before update | two states per dyadic unit | +| BurstCCN | one burst/event-probability phase with plastic W/Y/Q pathways | separate plastic feedback pathways | ordinary supervised target | architecture-dependent local W/Y/Q updates | event and burst states plus feedback weights | + +The project ledger reports the exact realized loss observations, +forward-equivalent examples, peak allocated memory, and wall time. Asymptotic +labels never replace those measurements. In particular, the frozen direct-NP +C2 diagnosis costs `68.4x` ordinary work, whereas the frozen CIFAR d20 K1/e4 +protocol uses 16x fewer logical queries, 11x less calibration work, and 5.3x +less total work than K16/e4 while retaining its gain over DFA. + +## 5. Executable checks + +Run: + +```bash +python experiments/verify_theory.py +``` + +The verifier checks the exact Rademacher MSE across depth, width, and K; the +quadratic finite-sigma bias law; the smooth-descent step threshold; neutral +predictor convergence across `eta_P` and update counts; and the task-fit +teaching-signal absorption predicted above. |