path: root/TRANSFORMER_CROSSOVER.md

# Decoder-Transformer local-learning crossover

## Claim and immutable cell scope

This family protocol instantiates the Transformer4/8/12 rows registered in
`CROSS_ARCHITECTURE_CROSSOVER.md`.  The three depths are three separate
architecture--scale points, not one averaged "Transformer" result.  Every
stage retains all nine methods:

`BP, FA, DFA, PEPITA, Forward-Forward, EP, DualProp, clean-KP, SDIL`.

A failed, chance-level, nonfinite, or timed-out cell remains in the table.
Neither a weak depth-4 selector result nor the cost of a state method permits
its depth-8/12 cell to be dropped.  Any later width, context, or additional
depth ladder is a new complete revision containing all nine methods.  It
cannot selectively extend SDIL or replace these 27 primary endpoints.

## Frozen task, bytes, and forward graph

The task is next-character prediction on the nanoGPT tiny-Shakespeare split.
The only formal input files are uint16 token arrays:

| split | bytes | tokens | SHA-256 |
|---|---:|---:|---|
| train | 2,007,708 | 1,003,854 | `6ec305602a99ac2802745a134e1f5e33e2231b4855525b00b9aebb730ac2626f` |
| validation | 223,080 | 111,540 | `d37d30cc0c8327c270d493299c3dca54135f6d5f1c9ef60cda78076e311204b1` |

There is no test access in T1.  Every training batch is drawn by the same
frozen sequence-start generator.  Validation uses deterministic nonoverlap
windows over the complete validation file.  Each token window has context
64 and target offset one.  Vocabulary size is 65.

The matched forward graph is a causal-softmax, pre-LayerNorm decoder with
learned untied token and position embeddings, four attention heads, width
128, MLP ratio four, no affine bias, no dropout, and an untied vocabulary
head.  Depth is the only primary architecture variable:

| ID | blocks | forward parameters |
|---|---:|---:|
| `transformer4` | 4 | 813,568 |
| `transformer8` | 8 | 1,602,048 |
| `transformer12` | 12 | 2,390,528 |

Forward parameters and their initialization are bit-identical within a
depth--seed pair.  Feedback draws use separate generators and therefore
cannot shift later forward draws.  The implementation is
`sdil/transformer.py`; `experiments/transformer_feedback_smoke.py` is the
required equation gate.

## Frozen method adapters

**BP.** Exact reverse-mode differentiation of training cross-entropy.

**Ordinary FA.** Every parameterized affine transports its recursively
arriving field through an independent fixed random matrix.  Residual
addition, causal softmax attention, GELU, and LayerNorm retain their exact
parameter-free local Jacobians.  FA feedback never reads, copies, or changes
with a forward weight.  It is not DFA.

**Strict block DFA.** The analytical vocabulary error is independently
projected to every decoder-block output, the embedding/position population,
and final LayerNorm.  Every block input is detached and only the explicit
within-block objective is differentiated.  The vocabulary head uses its
analytical local delta.  No embedding, normalization, or affine parameter
retains a hidden BP gradient.

**PEPITA.** The first presentation's analytical vocabulary error is projected
by a fixed `65 x 128` matrix into the continuous token-embedding stream.  A
second presentation uses the perturbed embedding.  Each block receives its
detached first-minus-second output field and second-presentation input.  The
head receives the second-presentation output error.  Because discrete token
IDs cannot be perturbed, the token table and positional code use their
directly observed embedding difference.  This Transformer-specific choice is
reported as a PEPITA/ERIN adapter, not an assertion that the original paper
studied language models.  Projection scale is 0.05.

**Forward-Forward.** The true next character forms a positive candidate and a
uniformly sampled guaranteed-wrong character forms a negative candidate.
The candidate is overlaid in the first 65 coordinates of each continuous
token representation, mirroring supervised label overlay.  Token/position
embedding, every decoder block, and final-LayerNorm-plus-head are greedy
layers.  Prefixes and non-target layers are detached.  Inputs are normalized
per token; the local loss is the positive/negative softplus goodness loss
with threshold 2.  Validation enumerates all 65 candidates and sums goodness
from greedy layer one onward.  All 65 candidate presentations are charged.

**Dual Propagation.** One plus and one minus state are exposed for the
embedding population, every decoder-block output, and logits.  They start at
the ordinary forward state.  With author-style `alpha=0`, `beta=0.1`, 16
sequential fwK passes update the graph.  A downstream state difference is
transported by an exact local VJP through that same child block or head.
This intentional symmetric feedback is a strong, weight-transport-using
baseline.  Contrastive parameter correlations are evaluated only inside
detached edges.

**Equilibrium Propagation.** The same state graph has one hard-sigmoid state
per population.  Synchronous leaky relaxation uses 20 free and four nudged
steps, `dt=0.5`, nudge magnitude 0.5, and a deterministic random nudge sign
per batch.  Child activity returns through the exact local VJP of the same
edge.  Updates are the signed-beta-normalized difference between nudged and
free local correlations.  Validation uses the 20-step free output state.
EP's relaxation and symmetric transport are reported separately.

**Clean KP.** The ordinary-FA feedback matrix is plastic.  Its update is
independently recomputed from the same locally available pre-activity and
post-field correlation as the forward affine; it is not assigned from a
forward gradient.  Shared optimizer dynamics and decay contract reciprocal
misalignment.

**SDIL.** The transport and reciprocal update are exactly clean-KP.  At every
affine, the apical signal additionally contains soma-predictable traffic with
four times the task-field RMS.  A paired neutral observation predicts and
subtracts that component before transport and plasticity.  Thus the clean
task is a prespecified KP/SDIL identity check while the raw apical field is
strongly contaminated; raw and innovation RMS are recorded.  The separate
nuisance ablation, rather than this clean identity, tests the benefit of
residualization.

## X0 equation gate

Before a formal endpoint, the deterministic smoke must establish:

- bit-identical forward logits and parameter counts for every adapter;
- tensorwise BP agreement when ordinary-FA feedback is set to its forward
  matrix;
- fixed-FA and fixed-DFA/PEPITA projection independence;
- independently recomputed KP feedback correlations;
- KP/SDIL gradient identity under paired-neutral subtraction and a
  measurably contaminated raw field;
- no cross-block gradient leakage in DFA, PEPITA, or Forward-Forward;
- target-layer-only Forward-Forward gradients and `B x T x 65` scores;
- paper-equation agreement for PEPITA, Dual Propagation, and EP;
- finite state ranges, explicit local-VJP counts, and zero calls to
  task-loss backward for DFA/PEPITA/DP/EP.

The gate must pass both the development PyTorch environment and the Pascal
server's frozen PyTorch 2.3 environment.

## T1 validation-only selector

T1 contains 27 immutable depth-4 jobs: every method at all three of its
predeclared learning rates.  It uses run seed 0, model seed 2027, loader seed
0, negative seed 5001, EP-sign seed 5002, batch size 32, 1,000 optimizer
steps, constant learning rate, AdamW `(beta1,beta2)=(0.9,0.95)`, weight decay
0.1, and the full validation file once at the endpoint.  Each job therefore
receives exactly 2,048,000 ordinary training tokens.  Forward-Forward spends
this fixed total greedily in layer order rather than multiplying the token
budget by its number of layers.

| method | learning-rate grid |
|---|---|
| BP, FA, DFA, clean-KP, SDIL | `0.0003, 0.001, 0.003` |
| PEPITA, Forward-Forward | `0.0001, 0.0003, 0.001` |
| EP, Dual Propagation | `0.0001, 0.0003, 0.001` |

Selection minimizes finite final validation NLL.  Exact ties choose the
smaller learning rate.  Nonfinite and timed-out jobs rank after every finite
job but remain reported.  No intermediate validation, test token, development
pilot endpoint, or cross-method setting is used by the selector.

The per-job cap is 48 hours.  Only timan107 physical GPUs 5 and 7 are allowed.
A launch manifest must record the physical GPU UUID, source commit, tracked
dirty state, environment freeze, dataset hashes, arguments, peak memory,
wall time, and exit state.  A failure advances the shard rather than deleting
the cell.

## T2 complete depth crossover

After the selector JSON and its hash are committed, T2 applies each selected
method setting without a new search to depths 4, 8, and 12: 27 cells.  All
cells use 5,000 optimizer steps, batch size 32, 100-step linear warmup, cosine
decay from the selected learning rate to one tenth of it, and the same
ordinary-token, validation, failure, source, and hardware rules.  Depth does
not alter DP/EP inference counts, FF threshold, PEPITA scale, traffic ratio,
or any method variant.

T2 is a single-seed scaling panel, not the final uncertainty estimate.  An
untouched multi-seed T3 registry must contain every one of the same 27 cells;
winner-only or SDIL-only replication cannot support the scaling claim.

## Metrics and cost accounting

The primary task metric is full-validation next-token NLL; perplexity and
token accuracy are secondary.  Each record also reports wall time, peak
allocated/reserved memory, forward/feedback parameters, ordinary training
and validation tokens, positive/negative or perturbed presentations,
candidate-token presentations, relaxation-token passes, local-VJP token
evaluations, first nonfinite step, and completed optimizer steps.  The
hardware-independent ledger additionally enumerates full-forward token
passes, extra local block/head evaluations, block affine MACs per token,
causal-attention MACs per token, and whole-forward MACs per token.  These
primitive counts remain separate for DP/EP because a local VJP is not
silently priced as an ordinary forward.

The accuracy--cost plot and the within-family depth plot retain timeouts and
failures.  A low-cost method is not declared scalable from depth 4 alone, and
an accurate DP/EP point is not declared local in the weight-transport-free
sense merely because its VJPs were evaluated at detached edges.