# Decoder-Transformer local-learning crossover ## Claim and immutable cell scope This family protocol instantiates the Transformer4/8/12 rows registered in `CROSS_ARCHITECTURE_CROSSOVER.md`. The three depths are three separate architecture--scale points, not one averaged "Transformer" result. Every stage retains all nine methods: `BP, FA, DFA, PEPITA, Forward-Forward, EP, DualProp, clean-KP, SDIL`. A failed, chance-level, nonfinite, or timed-out cell remains in the table. Neither a weak depth-4 selector result nor the cost of a state method permits its depth-8/12 cell to be dropped. Any later width, context, or additional depth ladder is a new complete revision containing all nine methods. It cannot selectively extend SDIL or replace these 27 primary endpoints. ## Frozen task, bytes, and forward graph The task is next-character prediction on the nanoGPT tiny-Shakespeare split. The only formal input files are uint16 token arrays: | split | bytes | tokens | SHA-256 | |---|---:|---:|---| | train | 2,007,708 | 1,003,854 | `6ec305602a99ac2802745a134e1f5e33e2231b4855525b00b9aebb730ac2626f` | | validation | 223,080 | 111,540 | `d37d30cc0c8327c270d493299c3dca54135f6d5f1c9ef60cda78076e311204b1` | There is no test access in T1. Every training batch is drawn by the same frozen sequence-start generator. Validation uses deterministic nonoverlap windows over the complete validation file. Each token window has context 64 and target offset one. Vocabulary size is 65. The matched forward graph is a causal-softmax, pre-LayerNorm decoder with learned untied token and position embeddings, four attention heads, width 128, MLP ratio four, no affine bias, no dropout, and an untied vocabulary head. Depth is the only primary architecture variable: | ID | blocks | forward parameters | |---|---:|---:| | `transformer4` | 4 | 813,568 | | `transformer8` | 8 | 1,602,048 | | `transformer12` | 12 | 2,390,528 | Forward parameters and their initialization are bit-identical within a depth--seed pair. Feedback draws use separate generators and therefore cannot shift later forward draws. The implementation is `sdil/transformer.py`; `experiments/transformer_feedback_smoke.py` is the required equation gate. ## Frozen method adapters **BP.** Exact reverse-mode differentiation of training cross-entropy. **Ordinary FA.** Every parameterized affine transports its recursively arriving field through an independent fixed random matrix. Residual addition, causal softmax attention, GELU, and LayerNorm retain their exact parameter-free local Jacobians. FA feedback never reads, copies, or changes with a forward weight. It is not DFA. **Strict block DFA.** The analytical vocabulary error is independently projected to every decoder-block output, the embedding/position population, and final LayerNorm. Every block input is detached and only the explicit within-block objective is differentiated. The vocabulary head uses its analytical local delta. No embedding, normalization, or affine parameter retains a hidden BP gradient. **PEPITA.** The first presentation's analytical vocabulary error is projected by a fixed `65 x 128` matrix into the continuous token-embedding stream. A second presentation uses the perturbed embedding. Each block receives its detached first-minus-second output field and second-presentation input. The head receives the second-presentation output error. Because discrete token IDs cannot be perturbed, the token table and positional code use their directly observed embedding difference. This Transformer-specific choice is reported as a PEPITA/ERIN adapter, not an assertion that the original paper studied language models. Projection scale is 0.05. **Forward-Forward.** The true next character forms a positive candidate and a uniformly sampled guaranteed-wrong character forms a negative candidate. The candidate is overlaid in the first 65 coordinates of each continuous token representation, mirroring supervised label overlay. Token/position embedding, every decoder block, and final-LayerNorm-plus-head are greedy layers. Prefixes and non-target layers are detached. Inputs are normalized per token; the local loss is the positive/negative softplus goodness loss with threshold 2. Validation enumerates all 65 candidates and sums goodness from greedy layer one onward. All 65 candidate presentations are charged. **Dual Propagation.** One plus and one minus state are exposed for the embedding population, every decoder-block output, and logits. They start at the ordinary forward state. With author-style `alpha=0`, `beta=0.1`, 16 sequential fwK passes update the graph. A downstream state difference is transported by an exact local VJP through that same child block or head. This intentional symmetric feedback is a strong, weight-transport-using baseline. Contrastive parameter correlations are evaluated only inside detached edges. **Equilibrium Propagation.** The same state graph has one hard-sigmoid state per population. Synchronous leaky relaxation uses 20 free and four nudged steps, `dt=0.5`, nudge magnitude 0.5, and a deterministic random nudge sign per batch. Child activity returns through the exact local VJP of the same edge. Updates are the signed-beta-normalized difference between nudged and free local correlations. Validation uses the 20-step free output state. EP's relaxation and symmetric transport are reported separately. **Clean KP.** The ordinary-FA feedback matrix is plastic. Its update is independently recomputed from the same locally available pre-activity and post-field correlation as the forward affine; it is not assigned from a forward gradient. Shared optimizer dynamics and decay contract reciprocal misalignment. **SDIL.** The transport and reciprocal update are exactly clean-KP. At every affine, the apical signal additionally contains soma-predictable traffic with four times the task-field RMS. A paired neutral observation predicts and subtracts that component before transport and plasticity. Thus the clean task is a prespecified KP/SDIL identity check while the raw apical field is strongly contaminated; raw and innovation RMS are recorded. The separate nuisance ablation, rather than this clean identity, tests the benefit of residualization. ## X0 equation gate Before a formal endpoint, the deterministic smoke must establish: - bit-identical forward logits and parameter counts for every adapter; - tensorwise BP agreement when ordinary-FA feedback is set to its forward matrix; - fixed-FA and fixed-DFA/PEPITA projection independence; - independently recomputed KP feedback correlations; - KP/SDIL gradient identity under paired-neutral subtraction and a measurably contaminated raw field; - no cross-block gradient leakage in DFA, PEPITA, or Forward-Forward; - target-layer-only Forward-Forward gradients and `B x T x 65` scores; - paper-equation agreement for PEPITA, Dual Propagation, and EP; - finite state ranges, explicit local-VJP counts, and zero calls to task-loss backward for DFA/PEPITA/DP/EP. The gate must pass both the development PyTorch environment and the Pascal server's frozen PyTorch 2.3 environment. ## T1 validation-only selector T1 contains 27 immutable depth-4 jobs: every method at all three of its predeclared learning rates. It uses run seed 0, model seed 2027, loader seed 0, negative seed 5001, EP-sign seed 5002, batch size 32, 1,000 optimizer steps, constant learning rate, AdamW `(beta1,beta2)=(0.9,0.95)`, weight decay 0.1, and the full validation file once at the endpoint. Each job therefore receives exactly 2,048,000 ordinary training tokens. Forward-Forward spends this fixed total greedily in layer order rather than multiplying the token budget by its number of layers. | method | learning-rate grid | |---|---| | BP, FA, DFA, clean-KP, SDIL | `0.0003, 0.001, 0.003` | | PEPITA, Forward-Forward | `0.0001, 0.0003, 0.001` | | EP, Dual Propagation | `0.0001, 0.0003, 0.001` | Selection minimizes finite final validation NLL. Exact ties choose the smaller learning rate. Nonfinite and timed-out jobs rank after every finite job but remain reported. No intermediate validation, test token, development pilot endpoint, or cross-method setting is used by the selector. The per-job cap is 48 hours. Only timan107 physical GPUs 5 and 7 are allowed. Before either shard starts, one immutable launch lock binds the complete 27-job registry hash to the source commit, tracked-file hashes, environment freeze, and allowed physical GPUs. Every subsequent cell rechecks that source lock. Its manifest records the physical GPU UUID, source commit, tracked dirty state, environment freeze, dataset hashes, arguments, peak memory, wall time, and exit state. A failure advances the shard rather than deleting the cell. ## T2 complete depth crossover After the selector JSON and its hash are committed, T2 applies each selected method setting without a new search to depths 4, 8, and 12: 27 cells. All cells use 5,000 optimizer steps, batch size 32, 100-step linear warmup, cosine decay from the selected learning rate to one tenth of it, and the same ordinary-token, validation, failure, source, and hardware rules. Depth does not alter DP/EP inference counts, FF threshold, PEPITA scale, traffic ratio, or any method variant. T2 is a single-seed scaling panel, not the final uncertainty estimate. An untouched multi-seed T3 registry must contain every one of the same 27 cells; winner-only or SDIL-only replication cannot support the scaling claim. The executable T2 contract is `experiments/transformer_crossover_t2.py`. It refuses an incomplete T1 selector, requires the selector to be committed, binds its hash, selected rates, clean source, environment freeze, and GPU5/7 allowlist into a new launch lock, and enumerates all 27 method--depth cells before sharding. Expensive EP, Dual Propagation, and Forward--Forward cells are interleaved across shards. `experiments/analyze_transformer_crossover_t2.py` requires all manifests, retains timeout and nonfinite outcomes, rejects data/source/architecture/test drift, and checks token presentations, candidate evaluations, relaxation passes, local VJPs, logical queries, and enumerated MACs. The immutable registry can be checked before T1 finishes with: ```bash python experiments/transformer_crossover_t2_smoke.py ``` ## Metrics and cost accounting The primary task metric is full-validation next-token NLL; perplexity and token accuracy are secondary. Each record also reports wall time, peak allocated/reserved memory, forward/feedback parameters, ordinary training and validation tokens, positive/negative or perturbed presentations, candidate-token presentations, relaxation-token passes, local-VJP token evaluations, first nonfinite step, and completed optimizer steps. The hardware-independent ledger additionally enumerates full-forward token passes, extra local block/head evaluations, block affine MACs per token, causal-attention MACs per token, and whole-forward MACs per token. These primitive counts remain separate for DP/EP because a local VJP is not silently priced as an ordinary forward. The accuracy--cost plot and the within-family depth plot retain timeouts and failures. A low-cost method is not declared scalable from depth 4 alone, and an accurate DP/EP point is not declared local in the weight-transport-free sense merely because its VJPs were evaluated at detached edges.