diff options
Diffstat (limited to 'TRANSFORMER_CROSSOVER.md')
| -rw-r--r-- | TRANSFORMER_CROSSOVER.md | 192 |
1 files changed, 192 insertions, 0 deletions
diff --git a/TRANSFORMER_CROSSOVER.md b/TRANSFORMER_CROSSOVER.md new file mode 100644 index 0000000..2bcff12 --- /dev/null +++ b/TRANSFORMER_CROSSOVER.md @@ -0,0 +1,192 @@ +# Decoder-Transformer local-learning crossover + +## Claim and immutable cell scope + +This family protocol instantiates the Transformer4/8/12 rows registered in +`CROSS_ARCHITECTURE_CROSSOVER.md`. The three depths are three separate +architecture--scale points, not one averaged "Transformer" result. Every +stage retains all nine methods: + +`BP, FA, DFA, PEPITA, Forward-Forward, EP, DualProp, clean-KP, SDIL`. + +A failed, chance-level, nonfinite, or timed-out cell remains in the table. +Neither a weak depth-4 selector result nor the cost of a state method permits +its depth-8/12 cell to be dropped. Any later width, context, or additional +depth ladder is a new complete revision containing all nine methods. It +cannot selectively extend SDIL or replace these 27 primary endpoints. + +## Frozen task, bytes, and forward graph + +The task is next-character prediction on the nanoGPT tiny-Shakespeare split. +The only formal input files are uint16 token arrays: + +| split | bytes | tokens | SHA-256 | +|---|---:|---:|---| +| train | 2,007,708 | 1,003,854 | `6ec305602a99ac2802745a134e1f5e33e2231b4855525b00b9aebb730ac2626f` | +| validation | 223,080 | 111,540 | `d37d30cc0c8327c270d493299c3dca54135f6d5f1c9ef60cda78076e311204b1` | + +There is no test access in T1. Every training batch is drawn by the same +frozen sequence-start generator. Validation uses deterministic nonoverlap +windows over the complete validation file. Each token window has context +64 and target offset one. Vocabulary size is 65. + +The matched forward graph is a causal-softmax, pre-LayerNorm decoder with +learned untied token and position embeddings, four attention heads, width +128, MLP ratio four, no affine bias, no dropout, and an untied vocabulary +head. Depth is the only primary architecture variable: + +| ID | blocks | forward parameters | +|---|---:|---:| +| `transformer4` | 4 | 813,568 | +| `transformer8` | 8 | 1,602,048 | +| `transformer12` | 12 | 2,390,528 | + +Forward parameters and their initialization are bit-identical within a +depth--seed pair. Feedback draws use separate generators and therefore +cannot shift later forward draws. The implementation is +`sdil/transformer.py`; `experiments/transformer_feedback_smoke.py` is the +required equation gate. + +## Frozen method adapters + +**BP.** Exact reverse-mode differentiation of training cross-entropy. + +**Ordinary FA.** Every parameterized affine transports its recursively +arriving field through an independent fixed random matrix. Residual +addition, causal softmax attention, GELU, and LayerNorm retain their exact +parameter-free local Jacobians. FA feedback never reads, copies, or changes +with a forward weight. It is not DFA. + +**Strict block DFA.** The analytical vocabulary error is independently +projected to every decoder-block output, the embedding/position population, +and final LayerNorm. Every block input is detached and only the explicit +within-block objective is differentiated. The vocabulary head uses its +analytical local delta. No embedding, normalization, or affine parameter +retains a hidden BP gradient. + +**PEPITA.** The first presentation's analytical vocabulary error is projected +by a fixed `65 x 128` matrix into the continuous token-embedding stream. A +second presentation uses the perturbed embedding. Each block receives its +detached first-minus-second output field and second-presentation input. The +head receives the second-presentation output error. Because discrete token +IDs cannot be perturbed, the token table and positional code use their +directly observed embedding difference. This Transformer-specific choice is +reported as a PEPITA/ERIN adapter, not an assertion that the original paper +studied language models. Projection scale is 0.05. + +**Forward-Forward.** The true next character forms a positive candidate and a +uniformly sampled guaranteed-wrong character forms a negative candidate. +The candidate is overlaid in the first 65 coordinates of each continuous +token representation, mirroring supervised label overlay. Token/position +embedding, every decoder block, and final-LayerNorm-plus-head are greedy +layers. Prefixes and non-target layers are detached. Inputs are normalized +per token; the local loss is the positive/negative softplus goodness loss +with threshold 2. Validation enumerates all 65 candidates and sums goodness +from greedy layer one onward. All 65 candidate presentations are charged. + +**Dual Propagation.** One plus and one minus state are exposed for the +embedding population, every decoder-block output, and logits. They start at +the ordinary forward state. With author-style `alpha=0`, `beta=0.1`, 16 +sequential fwK passes update the graph. A downstream state difference is +transported by an exact local VJP through that same child block or head. +This intentional symmetric feedback is a strong, weight-transport-using +baseline. Contrastive parameter correlations are evaluated only inside +detached edges. + +**Equilibrium Propagation.** The same state graph has one hard-sigmoid state +per population. Synchronous leaky relaxation uses 20 free and four nudged +steps, `dt=0.5`, nudge magnitude 0.5, and a deterministic random nudge sign +per batch. Child activity returns through the exact local VJP of the same +edge. Updates are the signed-beta-normalized difference between nudged and +free local correlations. Validation uses the 20-step free output state. +EP's relaxation and symmetric transport are reported separately. + +**Clean KP.** The ordinary-FA feedback matrix is plastic. Its update is +independently recomputed from the same locally available pre-activity and +post-field correlation as the forward affine; it is not assigned from a +forward gradient. Shared optimizer dynamics and decay contract reciprocal +misalignment. + +**SDIL.** The transport and reciprocal update are exactly clean-KP. At every +affine, the apical signal additionally contains soma-predictable traffic with +four times the task-field RMS. A paired neutral observation predicts and +subtracts that component before transport and plasticity. Thus the clean +task is a prespecified KP/SDIL identity check while the raw apical field is +strongly contaminated; raw and innovation RMS are recorded. The separate +nuisance ablation, rather than this clean identity, tests the benefit of +residualization. + +## X0 equation gate + +Before a formal endpoint, the deterministic smoke must establish: + +- bit-identical forward logits and parameter counts for every adapter; +- tensorwise BP agreement when ordinary-FA feedback is set to its forward + matrix; +- fixed-FA and fixed-DFA/PEPITA projection independence; +- independently recomputed KP feedback correlations; +- KP/SDIL gradient identity under paired-neutral subtraction and a + measurably contaminated raw field; +- no cross-block gradient leakage in DFA, PEPITA, or Forward-Forward; +- target-layer-only Forward-Forward gradients and `B x T x 65` scores; +- paper-equation agreement for PEPITA, Dual Propagation, and EP; +- finite state ranges, explicit local-VJP counts, and zero calls to + task-loss backward for DFA/PEPITA/DP/EP. + +The gate must pass both the development PyTorch environment and the Pascal +server's frozen PyTorch 2.3 environment. + +## T1 validation-only selector + +T1 contains 27 immutable depth-4 jobs: every method at all three of its +predeclared learning rates. It uses run seed 0, model seed 2027, loader seed +0, negative seed 5001, EP-sign seed 5002, batch size 32, 1,000 optimizer +steps, constant learning rate, AdamW `(beta1,beta2)=(0.9,0.95)`, weight decay +0.1, and the full validation file once at the endpoint. Each job therefore +receives exactly 2,048,000 ordinary training tokens. Forward-Forward spends +this fixed total greedily in layer order rather than multiplying the token +budget by its number of layers. + +| method | learning-rate grid | +|---|---| +| BP, FA, DFA, clean-KP, SDIL | `0.0003, 0.001, 0.003` | +| PEPITA, Forward-Forward | `0.0001, 0.0003, 0.001` | +| EP, Dual Propagation | `0.0001, 0.0003, 0.001` | + +Selection minimizes finite final validation NLL. Exact ties choose the +smaller learning rate. Nonfinite and timed-out jobs rank after every finite +job but remain reported. No intermediate validation, test token, development +pilot endpoint, or cross-method setting is used by the selector. + +The per-job cap is 48 hours. Only timan107 physical GPUs 5 and 7 are allowed. +A launch manifest must record the physical GPU UUID, source commit, tracked +dirty state, environment freeze, dataset hashes, arguments, peak memory, +wall time, and exit state. A failure advances the shard rather than deleting +the cell. + +## T2 complete depth crossover + +After the selector JSON and its hash are committed, T2 applies each selected +method setting without a new search to depths 4, 8, and 12: 27 cells. All +cells use 5,000 optimizer steps, batch size 32, 100-step linear warmup, cosine +decay from the selected learning rate to one tenth of it, and the same +ordinary-token, validation, failure, source, and hardware rules. Depth does +not alter DP/EP inference counts, FF threshold, PEPITA scale, traffic ratio, +or any method variant. + +T2 is a single-seed scaling panel, not the final uncertainty estimate. An +untouched multi-seed T3 registry must contain every one of the same 27 cells; +winner-only or SDIL-only replication cannot support the scaling claim. + +## Metrics and cost accounting + +The primary task metric is full-validation next-token NLL; perplexity and +token accuracy are secondary. Each record also reports wall time, peak +allocated/reserved memory, forward/feedback parameters, ordinary training +and validation tokens, positive/negative or perturbed presentations, +candidate-token presentations, relaxation-token passes, local-VJP token +evaluations, first nonfinite step, and completed optimizer steps. The +accuracy--cost plot and the within-family depth plot retain timeouts and +failures. A low-cost method is not declared scalable from depth 4 alone, and +an accurate DP/EP point is not declared local in the weight-transport-free +sense merely because its VJPs were evaluated at detached edges. |