experiment: freeze Transformer crossover protocol

1 files changed, 192 insertions, 0 deletions

diff --git a/TRANSFORMER_CROSSOVER.md b/TRANSFORMER_CROSSOVER.md
new file mode 100644
index 0000000..2bcff12
--- /dev/null
+++ b/TRANSFORMER_CROSSOVER.md
@@ -0,0 +1,192 @@
+# Decoder-Transformer local-learning crossover
+
+## Claim and immutable cell scope
+
+This family protocol instantiates the Transformer4/8/12 rows registered in
+`CROSS_ARCHITECTURE_CROSSOVER.md`.  The three depths are three separate
+architecture--scale points, not one averaged "Transformer" result.  Every
+stage retains all nine methods:
+
+`BP, FA, DFA, PEPITA, Forward-Forward, EP, DualProp, clean-KP, SDIL`.
+
+A failed, chance-level, nonfinite, or timed-out cell remains in the table.
+Neither a weak depth-4 selector result nor the cost of a state method permits
+its depth-8/12 cell to be dropped.  Any later width, context, or additional
+depth ladder is a new complete revision containing all nine methods.  It
+cannot selectively extend SDIL or replace these 27 primary endpoints.
+
+## Frozen task, bytes, and forward graph
+
+The task is next-character prediction on the nanoGPT tiny-Shakespeare split.
+The only formal input files are uint16 token arrays:
+
+| split | bytes | tokens | SHA-256 |
+|---|---:|---:|---|
+| train | 2,007,708 | 1,003,854 | `6ec305602a99ac2802745a134e1f5e33e2231b4855525b00b9aebb730ac2626f` |
+| validation | 223,080 | 111,540 | `d37d30cc0c8327c270d493299c3dca54135f6d5f1c9ef60cda78076e311204b1` |
+
+There is no test access in T1.  Every training batch is drawn by the same
+frozen sequence-start generator.  Validation uses deterministic nonoverlap
+windows over the complete validation file.  Each token window has context
+64 and target offset one.  Vocabulary size is 65.
+
+The matched forward graph is a causal-softmax, pre-LayerNorm decoder with
+learned untied token and position embeddings, four attention heads, width
+128, MLP ratio four, no affine bias, no dropout, and an untied vocabulary
+head.  Depth is the only primary architecture variable:
+
+| ID | blocks | forward parameters |
+|---|---:|---:|
+| `transformer4` | 4 | 813,568 |
+| `transformer8` | 8 | 1,602,048 |
+| `transformer12` | 12 | 2,390,528 |
+
+Forward parameters and their initialization are bit-identical within a
+depth--seed pair.  Feedback draws use separate generators and therefore
+cannot shift later forward draws.  The implementation is
+`sdil/transformer.py`; `experiments/transformer_feedback_smoke.py` is the
+required equation gate.
+
+## Frozen method adapters
+
+**BP.** Exact reverse-mode differentiation of training cross-entropy.
+
+**Ordinary FA.** Every parameterized affine transports its recursively
+arriving field through an independent fixed random matrix.  Residual
+addition, causal softmax attention, GELU, and LayerNorm retain their exact
+parameter-free local Jacobians.  FA feedback never reads, copies, or changes
+with a forward weight.  It is not DFA.
+
+**Strict block DFA.** The analytical vocabulary error is independently
+projected to every decoder-block output, the embedding/position population,
+and final LayerNorm.  Every block input is detached and only the explicit
+within-block objective is differentiated.  The vocabulary head uses its
+analytical local delta.  No embedding, normalization, or affine parameter
+retains a hidden BP gradient.
+
+**PEPITA.** The first presentation's analytical vocabulary error is projected
+by a fixed `65 x 128` matrix into the continuous token-embedding stream.  A
+second presentation uses the perturbed embedding.  Each block receives its
+detached first-minus-second output field and second-presentation input.  The
+head receives the second-presentation output error.  Because discrete token
+IDs cannot be perturbed, the token table and positional code use their
+directly observed embedding difference.  This Transformer-specific choice is
+reported as a PEPITA/ERIN adapter, not an assertion that the original paper
+studied language models.  Projection scale is 0.05.
+
+**Forward-Forward.** The true next character forms a positive candidate and a
+uniformly sampled guaranteed-wrong character forms a negative candidate.
+The candidate is overlaid in the first 65 coordinates of each continuous
+token representation, mirroring supervised label overlay.  Token/position
+embedding, every decoder block, and final-LayerNorm-plus-head are greedy
+layers.  Prefixes and non-target layers are detached.  Inputs are normalized
+per token; the local loss is the positive/negative softplus goodness loss
+with threshold 2.  Validation enumerates all 65 candidates and sums goodness
+from greedy layer one onward.  All 65 candidate presentations are charged.
+
+**Dual Propagation.** One plus and one minus state are exposed for the
+embedding population, every decoder-block output, and logits.  They start at
+the ordinary forward state.  With author-style `alpha=0`, `beta=0.1`, 16
+sequential fwK passes update the graph.  A downstream state difference is
+transported by an exact local VJP through that same child block or head.
+This intentional symmetric feedback is a strong, weight-transport-using
+baseline.  Contrastive parameter correlations are evaluated only inside
+detached edges.
+
+**Equilibrium Propagation.** The same state graph has one hard-sigmoid state
+per population.  Synchronous leaky relaxation uses 20 free and four nudged
+steps, `dt=0.5`, nudge magnitude 0.5, and a deterministic random nudge sign
+per batch.  Child activity returns through the exact local VJP of the same
+edge.  Updates are the signed-beta-normalized difference between nudged and
+free local correlations.  Validation uses the 20-step free output state.
+EP's relaxation and symmetric transport are reported separately.
+
+**Clean KP.** The ordinary-FA feedback matrix is plastic.  Its update is
+independently recomputed from the same locally available pre-activity and
+post-field correlation as the forward affine; it is not assigned from a
+forward gradient.  Shared optimizer dynamics and decay contract reciprocal
+misalignment.
+
+**SDIL.** The transport and reciprocal update are exactly clean-KP.  At every
+affine, the apical signal additionally contains soma-predictable traffic with
+four times the task-field RMS.  A paired neutral observation predicts and
+subtracts that component before transport and plasticity.  Thus the clean
+task is a prespecified KP/SDIL identity check while the raw apical field is
+strongly contaminated; raw and innovation RMS are recorded.  The separate
+nuisance ablation, rather than this clean identity, tests the benefit of
+residualization.
+
+## X0 equation gate
+
+Before a formal endpoint, the deterministic smoke must establish:
+
+- bit-identical forward logits and parameter counts for every adapter;
+- tensorwise BP agreement when ordinary-FA feedback is set to its forward
+  matrix;
+- fixed-FA and fixed-DFA/PEPITA projection independence;
+- independently recomputed KP feedback correlations;
+- KP/SDIL gradient identity under paired-neutral subtraction and a
+  measurably contaminated raw field;
+- no cross-block gradient leakage in DFA, PEPITA, or Forward-Forward;
+- target-layer-only Forward-Forward gradients and `B x T x 65` scores;
+- paper-equation agreement for PEPITA, Dual Propagation, and EP;
+- finite state ranges, explicit local-VJP counts, and zero calls to
+  task-loss backward for DFA/PEPITA/DP/EP.
+
+The gate must pass both the development PyTorch environment and the Pascal
+server's frozen PyTorch 2.3 environment.
+
+## T1 validation-only selector
+
+T1 contains 27 immutable depth-4 jobs: every method at all three of its
+predeclared learning rates.  It uses run seed 0, model seed 2027, loader seed
+0, negative seed 5001, EP-sign seed 5002, batch size 32, 1,000 optimizer
+steps, constant learning rate, AdamW `(beta1,beta2)=(0.9,0.95)`, weight decay
+0.1, and the full validation file once at the endpoint.  Each job therefore
+receives exactly 2,048,000 ordinary training tokens.  Forward-Forward spends
+this fixed total greedily in layer order rather than multiplying the token
+budget by its number of layers.
+
+| method | learning-rate grid |
+|---|---|
+| BP, FA, DFA, clean-KP, SDIL | `0.0003, 0.001, 0.003` |
+| PEPITA, Forward-Forward | `0.0001, 0.0003, 0.001` |
+| EP, Dual Propagation | `0.0001, 0.0003, 0.001` |
+
+Selection minimizes finite final validation NLL.  Exact ties choose the
+smaller learning rate.  Nonfinite and timed-out jobs rank after every finite
+job but remain reported.  No intermediate validation, test token, development
+pilot endpoint, or cross-method setting is used by the selector.
+
+The per-job cap is 48 hours.  Only timan107 physical GPUs 5 and 7 are allowed.
+A launch manifest must record the physical GPU UUID, source commit, tracked
+dirty state, environment freeze, dataset hashes, arguments, peak memory,
+wall time, and exit state.  A failure advances the shard rather than deleting
+the cell.
+
+## T2 complete depth crossover
+
+After the selector JSON and its hash are committed, T2 applies each selected
+method setting without a new search to depths 4, 8, and 12: 27 cells.  All
+cells use 5,000 optimizer steps, batch size 32, 100-step linear warmup, cosine
+decay from the selected learning rate to one tenth of it, and the same
+ordinary-token, validation, failure, source, and hardware rules.  Depth does
+not alter DP/EP inference counts, FF threshold, PEPITA scale, traffic ratio,
+or any method variant.
+
+T2 is a single-seed scaling panel, not the final uncertainty estimate.  An
+untouched multi-seed T3 registry must contain every one of the same 27 cells;
+winner-only or SDIL-only replication cannot support the scaling claim.
+
+## Metrics and cost accounting
+
+The primary task metric is full-validation next-token NLL; perplexity and
+token accuracy are secondary.  Each record also reports wall time, peak
+allocated/reserved memory, forward/feedback parameters, ordinary training
+and validation tokens, positive/negative or perturbed presentations,
+candidate-token presentations, relaxation-token passes, local-VJP token
+evaluations, first nonfinite step, and completed optimizer steps.  The
+accuracy--cost plot and the within-family depth plot retain timeouts and
+failures.  A low-cost method is not declared scalable from depth 4 alone, and
+an accurate DP/EP point is not declared local in the weight-transport-free
+sense merely because its VJPs were evaluated at detached edges.