diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/CLAIM_LARGEST_NOBP.md | 55 |
1 files changed, 55 insertions, 0 deletions
diff --git a/docs/CLAIM_LARGEST_NOBP.md b/docs/CLAIM_LARGEST_NOBP.md new file mode 100644 index 0000000..ace8678 --- /dev/null +++ b/docs/CLAIM_LARGEST_NOBP.md @@ -0,0 +1,55 @@ +# Claim verification: "largest neural network trained without backpropagation" + +**Verified 2026-07-11** (web sweep + primary-source close reading). Status: **the claim holds today +at 42.75M with one qualifier; holds unqualified at 1B+; is uncontested-by-an-order-of-magnitude at 3-7B.** + +## The claim (Scellier-proof wording) + +> "the largest neural network ever trained **end-to-end without backpropagation** — every parameter +> learned by a local two-phase rule, at quality parity with a tuned backprop twin, on a standard +> transformer whose inference is a plain forward pass, with every **trained** component +> analog-compatible." + +Wording hygiene: (a) "end-to-end / every parameter" is the load-bearing qualifier (see optical DFA +below); (b) say "every *trained* component analog-compatible" — optimizer state is digital-side by +standing doctrine, same as every analog-training proposal; (c) run record attempts with +`--gate_every 0` (fully BP-free mode, formally audited by `ep_run/test_bp_free.py`). + +## Competitor autopsy (primary sources) + +| Claimant | What actually happened | Why it does not hold the title | +|---|---|---| +| **Optical DFA** — LightOn-line, arXiv:2409.12965 ("Optical training of large-scale Transformers") | 1.07B GPT-2-shape LM | **Partial coverage**: only ~400M of 1.07B params received ODFA signals; **the last layer is trained by BP** (DFA structurally broadcasts a BP-computed top error through fixed random matrices — a global error broadcast, not a local rule); corpus = Cornell Movie-Dialogs, **1.2M tokens, vocab 1016, context 24**; authors: quality "remains behind the state of the art", "lack[s] logical connections" | +| **EGGROLL / ES** — arXiv:2511.16652 (Oxford/FLAIR/Mila/**NVIDIA**) | "billion-parameter" throughput + RWKV7 experiments | Billion-scale = **fine-tuning of BP-pretrained RWKV7** (post-training). The from-scratch component is **EGG: a D256-L6 integer GRU (~1-2M params), char-level MiniPile, 3.40 bits/byte** — not a transformer, weak quality | +| ES fine-tuning — arXiv:2509.24372 (Cognizant) | full-param ES fine-tuning of billion LLMs | Fine-tuning only; no pretraining | +| DFA original — arXiv:2006.12878 (Launay 2020) | transformers among tasks | Tens-of-M scale; authors report a DFA-vs-BP gap specifically on transformers | +| Forward-Forward / PC / Mono-Forward / CaFo line | MNIST/CIFAR-class | <10M, no LM | +| Physical NN — Science 2023 (adi8474, PhyLL) | physical substrates | Small scale | +| EP line record (Laborieux et al. 2021) | ImageNet-32 ConvNets | ~10-20M; **our 42.75M full-epoch already exceeds the EP-line record** | + +## What we hold at each scale + +1. **Today (42.75M, done 2026-07-10/11):** largest **fully** backprop-free (all parameters, no BP + anywhere in the training loop incl. readout) language model; only one at quality parity with a + tuned BP twin (4k matched-tuning parity; epoch-scale gap 0.04 with Muon+floor-ramp, stage1b); + only one generating coherent text; standard OLMo2-form transformer, plain-forward inference. + The only larger *number* in the room (optical DFA 1.07B) is partial + BP-readout + toy corpus. +2. **At 1B+:** largest without qualifiers — beats the partial claim on its own terms. +3. **At 3-7B:** an order of magnitude past every claim including partial ones. "EVER" grade. + +## Differentiation axes (for related-work / rebuttals) + +all-parameters coverage · zero BP in the loop (formally audited mode) · standard architecture · +standard inference (plain forward) · quality parity vs tuned BP twin · coherent generation · +gradient exactness (cos ~= 1 vs autograd reference) · analog-compatible trained components +(OLMo2 block analog audit, docs/hardware/HW_RESEARCH_FINDINGS.md). + +## Sources + +- https://arxiv.org/abs/2409.12965 (+ html v1 experimental section, read 2026-07-11) +- https://arxiv.org/abs/2511.16652 + https://eshyperscale.github.io/ +- https://arxiv.org/abs/2509.24372 +- https://arxiv.org/abs/2006.12878 +- https://www.science.org/doi/10.1126/science.adi8474 +- https://github.com/UbiquitousLearning/Backpropagation_Free_Training_Survey +- https://arxiv.org/abs/2511.01061 (forward-only rigorous evaluation, 2025) |