From 47ab7f9777852b5fd70e0dad90557057a3f42c13 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Wed, 29 Jul 2026 20:42:34 -0500 Subject: Suspend positive scaling claims; state the negative result and the pre-registered protocol (user doctrine) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- index.html | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/index.html b/index.html index 5870fb8..b15f65f 100644 --- a/index.html +++ b/index.html @@ -252,8 +252,8 @@ ModelDataEP (val CE)BP twinGap 72M transformer LMFineWeb-Edu, 1.4B tok3.333.29+4-5% ppl - 135M transformer LMFineWeb-Edu, 2.7B toktrained end-to-end, zero instability - events; scaling analysis below + 135M transformer LMFineWeb-Edu, 2.7B toktrains stably end to end; the frozen + 72M recipe does not transfer (gap grows to roughly 30% ppl). See the scaling note below.

Twin discipline: identical architecture, tokenizer, data order, @@ -276,16 +276,18 @@

-

The scaling science

+

The scaling note

-

Scaling a physical learning rule surfaces phenomena backprop never meets. Between widths 512 and 768 - we identified a width-scaling loss in the EP gradient: localized to the top half of the network, - invisible to every per-step alignment metric, and traced to response components that finite nudge - displacement under-reaches. We built a screening instrument that measures this leak in 90 minutes per - candidate recipe, mapped its dose-response law (logarithmic across two decades of displacement - amplification), and demonstrated a pure estimator-side treatment that closes 97% of it: no change to - the model, its inference path, or the cost budget. The same instruments provide the go/no-go protocol - for each next rung of the ladder.

+

Scaling the frozen 72M recipe one width step (512 to 768) fails: the EP to BP gap grows from + 4-5% to roughly 30% perplexity. We treat this as the central result of the current phase and the + central research object. It has sharp structure: the loss is fully localized to the top half of the + network, is invisible to every per-step alignment metric, is untouched by estimator symmetrization, + precision, relaxation depth, and output-head interventions, and is removed completely by + substituting true gradients in the top half. A mechanism-identification campaign across four widths + is in progress, with a pre-registered transfer law and a blind holdout at the next width. Positive + scaling claims are suspended until the law predicts an unseen width with zero tuning.

+
+

Hardware line

@@ -301,11 +303,12 @@

Roadmap

-

Staged scaling with matched BP controls and hardware-relevant ablations at every rung: a 150M-600M - ladder (the gap is measured against a backprop twin at every rung), then 1B-3B; each stage - gated on the previous stage’s loss, alignment, and throughput numbers. In parallel: the - algorithm→regime map across the activity-difference family (contrastive / coupled-learning arms on - the same harness), and a bounded single-column hardware feasibility study.

+

Mechanism first: identify the width-scaling term across four measured widths, freeze a transfer + law with its thresholds and budgets, then validate blind at 270M with from-scratch runs and two + seeds. The scaling ladder resumes only on a zero-tuning hit at the holdout width. In parallel: the + algorithm-to-regime map across the activity-difference family, and a bounded single-column hardware + feasibility study.

+
-- cgit v1.2.3