From 50a31c4e56516b266e93fe8645355e6df03bef79 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Tue, 7 Jul 2026 14:51:16 -0500 Subject: =?UTF-8?q?FLAGSHIP:=20warm=5Ffast=201.7065=20final=20=E2=80=94=20?= =?UTF-8?q?beats=20tuned=20BP=20anchor=20by=200.086,=20param-matched=20twi?= =?UTF-8?q?n=20by=200.257?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn --- ONBOARDING.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) (limited to 'ONBOARDING.md') diff --git a/ONBOARDING.md b/ONBOARDING.md index 4361271..ce6e549 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -29,8 +29,9 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z ## 3. Where we are (results, C512, TinyStories-BPE) - **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the learning rule is validated, not approximate. -- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05; - previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories. +- **Best val cross-entropy 1.7065** (warm-started `ep_warm_fast`, FINISHED 2026-07-07) — **beats the tuned depth-1 + BP reference (1.7921) by 0.086 and the parameterization-matched BP+EMA twin (1.9632, best of 8 configs) by 0.257**: + the equilibrium computation's iteration-depth dividend, measured. Generates coherent children's stories. - **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) — the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5). - The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. -- cgit v1.2.3