diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-07 14:51:16 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-07 14:51:16 -0500 |
| commit | 50a31c4e56516b266e93fe8645355e6df03bef79 (patch) | |
| tree | 55b2484f20140a2445a46713caed57d5c088a3ce /ONBOARDING.md | |
| parent | 096eab0b6c0d6db8a245ab242dbadeb13fb30f6e (diff) | |
FLAGSHIP: warm_fast 1.7065 final — beats tuned BP anchor by 0.086, param-matched twin by 0.257
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'ONBOARDING.md')
| -rw-r--r-- | ONBOARDING.md | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md index 4361271..ce6e549 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -29,8 +29,9 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z ## 3. Where we are (results, C512, TinyStories-BPE) - **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the learning rule is validated, not approximate. -- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05; - previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories. +- **Best val cross-entropy 1.7065** (warm-started `ep_warm_fast`, FINISHED 2026-07-07) — **beats the tuned depth-1 + BP reference (1.7921) by 0.086 and the parameterization-matched BP+EMA twin (1.9632, best of 8 configs) by 0.257**: + the equilibrium computation's iteration-depth dividend, measured. Generates coherent children's stories. - **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) — the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5). - The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. |