diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-05 07:06:19 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-05 07:06:19 -0500 |
| commit | 66b4ad978585d1b8008e02d06787811e2ee93da7 (patch) | |
| tree | dc7bb6cae6f98564b043189fed4a8bbf44eccb1f /ONBOARDING.md | |
| parent | 1e5619cf4f3e45acc880f4eef0562e967f7fe39e (diff) | |
ONBOARDING: from-scratch 2.0119 (self_restart DONE), warm best 1.8977 in flight
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'ONBOARDING.md')
| -rw-r--r-- | ONBOARDING.md | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md index 25f1286..4361271 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -29,10 +29,13 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z ## 3. Where we are (results, C512, TinyStories-BPE) - **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the learning rule is validated, not approximate. -- **Best val cross-entropy 1.9313** (vs a same-parameter BP transformer ~1.79); generates coherent children's stories. -- The recipe **trains stably and matches/approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. -- ⚠️ **The 1.93 number is warm-started** from a stable early checkpoint (`s2000`); a single from-scratch run currently - plateaus at **~2.10** (see §5, the crux). +- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05; + previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories. +- **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) — + the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5). +- The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. +- ⚠️ The warm numbers start from the `s2000` stability-dip checkpoint (mechanism: `docs/campaign/FINDINGS.md` + 2026-07-05 magic-s2000 entry); manufacturing such seeds from scratch is the active experiment (§5). ## 4. The one hard problem (and the paper it spun off) The binding constraint is **NOT the gradient** — it's **forward fixed-point STABILITY during training**. As |
