diff options
Diffstat (limited to 'ONBOARDING.md')
| -rw-r--r-- | ONBOARDING.md | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md index 25f1286..4361271 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -29,10 +29,13 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z ## 3. Where we are (results, C512, TinyStories-BPE) - **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the learning rule is validated, not approximate. -- **Best val cross-entropy 1.9313** (vs a same-parameter BP transformer ~1.79); generates coherent children's stories. -- The recipe **trains stably and matches/approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. -- ⚠️ **The 1.93 number is warm-started** from a stable early checkpoint (`s2000`); a single from-scratch run currently - plateaus at **~2.10** (see §5, the crux). +- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05; + previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories. +- **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) — + the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5). +- The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. +- ⚠️ The warm numbers start from the `s2000` stability-dip checkpoint (mechanism: `docs/campaign/FINDINGS.md` + 2026-07-05 magic-s2000 entry); manufacturing such seeds from scratch is the active experiment (§5). ## 4. The one hard problem (and the paper it spun off) The binding constraint is **NOT the gradient** — it's **forward fixed-point STABILITY during training**. As |