diff options
Diffstat (limited to 'ONBOARDING.md')
| -rw-r--r-- | ONBOARDING.md | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md index 4361271..ce6e549 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -29,8 +29,9 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z ## 3. Where we are (results, C512, TinyStories-BPE) - **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the learning rule is validated, not approximate. -- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05; - previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories. +- **Best val cross-entropy 1.7065** (warm-started `ep_warm_fast`, FINISHED 2026-07-07) — **beats the tuned depth-1 + BP reference (1.7921) by 0.086 and the parameterization-matched BP+EMA twin (1.9632, best of 8 configs) by 0.257**: + the equilibrium computation's iteration-depth dividend, measured. Generates coherent children's stories. - **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) — the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5). - The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block. |