summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorYuren Hao <yurenh2@illinois.edu>2026-07-05 07:06:19 -0500
committerYuren Hao <yurenh2@illinois.edu>2026-07-05 07:06:19 -0500
commit66b4ad978585d1b8008e02d06787811e2ee93da7 (patch)
treedc7bb6cae6f98564b043189fed4a8bbf44eccb1f
parent1e5619cf4f3e45acc880f4eef0562e967f7fe39e (diff)
ONBOARDING: from-scratch 2.0119 (self_restart DONE), warm best 1.8977 in flight
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
-rw-r--r--ONBOARDING.md11
1 files changed, 7 insertions, 4 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md
index 25f1286..4361271 100644
--- a/ONBOARDING.md
+++ b/ONBOARDING.md
@@ -29,10 +29,13 @@ F(z) = −(z − x_in) + Attn(LN(z)) + FFN(LN(z)) − c·z
## 3. Where we are (results, C512, TinyStories-BPE)
- **EP gradient ≈ exact BPTT gradient** (cosine ≈ 0.92–0.99 per component when the free phase is converged) — the
learning rule is validated, not approximate.
-- **Best val cross-entropy 1.9313** (vs a same-parameter BP transformer ~1.79); generates coherent children's stories.
-- The recipe **trains stably and matches/approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block.
-- ⚠️ **The 1.93 number is warm-started** from a stable early checkpoint (`s2000`); a single from-scratch run currently
- plateaus at **~2.10** (see §5, the crux).
+- **Best val cross-entropy 1.8977** (warm-started, `ep_warm_fast`, still descending pre-lr-decay as of 2026-07-05;
+ previous era best 1.9313) vs a same-parameter BP transformer ~1.79; generates coherent children's stories.
+- **From scratch: 2.0119** (`ep_self_restart`, a from-scratch lineage with restart-on-abort; finished 2026-07-05) —
+ the old "~2.10 plateau" is broken, but the warm/scratch gap (~0.11) remains the crux (see §5).
+- The recipe **trains stably and approaches BP** at this scale. Model = C512 / H16 / T256, damped DEQ block.
+- ⚠️ The warm numbers start from the `s2000` stability-dip checkpoint (mechanism: `docs/campaign/FINDINGS.md`
+ 2026-07-05 magic-s2000 entry); manufacturing such seeds from scratch is the active experiment (§5).
## 4. The one hard problem (and the paper it spun off)
The binding constraint is **NOT the gradient** — it's **forward fixed-point STABILITY during training**. As