From ef5bd494087a46ee80d8bc17796074efdae81ff4 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Wed, 25 Mar 2026 10:23:19 -0500 Subject: Add Phase 7A: snapshot time sweep shows early snapshots have positive held-out transfer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit At epoch 5 (acc=49%), Vec_M4 5-step: dL_held=-0.005 (PUR=0.70) Oracle BP 5-step: dL_held=-0.009 (PUR=1.05) DFA 5-step: dL_held=+0.003 (always hurts held-out) By epoch 20, generalization window closes. Held-out failure is late-snapshot artifact. Better credit → lower update variance (Vec=0.8 vs DFA=40), not higher. Key implication: DFA warmup delays credit bridge past its useful window. Credit should be used from epoch 0, not after 20% warmup. Co-Authored-By: Claude Opus 4.6 (1M context) --- NOTE.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) (limited to 'NOTE.md') diff --git a/NOTE.md b/NOTE.md index a57fd30..892cf1e 100644 --- a/NOTE.md +++ b/NOTE.md @@ -5,7 +5,7 @@ - **pilot**: Controlled iteration (commits 0b9ebb2, 7baf7ae) - **frozen**: Code at commit 0b9ebb2 for all reported results -## Status: PHASE 6.5 PROTOCOL AUDIT — PHASE 6A CONCLUSION REVISED +## Status: PHASE 7A SNAPSHOT TIME SWEEP — EARLY SNAPSHOTS SHOW POSITIVE TRANSFER --- @@ -418,3 +418,34 @@ gradient noise) could make better credit usable. ### Experiment IDs (Phase 6.5) - `exploit_linesearch/`: Phase 6.5A smoke test (Oracle + Vec, last1, raw) - `exploit_linesearch_full/`: Phase 6.5A full sweep (all methods, ranges, norm modes) + +--- + +## Phase 7A: Snapshot Time Sweep + +**Setup**: BP snapshots at epoch {5, 20, 100} (acc 0.49/0.57/0.62). +Train Vec_M4 on each frozen snapshot. Test 1-step and 5-step with raw credit, last-block-only. + +**KEY FINDING: Held-out failure is primarily a LATE-SNAPSHOT artifact.** + +5-step DeltaLoss held-out: + +| Epoch | DFA dL_held | Vec dL_held | Oracle dL_held | Vec PUR | +|-------|-------------|-------------|----------------|---------| +| **5** | +0.003 | **-0.005** | **-0.009** | **0.70** | +| 20 | +0.001 | +0.002 | +0.000 | -3.87 | +| 100 | +0.000 | +0.001 | -0.001 | -1.01 | + +At epoch 5: Vec decreases held-out loss (PUR=0.70), Oracle too (PUR=1.05). +DFA INCREASES held-out at all snapshots. + +By epoch 20 the generalization window closes. + +**Better credit produces MORE consistent updates** (Vec variance=0.8 vs DFA variance=40). +The problem is not batch-specificity but snapshot timing: credit is useful early, useless late. + +**Implication**: The DFA warmup (which delays credit bridge to epoch ~20) is counterproductive. +Credit bridge should be used from epoch 0. + +### Experiment IDs (Phase 7) +- `snapshot_time/`: Phase 7A snapshot time sweep with BP checkpoints -- cgit v1.2.3