From 610e1169e19378cccd2d9b92a588c24dca7f3df7 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Thu, 26 Mar 2026 16:27:53 -0500 Subject: Add Phase 10A.5: blend gain is implicit regularization, not learned credit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dissection of 6 branches from same DFA checkpoint: - blend_random_frozen: 12.6% (CATASTROPHIC — frozen noise destroys training) - blend_random_trainable: 32.2% (+1.2% — trainable network helps) - blend_shuffled_trainable: 32.5% (+1.4% — even wrong targets work!) - blend_gaussian_noise: 30.8% (neutral) - scaled_DFA_norm_match: 31.0% (neutral) The gain comes from implicit regularization through a co-optimized auxiliary network, NOT from learned credit quality. Phase 9A's +1.5% was an optimization dynamics effect, not evidence of useful credit assignment. Co-Authored-By: Claude Opus 4.6 (1M context) --- NOTE.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) (limited to 'NOTE.md') diff --git a/NOTE.md b/NOTE.md index e6ba41c..62deba6 100644 --- a/NOTE.md +++ b/NOTE.md @@ -5,7 +5,7 @@ - **pilot**: Controlled iteration (commits 0b9ebb2, 7baf7ae) - **frozen**: Code at commit 0b9ebb2 for all reported results -## Status: PHASE 10A — NO PREFIT THRESHOLD, BLEND ITSELF IS THE ACTIVE INGREDIENT +## Status: PHASE 10A.5 — BLEND GAIN IS IMPLICIT REGULARIZATION, NOT LEARNED CREDIT --- @@ -570,5 +570,23 @@ The +1.5% gain from 9A's blend(0.75) at t0=5 is the project's best online result Frozen Gamma/rho are near zero at all prefit levels. The benefit comes from the blend mechanism itself — blending DFA with any additional signal provides regularization/diversification. +### Phase 10A.5: Blend Mechanism Dissection + +| Branch | final acc | diff vs DFA | +|--------|-----------|-------------| +| continue_DFA | 0.311 | baseline | +| blend_random_**frozen** | **0.126** | **-18.5%** (catastrophic) | +| blend_random_**trainable** | 0.322 | +1.2% | +| blend_shuffled_trainable | 0.325 | +1.4% | +| blend_gaussian_noise | 0.308 | -0.3% | +| scaled_DFA_norm_match | 0.310 | -0.0% | + +**Mechanism identified**: The gain is from **implicit regularization through a trainable +auxiliary network**, NOT from learned credit. Frozen random Vec crashes (12.6%). +Trainable Vec helps even with shuffled targets. Gaussian noise and norm scaling don't help. + +Phase 9A's +1.5% was not evidence of useful credit — it was an optimization dynamics effect. + ### Experiment IDs (Phase 10) - `prefit_threshold/`: Phase 10A prefit threshold curve +- `blend_dissection/`: Phase 10A.5 blend mechanism dissection -- cgit v1.2.3