blob: 3634cbab262942b7dfacd1cda453579e9cc24778 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
# Phase 10A.5 Memo: Blend Mechanism Dissection
**Date**: 2026-03-26
## Question
Phase 10A's gain from blend(random Vec, DFA) — is it learned correction or blend mechanism?
## Answer: Neither. It's implicit regularization from a trainable auxiliary network.
| Branch | final acc | diff vs DFA |
|--------|-----------|-------------|
| continue_DFA | 0.311 | baseline |
| blend_random_**frozen** | **0.126** | **-18.5%** |
| blend_random_**trainable** | 0.322 | +1.2% |
| blend_shuffled_trainable | 0.325 | +1.4% |
| blend_gaussian_noise | 0.308 | -0.3% |
| scaled_DFA_norm_match | 0.310 | -0.0% |
## Key findings
1. **Frozen random Vec destroys training** (12.6%). A fixed random direction blended at 75% is catastrophic. This rules out "any signal diversification helps."
2. **Trainable Vec helps** even from random init (+1.2%), even with shuffled targets (+1.4%). The Vec network doesn't need to learn correct credit — it just needs to be trainable.
3. **Gaussian noise and norm scaling are neutral.** The mechanism is NOT noise injection or step-size calibration.
4. **Gamma/rho stay near zero** for all trainable branches throughout training. The Vec never learns semantically correct credit.
## Mechanism
The gain comes from **implicit regularization through a co-optimized auxiliary network**. The Vec network, even with wrong training targets, adjusts its outputs during training in a way that smoothly regularizes the block-local updates. This is analogous to how auxiliary tasks in multi-task learning can improve main task performance even when the auxiliary task is unrelated — the shared optimization dynamics provide implicit regularization.
## Implications
1. **The Phase 9A narrative was wrong**: the +1.5% was NOT from Vec learning useful credit
2. **The credit bridge hypothesis is not validated by online results**: the gain has nothing to do with credit quality
3. **The gain is real but has a different cause**: it's an optimization dynamics phenomenon, not a credit assignment phenomenon
4. **This does NOT invalidate the frozen CIFAR results**: Vec truly learns better credit on frozen features. But that credit quality doesn't transfer to online improvement — the online improvement comes from a different mechanism entirely.
|
