diff options
| -rw-r--r-- | NOTE.md | 98 | ||||
| -rw-r--r-- | protocol/EVIDENCE_SUMMARY.md | 26 |
2 files changed, 124 insertions, 0 deletions
@@ -1,5 +1,103 @@ # Experiment Notes +## Current Status (2026-04-23): Paper v2.39 + FA experiments complete + +Paper: `paper/main.tex` (v2.39, commit `bcfec1d`). NeurIPS 2026 E&D track. +FA implementation + results: commit `88ff85c`. + +### Paper structure (v2.39) + +7 sections with 14 subsections. Key sections: +- §3 Mode 1: Measurement Degeneracy (residual-stream growth + LN gradient collapse) +- §4 Mode 2: Credit Quality (the paper's main contribution) + - §4.2 Functional triangulation: 3 metrics (accuracy, nudging, training-loss) rank SB ≫ CB ≈ DFA; only deep cos disagrees + - §4.3 Mode 2 → Mode 1 causal hypothesis: poor credit quality drives activation growth +- §5 Intervention + cross-arch evidence +- §6 4-diagnostic evaluation protocol + +### 6-method audit (4-block d=256 pre-LN ResMLP, CIFAR-10, 100ep, 3 seeds) + +| Method | Test Acc | Deep Cos | ‖h_L‖ | ‖g_L‖ | Mode 1(b)? | +|--------|----------|----------|--------|--------|------------| +| **BP** | 0.615 ± 0.004 | ≈ 1.0 | ~200 | ~4e-4 | no | +| **FA** (NEW) | **0.401 ± 0.009** | **+0.33** | **~1e5** | **~1e-6** | **no** | +| EP | 0.316 ± 0.037 | 0.008 | ~3e3 | ~1e-4 | no | +| DFA | 0.306 ± 0.008 | ~0 | ~5e8 | ~4e-10 | **YES** | +| CB | 0.289 ± 0.031 | 0.07 | ~7e7 | ~2e-9 | **YES** | +| SB | 0.205 ± 0.039 | 0.005 | ~2e8 | ~2e-9 | **YES** | +| Frozen baseline | 0.349 ± 0.003 | — | — | — | — | + +### FA vs DFA: the key comparison + +**Same local loss form** ⟨f_l, a_l⟩, same architecture, same optimizer. +**Only difference**: how a_l is computed. +- DFA: a_l = B_l^T @ e_T (direct projection, d×C random matrix) +- FA: a_l = B_l @ a_{l+1} (sequential backward, d×d random matrix, starts from exact ∂CE/∂h_L) + +| | FA | DFA | +|---|---|---| +| Test acc | **0.401** (+5.2 pp above frozen) | 0.306 (-4.3 pp below frozen) | +| Deep cos 100ep | **+0.33** (genuine) | ~0 (degenerate) | +| Deep cos 5ep | **+0.32** (from the start) | -0.008 (null) | +| ‖h_L‖ | ~10⁵ | ~5×10⁸ (3 OOM larger) | +| ‖g_L‖ | ~10⁻⁶ (meaningful) | ~10⁻¹⁰ (floor) | +| Mode 1(b) fires? | **NO** | YES | +| FA random-target h_L | ~1.3e5 | ~1.7e8 (DFA) | + +**Interpretation**: FA's sequential propagation preserves enough credit-direction quality to prevent the catastrophic activation growth that DFA exhibits. This is the **strongest empirical support for the Mode 2 → Mode 1 causal hypothesis**: better credit → less growth → no gradient-floor collapse. + +### FA depth sweep (d=512, 100ep, s42) + +| L | FA acc | DFA acc | FA deep cos | DFA deep cos | +|---|--------|---------|-------------|--------------| +| 2 | 0.350 | n/a | +0.96 | n/a | +| 4 | 0.424 | n/a | +0.29 | n/a | +| 6 | 0.401 | n/a | +0.16 | n/a | +| 8 | 0.409 | 0.306 | +0.11 | -0.0001 | +| 12 | 0.404 | 0.309 | +0.09 | -0.0001 | + +FA deep cos decreases with depth but stays positive. DFA is ~0 everywhere. + +### Penalty rescue comparison (30ep, d=256, 3 seeds) + +| | FA no-pen | FA+pen λ=1e-2 | DFA no-pen | DFA+pen λ=1e-2 | +|---|---|---|---|---| +| acc | 0.372 ± 0.007 | 0.369 ± 0.003 | 0.301 ± 0.006 | 0.360 ± 0.002 | + +Penalty barely helps FA (it doesn't need rescue) but substantially helps DFA (+5.9 pp). + +### Cross-method functional triangulation (§4.2, penalty-rescued d=256 30ep) + +| Metric | SB+pen | CB+pen | DFA+pen | Ordering | +|--------|--------|--------|---------|----------| +| Accuracy | **0.453** | 0.360 | 0.360 | SB ≫ CB ≈ DFA | +| Nudging (loss Δ) | **-1.93e-3** | -4.26e-4 | -4.98e-5 | SB ≫ CB ≈ DFA | +| Training loss decrease | **-0.447** | -0.121 | -0.095 | SB ≫ CB ≈ DFA | +| Deep cos | +0.322 | **+0.679** | +0.151 | CB > SB > DFA | + +3 functional metrics agree SB ≫ CB ≈ DFA. Deep cos is the ONLY one that disagrees. + +### Auditable JSON sources + +All paper numbers derive from saved files in `results/`. See `reproduce_all.ipynb` for the complete source map. Key files: +- `results/protocol_audit/audit_table_s42_s123_s456.json` — Table 1 +- `results/fa_main_audit/results_cifar10.json` — FA main results +- `results/nudging_test_3seed_summary.json` — §4.2 nudging +- `results/training_loss_decrease_3seed.json` — §4.2 loss trajectory +- `results/matched_30ep_control_summary.json` — §5.2 BP+DFA controls +- All ± values use ddof=1 (sample std with Bessel correction) as of v2.38. + +### Open items + +1. **FA results not yet in the paper** — need to add FA as a 6th method row in Table 1 and discuss in §3-§5. +2. **FA on ViT-Mini / CNN**: experiment scripts (`snapshot_evolution_vit.py`, `cnn_baseline.py`) don't yet support FA. Need implementation. +3. **§7 falsification test**: the FA result itself is a strong empirical test of Mode 2 → Mode 1. Could frame FA as the "better credit quality at fixed ‖f‖" control proposed in §7.2. +4. **Codex restructure** (v2.39): applied codex's subsection proposal. 14 subsections across 7 sections. + +--- + +## Historical Notes (pre-v2.xx, kept for reference) + ## Experiment Phases - **debug**: Initial implementation, rapid iteration (commits ce24e36) - **pilot**: Controlled iteration (commits 0b9ebb2, 7baf7ae) diff --git a/protocol/EVIDENCE_SUMMARY.md b/protocol/EVIDENCE_SUMMARY.md index bab8764..d6d3945 100644 --- a/protocol/EVIDENCE_SUMMARY.md +++ b/protocol/EVIDENCE_SUMMARY.md @@ -129,6 +129,32 @@ on deep layers. **Caught by direct per-layer cosine measurement.** | BP | 0.585 ± 0.001 | **0.532 ± 0.006** | −5.3 pp (capacity loss) | | DFA | 0.301 ± 0.005 | 0.360 ± 0.001 | +5.9 pp (rescue) | +### Vanilla FA vs DFA (2026-04-22, commit 88ff85c) + +**PAPER-CHANGING FINDING.** FA (Lillicrap 2016 sequential backward with d×d random matrices) is dramatically different from DFA on the same architecture. + +| | FA | DFA | +|---|---|---| +| Test acc (100ep, 3-seed, d=256) | **0.401 ± 0.009** | 0.306 ± 0.008 | +| vs frozen 0.349 | **+5.2 pp above** | -4.3 pp below | +| Deep cos | **+0.33** | ~0 (degenerate) | +| ‖h_L‖ | ~10⁵ | ~5×10⁸ | +| ‖g_L‖ | ~10⁻⁶ (meaningful) | ~10⁻¹⁰ (floor) | +| Mode 1(b) fires? | **NO** | YES | + +Same local loss ⟨f_l, a_l⟩, same architecture, same optimizer. Only difference: how a_l is computed (sequential vs direct projection). FA's sequential backward preserves credit quality → prevents catastrophic Mode 1 growth. **Strongest empirical support for Mode 2 → Mode 1 causal hypothesis.** + +Source: `results/fa_main_audit/results_cifar10.json` + +FA depth sweep (d=512, 100ep, s42): +| L | FA acc | FA deep cos | DFA acc | DFA deep cos | +|---|---|---|---|---| +| 2 | 0.350 | +0.96 | — | — | +| 4 | 0.424 | +0.29 | — | — | +| 6 | 0.401 | +0.16 | — | — | +| 8 | 0.409 | +0.11 | 0.306 | ~0 | +| 12 | 0.404 | +0.09 | 0.309 | ~0 | + ### Round 20 phrasing for the gap **Lower bound on non-capacity gap**: matched penalty controls show that only part of DFA's deficit is attributable to the representational/optimization cost of the penalty itself; a substantial residual remains and is consistent with poorer credit assignment. |