From 66dc2d38ad8187e111843b911d29b9fc23283aa0 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Thu, 28 May 2026 23:19:05 -0500 Subject: Add trajectory ensemble validation --- notes/02_experiment_notes.md | 52 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) (limited to 'notes/02_experiment_notes.md') diff --git a/notes/02_experiment_notes.md b/notes/02_experiment_notes.md index c44e0d0..896961b 100644 --- a/notes/02_experiment_notes.md +++ b/notes/02_experiment_notes.md @@ -342,3 +342,55 @@ Per-seed summary: This is only a smoke trajectory, not yet an ensemble result. It verifies that the logging pipeline can capture loss gaps, surrogate-gradient alignment, and weight-feedback alignment \(Q_l(t)\) from the same run. Important metric note: full-model gradient cosine can be inflated by the output layer, whose gradient is identical under BP and FA. Hidden-only gradient cosine is a sharper metric for feedback-induced mismatch. + +## Trajectory Ensemble Run Log + +Script: + +```bash +python scripts/trajectory_ensemble.py --architectures 16,16 24,24 32,32 48,48 24,24,24 --samples 256 --steps 200 --lr 0.02 --eval-every 20 --feedback-runs 40 --data-seed 20 --init-seed 30 --feedback-seed-start 1000 --plot +``` + +Setup: + +- architectures: `16,16`, `24,24`, `32,32`, `48,48`, `24,24,24` +- feedback seeds per architecture: `40` +- total FA trajectories: `200` +- synthetic full-batch regression +- shared BP baseline per architecture + +Architecture-level results: + +| architecture | BP final loss | FA gap mean | FA gap std | gap min | gap max | final hidden cosine mean | final capacity mean | +|---|---:|---:|---:|---:|---:|---:|---:| +| `h16x16` | `0.549502` | `0.029880` | `0.054179` | `-0.111327` | `0.113590` | `0.305739` | `6.435878` | +| `h24x24` | `0.560063` | `0.187094` | `0.064862` | `0.059168` | `0.316122` | `0.256078` | `7.749068` | +| `h32x32` | `0.427190` | `0.068749` | `0.023786` | `0.029412` | `0.124503` | `0.190019` | `8.956942` | +| `h48x48` | `0.349687` | `0.078056` | `0.018102` | `0.032782` | `0.113509` | `0.196928` | `15.784286` | +| `h24x24x24` | `0.336629` | `0.066436` | `0.020885` | `0.010641` | `0.106398` | `0.008886` | `4.814671` | + +Pooled result: + +- total FA runs: `200` +- final gap mean: `0.086042897` +- final gap std: `0.067060324` +- final gap min: `-0.11132673` +- final gap max: `0.31612214` +- negative-gap FA runs: `9/200`, all in `h16x16` + +Correlation takeaways: + +- Pooled correlations are weak because architecture-level differences dominate and confound the relationship between capacity proxies and gap. +- Within architecture, final hidden-gradient alignment is consistently predictive: larger final hidden cosine corresponds to smaller final FA/BP gap. +- Strongest within-architecture Spearman correlations with final gap: + - `h16x16`: final hidden cosine, \(\rho=-0.4587\), \(p=0.00291\) + - `h24x24`: final capacity, \(\rho=-0.4432\), \(p=0.00419\) + - `h32x32`: final hidden cosine, \(\rho=-0.4477\), \(p=0.00377\) + - `h48x48`: final hidden cosine, \(\rho=-0.5180\), \(p=0.000618\) + - `h24x24x24`: final hidden cosine, \(\rho=-0.7235\), \(p=1.34e-7\) + +Interpretation: + +- Static \(Q\) and capacity proxies alone do not explain pooled loss gaps across architectures. +- Same-architecture seed variation is much better explained by trajectory-level hidden-gradient alignment. +- This supports the current paper framing: static distributional capacity bounds quantify the initial burden, while trajectory bridge metrics are needed to explain realized FA/BP loss gaps. -- cgit v1.2.3