diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/01_theory_notes.md | 28 | ||||
| -rw-r--r-- | notes/02_experiment_notes.md | 50 | ||||
| -rw-r--r-- | notes/03_paper_outline.md | 6 |
3 files changed, 84 insertions, 0 deletions
diff --git a/notes/01_theory_notes.md b/notes/01_theory_notes.md index b22e535..d0ac8da 100644 --- a/notes/01_theory_notes.md +++ b/notes/01_theory_notes.md @@ -492,3 +492,31 @@ If \(\epsilon_t(B)\) is approximately Gaussian under random \(B\), then: \] This should be treated as a bridge approximation, not as a primary architecture-only theorem. + +### First Empirical Bridge Variant + +For a concrete first trajectory-distribution predictor, ignore the Hessian propagation factors and integrate the FA/BP gradient mismatch along the BP path: + +\[ +\widehat{\delta\theta}_T(B) += +-\eta +\sum_{t<T} +\left[ +g_t^{\mathrm{FA}}(\theta_t^{\mathrm{BP}};B) +- +g_t^{\mathrm{BP}}(\theta_t^{\mathrm{BP}}) +\right]. +\] + +Then predict the final FA/BP loss gap by evaluating the loss at the BP endpoint plus this bridge deviation: + +\[ +\widehat{\Delta L}_T(B) += +L(\theta_T^{\mathrm{BP}}+\widehat{\delta\theta}_T(B)) +- +L(\theta_T^{\mathrm{BP}}). +\] + +This is expected to be misscaled when curvature and contraction along the BP path are important. It is still useful as a distributional shape test: if the standardized \(\widehat{\Delta L}_T(B)\) distribution matches the standardized empirical \(\Delta L_T(B)\) distribution, then the bridge captures the random-feedback shape but not yet the absolute scale. A stronger next theorem should include a curvature or response operator replacing the identity propagation used here. diff --git a/notes/02_experiment_notes.md b/notes/02_experiment_notes.md index c7e7168..03666c6 100644 --- a/notes/02_experiment_notes.md +++ b/notes/02_experiment_notes.md @@ -552,3 +552,53 @@ Interpretation: - Static \(Q\) and capacity proxies alone do not explain pooled loss gaps across architectures. - Same-architecture seed variation is much better explained by trajectory-level hidden-gradient alignment. - This supports the current paper framing: static distributional capacity bounds quantify the initial burden, while trajectory bridge metrics are needed to explain realized FA/BP loss gaps. + +## Trajectory Gap Distribution Bridge Run Log + +Script for `h16x16`: + +```bash +python scripts/trajectory_gap_distribution.py --hidden-widths 16 16 --samples 128 --steps 60 --lr 0.02 --feedback-runs 1000 --data-seed 7 --init-seed 8 --feedback-seed-start 10000 --outdir outputs/trajectory_gap_distribution/h16x16_1000 --plot +``` + +Script for `h24x24`: + +```bash +python scripts/trajectory_gap_distribution.py --hidden-widths 24 24 --samples 128 --steps 60 --lr 0.02 --feedback-runs 500 --data-seed 7 --init-seed 8 --feedback-seed-start 20000 --outdir outputs/trajectory_gap_distribution/h24x24_500 --plot +``` + +Bridge predictor: + +\[ +\widehat{\delta\theta}_T(B) += +-\eta +\sum_{t<T} +\left[ +g_{\mathrm{FA}}(\theta_t^{\mathrm{BP}};B) +- +g_{\mathrm{BP}}(\theta_t^{\mathrm{BP}}) +\right], +\] + +\[ +\widehat{\Delta L}_T(B) += +L(\theta_T^{\mathrm{BP}}+\widehat{\delta\theta}_T(B)) +- +L(\theta_T^{\mathrm{BP}}). +\] + +Results: + +| architecture | runs | BP final loss | empirical gap mean | bridge gap mean | empirical std | bridge std | raw KS | standardized KS | standardized p | moment-matched W1 | Spearman | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| `h16x16` | `1000` | `0.604310` | `0.120109` | `0.318266` | `0.041817` | `0.081883` | `0.917` | `0.056` | `0.0869` | `0.00362` | `0.49288` | +| `h24x24` | `500` | `0.572197` | `0.145215` | `0.432834` | `0.055733` | `0.108192` | `0.944` | `0.056` | `0.4135` | `0.00356` | `0.39455` | + +Interpretation: + +- The raw bridge predictor overestimates absolute gap scale by roughly \(2\)-\(3\times\), so the current identity-propagation bridge is not yet a final calibrated theory for \(\Delta L_T\). +- The standardized distributions match surprisingly well: both architectures have standardized KS `0.056`, with non-rejected two-sample tests at these sample sizes. +- The paired seed-level rank signal is nontrivial: Spearman is `0.49288` for `h16x16` and `0.39455` for `h24x24`. +- Current conclusion: the BP-path mismatch bridge captures much of the random-feedback distributional shape but lacks the curvature/contraction operator needed to predict absolute scale. diff --git a/notes/03_paper_outline.md b/notes/03_paper_outline.md index 6055722..40ab86f 100644 --- a/notes/03_paper_outline.md +++ b/notes/03_paper_outline.md @@ -166,6 +166,12 @@ Introduce local approximation: Use this to motivate trajectory experiments. +First bridge experiment: + +- Predict a full loss-gap distribution by integrating FA/BP gradient mismatch along the BP path. +- Compare raw, standardized, and moment-matched predicted gap distributions against large FA trajectory ensembles. +- Current empirical status: standardized shape matches better than raw scale, motivating a response-operator refinement. + ## 8. Experiments Static distribution: |