path: root/notes/02_experiment_notes.md

# Experiment Notes

## Experiment Layer 1: Static Distribution Validation

Purpose: verify the beta law before training dynamics enter.

Procedure:

1. Choose matrix shape \(n_l\times n_{l+1}\), so \(D_l=n_l n_{l+1}\).
2. Sample \(A_l\) and \(B_l\) independently from isotropic distributions.
3. Compute:

\[
Q_l=
\frac{
\langle A_l,B_l\rangle_F^2
}{
\|A_l\|_F^2\|B_l\|_F^2
}.
\]

4. Compare empirical distribution to:

\[
\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]

Metrics:

- Histogram overlay.
- QQ plot.
- KS statistic.
- Tail calibration:

\[
\Pr(Q_l\ge q).
\]

Initialization variants:

- Gaussian.
- Rademacher.
- Uniform sphere.
- Orthogonal or semi-orthogonal.
- Sparse.
- Low-rank.
- Block-diagonal.

Expected:

- Dense isotropic variants match beta law.
- Structured variants deviate in predictable ways.

## Experiment Layer 2: Scaling Validation

Purpose: verify capacity scaling with depth and width.

Compute:

\[
C_l(q)=
-\log
\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]

Total:

\[
C_{\mathrm{all}}=\sum_l C_l(q_l).
\]

Sweeps:

- Width \(n\).
- Depth \(L\).
- Threshold \(q\).
- Threshold regime \(q=c/D_l\).
- Feedback rank.
- Feedback sparsity.

Predictions:

Fixed \(q\):

\[
C_{\mathrm{all}}=\Theta(Ln^2)
\]

for equal-width MLPs.

Chance-level \(q=c/D_l\):

\[
C_{\mathrm{all}}=\Theta(L).
\]

## Experiment Layer 3: Local Gradient Alignment

Purpose: connect static matrix alignment to update direction mismatch.

During training, record:

\[
\Gamma_t=
\frac{
\langle g_t^{\mathrm{BP}},g_t^{\mathrm{FA}}\rangle
}{
\|g_t^{\mathrm{BP}}\|\|g_t^{\mathrm{FA}}\|
}.
\]

Layerwise:

\[
\Gamma_{l,t}=
\frac{
\langle g_{l,t}^{\mathrm{BP}},g_{l,t}^{\mathrm{FA}}\rangle
}{
\|g_{l,t}^{\mathrm{BP}}\|\|g_{l,t}^{\mathrm{FA}}\|
}.
\]

Also record:

\[
Q_l(t)=\cos^2(W_{l+1}(t)^\top,B_l).
\]

Questions:

- Does \(Q_l(0)\) match the beta baseline?
- Does \(Q_l(t)\) shift right during the alignment phase?
- Does gradient alignment improve before memorization or loss reduction?

## Experiment Layer 4: Trajectory Ensemble

Purpose: validate whether capacity proxies explain FA/BP training gaps.

For each architecture and dataset:

1. Fix data seed and model architecture.
2. Train BP baseline.
3. Train many FA runs over feedback seeds \(B\).
4. Record:

\[
\Delta L_T(B)=L_T^{\mathrm{FA}}(B)-L_T^{\mathrm{BP}},
\]

\[
\Delta A_T(B)=A_T^{\mathrm{BP}}-A_T^{\mathrm{FA}},
\]

\[
C_{\mathrm{all}}(B,t),
\quad
\Gamma_t(B),
\quad
Q_l(t).
\]

Datasets:

- Synthetic Gaussian regression.
- MNIST MLP.
- Fashion-MNIST MLP.
- CIFAR-10 flattened MLP, optional later.

Architectures:

- Equal-width MLPs.
- Width sweep.
- Depth sweep.
- Narrow bottleneck sweep.

Expected:

- Overparameterized regimes: large parameter-volume cost can coexist with small functional gap.
- Near redundancy exhaustion: FA/BP gap should increase sharply.
- Poor feedback conditioning can worsen trajectory gap even when angular minimax bound is unchanged.

## Plots

Static:

- Histogram and beta density.
- QQ plot.
- Tail probability calibration.

Scaling:

- \(C_{\mathrm{all}}\) vs \(Ln^2\).
- \(-\log p_{\mathrm{all}}\) vs depth.
- Scaling collapse for \(D_lQ_l\Rightarrow \chi_1^2\).

Trajectory:

- \(Q_l(t)\) over training.
- \(\Gamma_t\) over training.
- \(\Delta L_T\) vs capacity proxy.
- \(\Delta L_T\) vs conditioning proxy.
- Phase transition plot against \(k-(P-d)\).

## Implementation Notes

Start with NumPy or PyTorch scripts that do not require full training.

First script target:

- Sample \(A,B\).
- Compute \(Q\).
- Save empirical moments and KS statistic.
- Produce beta overlay plots.

Only after this is clean, add FA/BP training loops.

## Baseline Run Log

Script:

```bash
python scripts/static_alignment_beta.py --rows 16 --cols 16 --samples 20000 --seed 7 --plot
```

Result:

- \(D=256\)
- empirical mean: `0.0039265884`
- theoretical mean: `0.00390625`
- empirical variance: `3.0311252e-05`
- theoretical variance: `3.0162723e-05`
- KS statistic: `0.00526931`
- KS p-value: `0.633285`

This is a clean first-pass validation for the isotropic Gaussian case.

## Scaling Run Log

Script:

```bash
python scripts/capacity_scaling.py --plot
```

Default sweep:

- widths: `16, 32, 64, 128`
- feedback-aligned layer counts: `1, 2, 4, 8, 16`
- fixed threshold: \(q=0.01\)
- chance-level threshold: \(q=1/D\)
- log unit: nats

Result:

- rows written: `40`
- fixed-threshold max total cost: `1361.74` nats at width `128`, layers `16`
- chance-level max total cost: `18.3652` nats at width `128`, layers `16`

This cleanly separates the fixed-threshold regime, where total cost scales like \(Ln^2\), from the chance-level regime, where per-layer cost is nearly width-independent.

## Large Empirical Capacity Validation Run Log

Script:

```bash
python scripts/capacity_empirical_validation.py --dimensions 64 128 256 512 1024 2048 4096 --samples 100000 --batch-size 2048 --seed 123 --plot
```

Setup:

- dimensions \(D\): `64, 128, 256, 512, 1024, 2048, 4096`
- samples per dimension: `100000`
- total \(Q\) samples: `700000`
- sampler: fixed-target Gaussian direction, using rotational invariance
- chance thresholds: \(q=c/D\), \(c\in\{0.5,1,2,5,10\}\)
- fixed thresholds: \(q\in\{0.001,0.002,0.005,0.01\}\)
- multilayer all-event validation: \(c\in\{0.5,1\}\), layers \(L\in\{1,2,4,8\}\)

Distribution calibration:

| \(D\) | empirical mean | theory mean | KS statistic | empirical q99 | theory q99 |
|---:|---:|---:|---:|---:|---:|
| `64` | `0.0156793` | `0.0156250` | `0.0034949` | `0.1007135` | `0.1007080` |
| `128` | `0.0078528` | `0.0078125` | `0.0019669` | `0.0511649` | `0.0510968` |
| `256` | `0.0039289` | `0.0039062` | `0.0029576` | `0.0259898` | `0.0257333` |
| `512` | `0.0019553` | `0.0019531` | `0.0016418` | `0.0129654` | `0.0129127` |
| `1024` | `0.0009706` | `0.0009766` | `0.0031219` | `0.0063694` | `0.0064679` |
| `2048` | `0.0004882` | `0.0004883` | `0.0031095` | `0.0032339` | `0.0032368` |
| `4096` | `0.0002444` | `0.0002441` | `0.0024262` | `0.0016322` | `0.0016191` |

Capacity-tail calibration:

- chance-threshold rows: `35/35` had expected hits \(\ge 20\)
- max absolute smoothed cost error over all chance rows: `0.2023` nats
- mean absolute smoothed cost error over all chance rows: `0.01915` nats
- for chance rows with expected hits \(\ge 500\): max error `0.02771` nats, mean error `0.00607` nats

Multilayer product-capacity calibration:

- rows with expected hits \(\ge 20\): `49`
- max absolute smoothed total-cost error: `0.45788` nats
- mean absolute smoothed total-cost error: `0.03880` nats
- for rows with expected hits \(\ge 500\): max error `0.04319` nats, mean error `0.00826` nats

Interpretation:

- The beta-law distribution calibration is very tight across two orders of magnitude in dimension.
- Empirical \(C(q)=-\log P(Q\ge q)\) matches the theoretical incomplete-beta cost when the expected tail count is large enough.
- Multilayer capacity accumulation matches the product law until the all-event becomes too rare for the finite sample budget.
- Fixed-threshold high-dimensional tails quickly become too rare for direct Monte Carlo, which is itself consistent with the exponential/geometric volume-collapse interpretation.

## Multilayer Observed Capacity Distribution Run Log

Script:

```bash
python scripts/multilayer_capacity_distribution.py --dimensions 64 256 1024 4096 --layers 1 2 4 8 16 --samples 100000 --batch-size 8192 --seed 456 --plot
```

Theory:

\[
S_l=-\log P(Q_l'\ge Q_l)\sim \mathrm{Exp}(1)
\]

and for independent layers:

\[
S_{1:L}=\sum_{l=1}^L S_l\sim \mathrm{Gamma}(L,1).
\]

Setup:

- dimensions \(D\): `64, 256, 1024, 4096`
- layers \(L\): `1, 2, 4, 8, 16`
- samples per \((D,L)\): `100000`
- total distribution-matching rows: `20`
- sampler: exact random-direction chi-square representation \(Q=X/(X+Y)\)

Summary:

- max KS statistic over all \((D,L)\): `0.0042004`
- mean absolute mean error: `0.0042410`
- mean absolute variance error: `0.0404870`

Selected rows:

| \(D\) | \(L\) | empirical mean | theory mean | empirical var | theory var | KS |
|---:|---:|---:|---:|---:|---:|---:|
| `64` | `1` | `1.00472` | `1` | `1.02000` | `1` | `0.00253` |
| `64` | `16` | `16.00773` | `16` | `15.87860` | `16` | `0.00246` |
| `4096` | `1` | `1.00041` | `1` | `1.00609` | `1` | `0.00194` |
| `4096` | `16` | `16.00106` | `16` | `16.04361` | `16` | `0.00265` |

Interpretation:

- The observed capacity surprisal distribution matches the predicted \(\mathrm{Exp}(1)\) and \(\mathrm{Gamma}(L,1)\) laws tightly.
- After the beta-tail transform, the null surprisal distribution is dimension-free; \(D\) controls the raw \(Q\) scale and fixed-threshold cost, while \(L\) controls the Gamma shape.
- This gives the cleanest “theory predicts a distribution, experiment recovers the same distribution” result for multilayer capacity.

## Minimax Initialization Run Log

Script:

```bash
python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot
```

Result:

- minimax bound \(1/D\): `0.03125`
- isotropic \(\lambda_{\min}\): `0.029084138`
- rademacher \(\lambda_{\min}\): `0.028946927`
- anisotropic \(\lambda_{\min}\): `0.006149976`
- subspace \(\lambda_{\min}\): `0`
- axis \(\lambda_{\min}\): `0`

Random-target means remain close to \(1/D\) for all distributions, but worst-case target coverage differs sharply:

- isotropic and rademacher nearly equalize all target directions;
- anisotropic improves some directions while sacrificing others;
- subspace and axis initializations leave entire orthogonal directions uncovered.

This empirically illustrates the prior-free minimax theorem: without target or weight prior information, anisotropic feedback cannot improve the worst-case angular bound.

## Initialization Coverage Distribution Matching Run Log

Script:

```bash
python scripts/initialization_distribution_matching.py --dimension 128 --target-samples 100000 --feedback-samples 100000 --batch-size 8192 --seed 2026 --subspace-dim 8 --anisotropy 64 --plot
```

Theory:

For a feedback initialization distribution \(\mu\):

\[
M_\mu=\mathbb E[\hat b\hat b^\top].
\]

For a random target direction \(a\):

\[
A(a)=a^\top M_\mu a.
\]

If \(\lambda_i\) are eigenvalues of \(M_\mu\), then:

\[
A(a)
=
\frac{\sum_i \lambda_i G_i}{\sum_i G_i},
\qquad
G_i\sim\chi^2_1.
\]

Setup:

- dimension \(D=128\)
- target samples: `100000`
- feedback samples for empirical \(M_\mu\): `100000`
- schemes: `isotropic`, `rademacher`, `subspace`, `axis`, `geometric_axis`
- subspace dimension: `8`
- geometric axis anisotropy: `64`

Results:

| scheme | population \(\lambda_{\min}\) | empirical \(\lambda_{\min}\) | predicted mean | empirical mean | predicted std | empirical std | KS |
|---|---:|---:|---:|---:|---:|---:|---:|
| `isotropic` | `0.0078125` | `0.0072928` | `0.0078125` | `0.0078124` | `~0` | `0.0000346` | `nan` |
| `rademacher` | `0.0078125` | `0.0072982` | `0.0078125` | `0.0078125` | `~0` | `0.0000342` | `nan` |
| `subspace` | `0` | `0` | `0.0077967` | `0.0078031` | `0.0037405` | `0.0037335` | `0.00299` |
| `axis` | `0` | `0` | `0.0078396` | `0.0078514` | `0.0109834` | `0.0109230` | `0.00351` |
| `geometric_axis` | `0.0005111` | `0.0004100` | `0.0078116` | `0.0078155` | `0.0010441` | `0.0010425` | `0.00470` |

Interpretation:

- All schemes have mean coverage near \(1/D\), as expected from averaging over uniformly random targets.
- Isotropic and rademacher have population point-mass coverage at \(1/D\); empirical spread comes from finite-sample \(M_\mu\) estimation noise, so KS against a point mass is not meaningful.
- Non-isotropic schemes match the predicted target-coverage distribution tightly, with max nondegenerate KS `0.00470`.
- The distribution shape, not just the mean, exposes the tradeoff: subspace and axis schemes keep the same average but create zero-coverage worst-case directions.

## Functional Capacity Overlap Run Log

Script:

```bash
python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot
```

Setup:

- parameter dimension \(P=96\)
- task-sensitive rank \(d=24\)
- redundant dimension \(P-d=72\)

Result:

- \(k=0\): hard loss `0`, theory `0`; soft overlap `0`, theory `0`
- \(k=24\): hard loss `0`, theory `0`; soft overlap `6.0315`, theory `6`
- \(k=48\): hard loss `0`, theory `0`; soft overlap `12.0018`, theory `12`
- \(k=72\): hard loss `0`, theory `0`; soft overlap `18.0018`, theory `18`
- \(k=84\): hard loss `12`, theory `12`; soft overlap `21.0029`, theory `21`
- \(k=96\): hard loss `24`, theory `24`; soft overlap `24`, theory `24`

This validates the redundancy-exhaustion interpretation: hard functional rank remains intact until alignment constraints exceed the redundant dimension \(P-d\), while soft overlap grows linearly as \(kd/P\).

## Synthetic Trajectory Run Log

Script:

```bash
python scripts/trajectory_mlp_fa.py --samples 128 --hidden-widths 24 24 --steps 80 --lr 0.02 --eval-every 10 --feedback-runs 3 --data-seed 3 --init-seed 4 --feedback-seed-start 50 --plot
```

Setup:

- student widths: `[16, 24, 24, 4]`
- synthetic teacher with matching widths
- full-batch MSE
- one BP baseline from the shared initialization
- three FA runs with feedback seeds `50, 51, 52`

Result:

- BP final loss: `0.60596695`
- FA final gap to BP: mean `0.2032423`, min `0.16939124`, max `0.25930484`
- FA final BP/FA gradient cosine: mean `0.31281339`, min `0.29023733`, max `0.35150802`
- FA final hidden-only BP/FA gradient cosine: mean `-0.0070103243`, min `-0.13335294`, max `0.072182807`

Per-seed summary:

- seed `50`: final loss `0.86527179`, final gap `0.25930484`, initial \(Q\) mean `0.00093627`, final \(Q\) mean `0.00281211`
- seed `51`: final loss `0.77535819`, final gap `0.16939124`, initial \(Q\) mean `0.01641749`, final \(Q\) mean `0.04462749`
- seed `52`: final loss `0.78699777`, final gap `0.18103082`, initial \(Q\) mean `0.02377742`, final \(Q\) mean `0.00360508`

This is only a smoke trajectory, not yet an ensemble result. It verifies that the logging pipeline can capture loss gaps, surrogate-gradient alignment, and weight-feedback alignment \(Q_l(t)\) from the same run.

Important metric note: full-model gradient cosine can be inflated by the output layer, whose gradient is identical under BP and FA. Hidden-only gradient cosine is a sharper metric for feedback-induced mismatch.

## Trajectory Ensemble Run Log

Script:

```bash
python scripts/trajectory_ensemble.py --architectures 16,16 24,24 32,32 48,48 24,24,24 --samples 256 --steps 200 --lr 0.02 --eval-every 20 --feedback-runs 40 --data-seed 20 --init-seed 30 --feedback-seed-start 1000 --plot
```

Setup:

- architectures: `16,16`, `24,24`, `32,32`, `48,48`, `24,24,24`
- feedback seeds per architecture: `40`
- total FA trajectories: `200`
- synthetic full-batch regression
- shared BP baseline per architecture

Architecture-level results:

| architecture | BP final loss | FA gap mean | FA gap std | gap min | gap max | final hidden cosine mean | final capacity mean |
|---|---:|---:|---:|---:|---:|---:|---:|
| `h16x16` | `0.549502` | `0.029880` | `0.054179` | `-0.111327` | `0.113590` | `0.305739` | `6.435878` |
| `h24x24` | `0.560063` | `0.187094` | `0.064862` | `0.059168` | `0.316122` | `0.256078` | `7.749068` |
| `h32x32` | `0.427190` | `0.068749` | `0.023786` | `0.029412` | `0.124503` | `0.190019` | `8.956942` |
| `h48x48` | `0.349687` | `0.078056` | `0.018102` | `0.032782` | `0.113509` | `0.196928` | `15.784286` |
| `h24x24x24` | `0.336629` | `0.066436` | `0.020885` | `0.010641` | `0.106398` | `0.008886` | `4.814671` |

Pooled result:

- total FA runs: `200`
- final gap mean: `0.086042897`
- final gap std: `0.067060324`
- final gap min: `-0.11132673`
- final gap max: `0.31612214`
- negative-gap FA runs: `9/200`, all in `h16x16`

Correlation takeaways:

- Pooled correlations are weak because architecture-level differences dominate and confound the relationship between capacity proxies and gap.
- Within architecture, final hidden-gradient alignment is consistently predictive: larger final hidden cosine corresponds to smaller final FA/BP gap.
- Strongest within-architecture Spearman correlations with final gap:
  - `h16x16`: final hidden cosine, \(\rho=-0.4587\), \(p=0.00291\)
  - `h24x24`: final capacity, \(\rho=-0.4432\), \(p=0.00419\)
  - `h32x32`: final hidden cosine, \(\rho=-0.4477\), \(p=0.00377\)
  - `h48x48`: final hidden cosine, \(\rho=-0.5180\), \(p=0.000618\)
  - `h24x24x24`: final hidden cosine, \(\rho=-0.7235\), \(p=1.34e-7\)

Interpretation:

- Static \(Q\) and capacity proxies alone do not explain pooled loss gaps across architectures.
- Same-architecture seed variation is much better explained by trajectory-level hidden-gradient alignment.
- This supports the current paper framing: static distributional capacity bounds quantify the initial burden, while trajectory bridge metrics are needed to explain realized FA/BP loss gaps.

## Trajectory Gap Distribution Bridge Run Log

Script for `h16x16`:

```bash
python scripts/trajectory_gap_distribution.py --hidden-widths 16 16 --samples 128 --steps 60 --lr 0.02 --feedback-runs 1000 --data-seed 7 --init-seed 8 --feedback-seed-start 10000 --outdir outputs/trajectory_gap_distribution/h16x16_1000 --plot
```

Script for `h24x24`:

```bash
python scripts/trajectory_gap_distribution.py --hidden-widths 24 24 --samples 128 --steps 60 --lr 0.02 --feedback-runs 500 --data-seed 7 --init-seed 8 --feedback-seed-start 20000 --outdir outputs/trajectory_gap_distribution/h24x24_500 --plot
```

Bridge predictor:

\[
\widehat{\delta\theta}_T(B)
=
-\eta
\sum_{t<T}
\left[
g_{\mathrm{FA}}(\theta_t^{\mathrm{BP}};B)
-
g_{\mathrm{BP}}(\theta_t^{\mathrm{BP}})
\right],
\]

\[
\widehat{\Delta L}_T(B)
=
L(\theta_T^{\mathrm{BP}}+\widehat{\delta\theta}_T(B))
-
L(\theta_T^{\mathrm{BP}}).
\]

Results:

| architecture | runs | BP final loss | empirical gap mean | bridge gap mean | empirical std | bridge std | raw KS | standardized KS | standardized p | moment-matched W1 | Spearman |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| `h16x16` | `1000` | `0.604310` | `0.120109` | `0.318266` | `0.041817` | `0.081883` | `0.917` | `0.056` | `0.0869` | `0.00362` | `0.49288` |
| `h24x24` | `500` | `0.572197` | `0.145215` | `0.432834` | `0.055733` | `0.108192` | `0.944` | `0.056` | `0.4135` | `0.00356` | `0.39455` |

Interpretation:

- The raw bridge predictor overestimates absolute gap scale by roughly \(2\)-\(3\times\), so the current identity-propagation bridge is not yet a final calibrated theory for \(\Delta L_T\).
- The standardized distributions match surprisingly well: both architectures have standardized KS `0.056`, with non-rejected two-sample tests at these sample sizes.
- The paired seed-level rank signal is nontrivial: Spearman is `0.49288` for `h16x16` and `0.39455` for `h24x24`.
- Current conclusion: the BP-path mismatch bridge captures much of the random-feedback distributional shape but lacks the curvature/contraction operator needed to predict absolute scale.