path: root/notes/02_experiment_notes.md

# Experiment Notes

## Experiment Layer 1: Static Distribution Validation

Purpose: verify the beta law before training dynamics enter.

Procedure:

1. Choose matrix shape \(n_l\times n_{l+1}\), so \(D_l=n_l n_{l+1}\).
2. Sample \(A_l\) and \(B_l\) independently from isotropic distributions.
3. Compute:

\[
Q_l=
\frac{
\langle A_l,B_l\rangle_F^2
}{
\|A_l\|_F^2\|B_l\|_F^2
}.
\]

4. Compare empirical distribution to:

\[
\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]

Metrics:

- Histogram overlay.
- QQ plot.
- KS statistic.
- Tail calibration:

\[
\Pr(Q_l\ge q).
\]

Initialization variants:

- Gaussian.
- Rademacher.
- Uniform sphere.
- Orthogonal or semi-orthogonal.
- Sparse.
- Low-rank.
- Block-diagonal.

Expected:

- Dense isotropic variants match beta law.
- Structured variants deviate in predictable ways.

## Experiment Layer 2: Scaling Validation

Purpose: verify capacity scaling with depth and width.

Compute:

\[
C_l(q)=
-\log
\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]

Total:

\[
C_{\mathrm{all}}=\sum_l C_l(q_l).
\]

Sweeps:

- Width \(n\).
- Depth \(L\).
- Threshold \(q\).
- Threshold regime \(q=c/D_l\).
- Feedback rank.
- Feedback sparsity.

Predictions:

Fixed \(q\):

\[
C_{\mathrm{all}}=\Theta(Ln^2)
\]

for equal-width MLPs.

Chance-level \(q=c/D_l\):

\[
C_{\mathrm{all}}=\Theta(L).
\]

## Experiment Layer 3: Local Gradient Alignment

Purpose: connect static matrix alignment to update direction mismatch.

During training, record:

\[
\Gamma_t=
\frac{
\langle g_t^{\mathrm{BP}},g_t^{\mathrm{FA}}\rangle
}{
\|g_t^{\mathrm{BP}}\|\|g_t^{\mathrm{FA}}\|
}.
\]

Layerwise:

\[
\Gamma_{l,t}=
\frac{
\langle g_{l,t}^{\mathrm{BP}},g_{l,t}^{\mathrm{FA}}\rangle
}{
\|g_{l,t}^{\mathrm{BP}}\|\|g_{l,t}^{\mathrm{FA}}\|
}.
\]

Also record:

\[
Q_l(t)=\cos^2(W_{l+1}(t)^\top,B_l).
\]

Questions:

- Does \(Q_l(0)\) match the beta baseline?
- Does \(Q_l(t)\) shift right during the alignment phase?
- Does gradient alignment improve before memorization or loss reduction?

## Experiment Layer 4: Trajectory Ensemble

Purpose: validate whether capacity proxies explain FA/BP training gaps.

For each architecture and dataset:

1. Fix data seed and model architecture.
2. Train BP baseline.
3. Train many FA runs over feedback seeds \(B\).
4. Record:

\[
\Delta L_T(B)=L_T^{\mathrm{FA}}(B)-L_T^{\mathrm{BP}},
\]

\[
\Delta A_T(B)=A_T^{\mathrm{BP}}-A_T^{\mathrm{FA}},
\]

\[
C_{\mathrm{all}}(B,t),
\quad
\Gamma_t(B),
\quad
Q_l(t).
\]

Datasets:

- Synthetic Gaussian regression.
- MNIST MLP.
- Fashion-MNIST MLP.
- CIFAR-10 flattened MLP, optional later.

Architectures:

- Equal-width MLPs.
- Width sweep.
- Depth sweep.
- Narrow bottleneck sweep.

Expected:

- Overparameterized regimes: large parameter-volume cost can coexist with small functional gap.
- Near redundancy exhaustion: FA/BP gap should increase sharply.
- Poor feedback conditioning can worsen trajectory gap even when angular minimax bound is unchanged.

## Plots

Static:

- Histogram and beta density.
- QQ plot.
- Tail probability calibration.

Scaling:

- \(C_{\mathrm{all}}\) vs \(Ln^2\).
- \(-\log p_{\mathrm{all}}\) vs depth.
- Scaling collapse for \(D_lQ_l\Rightarrow \chi_1^2\).

Trajectory:

- \(Q_l(t)\) over training.
- \(\Gamma_t\) over training.
- \(\Delta L_T\) vs capacity proxy.
- \(\Delta L_T\) vs conditioning proxy.
- Phase transition plot against \(k-(P-d)\).

## Implementation Notes

Start with NumPy or PyTorch scripts that do not require full training.

First script target:

- Sample \(A,B\).
- Compute \(Q\).
- Save empirical moments and KS statistic.
- Produce beta overlay plots.

Only after this is clean, add FA/BP training loops.

## Baseline Run Log

Script:

```bash
python scripts/static_alignment_beta.py --rows 16 --cols 16 --samples 20000 --seed 7 --plot
```

Result:

- \(D=256\)
- empirical mean: `0.0039265884`
- theoretical mean: `0.00390625`
- empirical variance: `3.0311252e-05`
- theoretical variance: `3.0162723e-05`
- KS statistic: `0.00526931`
- KS p-value: `0.633285`

This is a clean first-pass validation for the isotropic Gaussian case.

## Scaling Run Log

Script:

```bash
python scripts/capacity_scaling.py --plot
```

Default sweep:

- widths: `16, 32, 64, 128`
- feedback-aligned layer counts: `1, 2, 4, 8, 16`
- fixed threshold: \(q=0.01\)
- chance-level threshold: \(q=1/D\)
- log unit: nats

Result:

- rows written: `40`
- fixed-threshold max total cost: `1361.74` nats at width `128`, layers `16`
- chance-level max total cost: `18.3652` nats at width `128`, layers `16`

This cleanly separates the fixed-threshold regime, where total cost scales like \(Ln^2\), from the chance-level regime, where per-layer cost is nearly width-independent.

## Minimax Initialization Run Log

Script:

```bash
python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot
```

Result:

- minimax bound \(1/D\): `0.03125`
- isotropic \(\lambda_{\min}\): `0.029084138`
- rademacher \(\lambda_{\min}\): `0.028946927`
- anisotropic \(\lambda_{\min}\): `0.006149976`
- subspace \(\lambda_{\min}\): `0`
- axis \(\lambda_{\min}\): `0`

Random-target means remain close to \(1/D\) for all distributions, but worst-case target coverage differs sharply:

- isotropic and rademacher nearly equalize all target directions;
- anisotropic improves some directions while sacrificing others;
- subspace and axis initializations leave entire orthogonal directions uncovered.

This empirically illustrates the prior-free minimax theorem: without target or weight prior information, anisotropic feedback cannot improve the worst-case angular bound.

## Functional Capacity Overlap Run Log

Script:

```bash
python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot
```

Setup:

- parameter dimension \(P=96\)
- task-sensitive rank \(d=24\)
- redundant dimension \(P-d=72\)

Result:

- \(k=0\): hard loss `0`, theory `0`; soft overlap `0`, theory `0`
- \(k=24\): hard loss `0`, theory `0`; soft overlap `6.0315`, theory `6`
- \(k=48\): hard loss `0`, theory `0`; soft overlap `12.0018`, theory `12`
- \(k=72\): hard loss `0`, theory `0`; soft overlap `18.0018`, theory `18`
- \(k=84\): hard loss `12`, theory `12`; soft overlap `21.0029`, theory `21`
- \(k=96\): hard loss `24`, theory `24`; soft overlap `24`, theory `24`

This validates the redundancy-exhaustion interpretation: hard functional rank remains intact until alignment constraints exceed the redundant dimension \(P-d\), while soft overlap grows linearly as \(kd/P\).

## Synthetic Trajectory Run Log

Script:

```bash
python scripts/trajectory_mlp_fa.py --samples 128 --hidden-widths 24 24 --steps 80 --lr 0.02 --eval-every 10 --feedback-runs 3 --data-seed 3 --init-seed 4 --feedback-seed-start 50 --plot
```

Setup:

- student widths: `[16, 24, 24, 4]`
- synthetic teacher with matching widths
- full-batch MSE
- one BP baseline from the shared initialization
- three FA runs with feedback seeds `50, 51, 52`

Result:

- BP final loss: `0.60596695`
- FA final gap to BP: mean `0.2032423`, min `0.16939124`, max `0.25930484`
- FA final BP/FA gradient cosine: mean `0.31281339`, min `0.29023733`, max `0.35150802`
- FA final hidden-only BP/FA gradient cosine: mean `-0.0070103243`, min `-0.13335294`, max `0.072182807`

Per-seed summary:

- seed `50`: final loss `0.86527179`, final gap `0.25930484`, initial \(Q\) mean `0.00093627`, final \(Q\) mean `0.00281211`
- seed `51`: final loss `0.77535819`, final gap `0.16939124`, initial \(Q\) mean `0.01641749`, final \(Q\) mean `0.04462749`
- seed `52`: final loss `0.78699777`, final gap `0.18103082`, initial \(Q\) mean `0.02377742`, final \(Q\) mean `0.00360508`

This is only a smoke trajectory, not yet an ensemble result. It verifies that the logging pipeline can capture loss gaps, surrogate-gradient alignment, and weight-feedback alignment \(Q_l(t)\) from the same run.

Important metric note: full-model gradient cosine can be inflated by the output layer, whose gradient is identical under BP and FA. Hidden-only gradient cosine is a sharper metric for feedback-induced mismatch.