# Experiment Notes ## Experiment Layer 1: Static Distribution Validation Purpose: verify the beta law before training dynamics enter. Procedure: 1. Choose matrix shape \(n_l\times n_{l+1}\), so \(D_l=n_l n_{l+1}\). 2. Sample \(A_l\) and \(B_l\) independently from isotropic distributions. 3. Compute: \[ Q_l= \frac{ \langle A_l,B_l\rangle_F^2 }{ \|A_l\|_F^2\|B_l\|_F^2 }. \] 4. Compare empirical distribution to: \[ \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right). \] Metrics: - Histogram overlay. - QQ plot. - KS statistic. - Tail calibration: \[ \Pr(Q_l\ge q). \] Initialization variants: - Gaussian. - Rademacher. - Uniform sphere. - Orthogonal or semi-orthogonal. - Sparse. - Low-rank. - Block-diagonal. Expected: - Dense isotropic variants match beta law. - Structured variants deviate in predictable ways. ## Experiment Layer 2: Scaling Validation Purpose: verify capacity scaling with depth and width. Compute: \[ C_l(q)= -\log \left[ 1-I_q\left(\frac12,\frac{D_l-1}{2}\right) \right]. \] Total: \[ C_{\mathrm{all}}=\sum_l C_l(q_l). \] Sweeps: - Width \(n\). - Depth \(L\). - Threshold \(q\). - Threshold regime \(q=c/D_l\). - Feedback rank. - Feedback sparsity. Predictions: Fixed \(q\): \[ C_{\mathrm{all}}=\Theta(Ln^2) \] for equal-width MLPs. Chance-level \(q=c/D_l\): \[ C_{\mathrm{all}}=\Theta(L). \] ## Experiment Layer 3: Local Gradient Alignment Purpose: connect static matrix alignment to update direction mismatch. During training, record: \[ \Gamma_t= \frac{ \langle g_t^{\mathrm{BP}},g_t^{\mathrm{FA}}\rangle }{ \|g_t^{\mathrm{BP}}\|\|g_t^{\mathrm{FA}}\| }. \] Layerwise: \[ \Gamma_{l,t}= \frac{ \langle g_{l,t}^{\mathrm{BP}},g_{l,t}^{\mathrm{FA}}\rangle }{ \|g_{l,t}^{\mathrm{BP}}\|\|g_{l,t}^{\mathrm{FA}}\| }. \] Also record: \[ Q_l(t)=\cos^2(W_{l+1}(t)^\top,B_l). \] Questions: - Does \(Q_l(0)\) match the beta baseline? - Does \(Q_l(t)\) shift right during the alignment phase? - Does gradient alignment improve before memorization or loss reduction? ## Experiment Layer 4: Trajectory Ensemble Purpose: validate whether capacity proxies explain FA/BP training gaps. For each architecture and dataset: 1. Fix data seed and model architecture. 2. Train BP baseline. 3. Train many FA runs over feedback seeds \(B\). 4. Record: \[ \Delta L_T(B)=L_T^{\mathrm{FA}}(B)-L_T^{\mathrm{BP}}, \] \[ \Delta A_T(B)=A_T^{\mathrm{BP}}-A_T^{\mathrm{FA}}, \] \[ C_{\mathrm{all}}(B,t), \quad \Gamma_t(B), \quad Q_l(t). \] Datasets: - Synthetic Gaussian regression. - MNIST MLP. - Fashion-MNIST MLP. - CIFAR-10 flattened MLP, optional later. Architectures: - Equal-width MLPs. - Width sweep. - Depth sweep. - Narrow bottleneck sweep. Expected: - Overparameterized regimes: large parameter-volume cost can coexist with small functional gap. - Near redundancy exhaustion: FA/BP gap should increase sharply. - Poor feedback conditioning can worsen trajectory gap even when angular minimax bound is unchanged. ## Plots Static: - Histogram and beta density. - QQ plot. - Tail probability calibration. Scaling: - \(C_{\mathrm{all}}\) vs \(Ln^2\). - \(-\log p_{\mathrm{all}}\) vs depth. - Scaling collapse for \(D_lQ_l\Rightarrow \chi_1^2\). Trajectory: - \(Q_l(t)\) over training. - \(\Gamma_t\) over training. - \(\Delta L_T\) vs capacity proxy. - \(\Delta L_T\) vs conditioning proxy. - Phase transition plot against \(k-(P-d)\). ## Implementation Notes Start with NumPy or PyTorch scripts that do not require full training. First script target: - Sample \(A,B\). - Compute \(Q\). - Save empirical moments and KS statistic. - Produce beta overlay plots. Only after this is clean, add FA/BP training loops. ## Baseline Run Log Script: ```bash python scripts/static_alignment_beta.py --rows 16 --cols 16 --samples 20000 --seed 7 --plot ``` Result: - \(D=256\) - empirical mean: `0.0039265884` - theoretical mean: `0.00390625` - empirical variance: `3.0311252e-05` - theoretical variance: `3.0162723e-05` - KS statistic: `0.00526931` - KS p-value: `0.633285` This is a clean first-pass validation for the isotropic Gaussian case. ## Scaling Run Log Script: ```bash python scripts/capacity_scaling.py --plot ``` Default sweep: - widths: `16, 32, 64, 128` - feedback-aligned layer counts: `1, 2, 4, 8, 16` - fixed threshold: \(q=0.01\) - chance-level threshold: \(q=1/D\) - log unit: nats Result: - rows written: `40` - fixed-threshold max total cost: `1361.74` nats at width `128`, layers `16` - chance-level max total cost: `18.3652` nats at width `128`, layers `16` This cleanly separates the fixed-threshold regime, where total cost scales like \(Ln^2\), from the chance-level regime, where per-layer cost is nearly width-independent. ## Large Empirical Capacity Validation Run Log Script: ```bash python scripts/capacity_empirical_validation.py --dimensions 64 128 256 512 1024 2048 4096 --samples 100000 --batch-size 2048 --seed 123 --plot ``` Setup: - dimensions \(D\): `64, 128, 256, 512, 1024, 2048, 4096` - samples per dimension: `100000` - total \(Q\) samples: `700000` - sampler: fixed-target Gaussian direction, using rotational invariance - chance thresholds: \(q=c/D\), \(c\in\{0.5,1,2,5,10\}\) - fixed thresholds: \(q\in\{0.001,0.002,0.005,0.01\}\) - multilayer all-event validation: \(c\in\{0.5,1\}\), layers \(L\in\{1,2,4,8\}\) Distribution calibration: | \(D\) | empirical mean | theory mean | KS statistic | empirical q99 | theory q99 | |---:|---:|---:|---:|---:|---:| | `64` | `0.0156793` | `0.0156250` | `0.0034949` | `0.1007135` | `0.1007080` | | `128` | `0.0078528` | `0.0078125` | `0.0019669` | `0.0511649` | `0.0510968` | | `256` | `0.0039289` | `0.0039062` | `0.0029576` | `0.0259898` | `0.0257333` | | `512` | `0.0019553` | `0.0019531` | `0.0016418` | `0.0129654` | `0.0129127` | | `1024` | `0.0009706` | `0.0009766` | `0.0031219` | `0.0063694` | `0.0064679` | | `2048` | `0.0004882` | `0.0004883` | `0.0031095` | `0.0032339` | `0.0032368` | | `4096` | `0.0002444` | `0.0002441` | `0.0024262` | `0.0016322` | `0.0016191` | Capacity-tail calibration: - chance-threshold rows: `35/35` had expected hits \(\ge 20\) - max absolute smoothed cost error over all chance rows: `0.2023` nats - mean absolute smoothed cost error over all chance rows: `0.01915` nats - for chance rows with expected hits \(\ge 500\): max error `0.02771` nats, mean error `0.00607` nats Multilayer product-capacity calibration: - rows with expected hits \(\ge 20\): `49` - max absolute smoothed total-cost error: `0.45788` nats - mean absolute smoothed total-cost error: `0.03880` nats - for rows with expected hits \(\ge 500\): max error `0.04319` nats, mean error `0.00826` nats Interpretation: - The beta-law distribution calibration is very tight across two orders of magnitude in dimension. - Empirical \(C(q)=-\log P(Q\ge q)\) matches the theoretical incomplete-beta cost when the expected tail count is large enough. - Multilayer capacity accumulation matches the product law until the all-event becomes too rare for the finite sample budget. - Fixed-threshold high-dimensional tails quickly become too rare for direct Monte Carlo, which is itself consistent with the exponential/geometric volume-collapse interpretation. ## Multilayer Observed Capacity Distribution Run Log Script: ```bash python scripts/multilayer_capacity_distribution.py --dimensions 64 256 1024 4096 --layers 1 2 4 8 16 --samples 100000 --batch-size 8192 --seed 456 --plot ``` Theory: \[ S_l=-\log P(Q_l'\ge Q_l)\sim \mathrm{Exp}(1) \] and for independent layers: \[ S_{1:L}=\sum_{l=1}^L S_l\sim \mathrm{Gamma}(L,1). \] Setup: - dimensions \(D\): `64, 256, 1024, 4096` - layers \(L\): `1, 2, 4, 8, 16` - samples per \((D,L)\): `100000` - total distribution-matching rows: `20` - sampler: exact random-direction chi-square representation \(Q=X/(X+Y)\) Summary: - max KS statistic over all \((D,L)\): `0.0042004` - mean absolute mean error: `0.0042410` - mean absolute variance error: `0.0404870` Selected rows: | \(D\) | \(L\) | empirical mean | theory mean | empirical var | theory var | KS | |---:|---:|---:|---:|---:|---:|---:| | `64` | `1` | `1.00472` | `1` | `1.02000` | `1` | `0.00253` | | `64` | `16` | `16.00773` | `16` | `15.87860` | `16` | `0.00246` | | `4096` | `1` | `1.00041` | `1` | `1.00609` | `1` | `0.00194` | | `4096` | `16` | `16.00106` | `16` | `16.04361` | `16` | `0.00265` | Interpretation: - The observed capacity surprisal distribution matches the predicted \(\mathrm{Exp}(1)\) and \(\mathrm{Gamma}(L,1)\) laws tightly. - After the beta-tail transform, the null surprisal distribution is dimension-free; \(D\) controls the raw \(Q\) scale and fixed-threshold cost, while \(L\) controls the Gamma shape. - This gives the cleanest “theory predicts a distribution, experiment recovers the same distribution” result for multilayer capacity. ## Minimax Initialization Run Log Script: ```bash python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot ``` Result: - minimax bound \(1/D\): `0.03125` - isotropic \(\lambda_{\min}\): `0.029084138` - rademacher \(\lambda_{\min}\): `0.028946927` - anisotropic \(\lambda_{\min}\): `0.006149976` - subspace \(\lambda_{\min}\): `0` - axis \(\lambda_{\min}\): `0` Random-target means remain close to \(1/D\) for all distributions, but worst-case target coverage differs sharply: - isotropic and rademacher nearly equalize all target directions; - anisotropic improves some directions while sacrificing others; - subspace and axis initializations leave entire orthogonal directions uncovered. This empirically illustrates the prior-free minimax theorem: without target or weight prior information, anisotropic feedback cannot improve the worst-case angular bound. ## Initialization Coverage Distribution Matching Run Log Script: ```bash python scripts/initialization_distribution_matching.py --dimension 128 --target-samples 100000 --feedback-samples 100000 --batch-size 8192 --seed 2026 --subspace-dim 8 --anisotropy 64 --plot ``` Theory: For a feedback initialization distribution \(\mu\): \[ M_\mu=\mathbb E[\hat b\hat b^\top]. \] For a random target direction \(a\): \[ A(a)=a^\top M_\mu a. \] If \(\lambda_i\) are eigenvalues of \(M_\mu\), then: \[ A(a) = \frac{\sum_i \lambda_i G_i}{\sum_i G_i}, \qquad G_i\sim\chi^2_1. \] Setup: - dimension \(D=128\) - target samples: `100000` - feedback samples for empirical \(M_\mu\): `100000` - schemes: `isotropic`, `rademacher`, `subspace`, `axis`, `geometric_axis` - subspace dimension: `8` - geometric axis anisotropy: `64` Results: | scheme | population \(\lambda_{\min}\) | empirical \(\lambda_{\min}\) | predicted mean | empirical mean | predicted std | empirical std | KS | |---|---:|---:|---:|---:|---:|---:|---:| | `isotropic` | `0.0078125` | `0.0072928` | `0.0078125` | `0.0078124` | `~0` | `0.0000346` | `nan` | | `rademacher` | `0.0078125` | `0.0072982` | `0.0078125` | `0.0078125` | `~0` | `0.0000342` | `nan` | | `subspace` | `0` | `0` | `0.0077967` | `0.0078031` | `0.0037405` | `0.0037335` | `0.00299` | | `axis` | `0` | `0` | `0.0078396` | `0.0078514` | `0.0109834` | `0.0109230` | `0.00351` | | `geometric_axis` | `0.0005111` | `0.0004100` | `0.0078116` | `0.0078155` | `0.0010441` | `0.0010425` | `0.00470` | Interpretation: - All schemes have mean coverage near \(1/D\), as expected from averaging over uniformly random targets. - Isotropic and rademacher have population point-mass coverage at \(1/D\); empirical spread comes from finite-sample \(M_\mu\) estimation noise, so KS against a point mass is not meaningful. - Non-isotropic schemes match the predicted target-coverage distribution tightly, with max nondegenerate KS `0.00470`. - The distribution shape, not just the mean, exposes the tradeoff: subspace and axis schemes keep the same average but create zero-coverage worst-case directions. ## Functional Capacity Overlap Run Log Script: ```bash python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot ``` Setup: - parameter dimension \(P=96\) - task-sensitive rank \(d=24\) - redundant dimension \(P-d=72\) Result: - \(k=0\): hard loss `0`, theory `0`; soft overlap `0`, theory `0` - \(k=24\): hard loss `0`, theory `0`; soft overlap `6.0315`, theory `6` - \(k=48\): hard loss `0`, theory `0`; soft overlap `12.0018`, theory `12` - \(k=72\): hard loss `0`, theory `0`; soft overlap `18.0018`, theory `18` - \(k=84\): hard loss `12`, theory `12`; soft overlap `21.0029`, theory `21` - \(k=96\): hard loss `24`, theory `24`; soft overlap `24`, theory `24` This validates the redundancy-exhaustion interpretation: hard functional rank remains intact until alignment constraints exceed the redundant dimension \(P-d\), while soft overlap grows linearly as \(kd/P\). ## Synthetic Trajectory Run Log Script: ```bash python scripts/trajectory_mlp_fa.py --samples 128 --hidden-widths 24 24 --steps 80 --lr 0.02 --eval-every 10 --feedback-runs 3 --data-seed 3 --init-seed 4 --feedback-seed-start 50 --plot ``` Setup: - student widths: `[16, 24, 24, 4]` - synthetic teacher with matching widths - full-batch MSE - one BP baseline from the shared initialization - three FA runs with feedback seeds `50, 51, 52` Result: - BP final loss: `0.60596695` - FA final gap to BP: mean `0.2032423`, min `0.16939124`, max `0.25930484` - FA final BP/FA gradient cosine: mean `0.31281339`, min `0.29023733`, max `0.35150802` - FA final hidden-only BP/FA gradient cosine: mean `-0.0070103243`, min `-0.13335294`, max `0.072182807` Per-seed summary: - seed `50`: final loss `0.86527179`, final gap `0.25930484`, initial \(Q\) mean `0.00093627`, final \(Q\) mean `0.00281211` - seed `51`: final loss `0.77535819`, final gap `0.16939124`, initial \(Q\) mean `0.01641749`, final \(Q\) mean `0.04462749` - seed `52`: final loss `0.78699777`, final gap `0.18103082`, initial \(Q\) mean `0.02377742`, final \(Q\) mean `0.00360508` This is only a smoke trajectory, not yet an ensemble result. It verifies that the logging pipeline can capture loss gaps, surrogate-gradient alignment, and weight-feedback alignment \(Q_l(t)\) from the same run. Important metric note: full-model gradient cosine can be inflated by the output layer, whose gradient is identical under BP and FA. Hidden-only gradient cosine is a sharper metric for feedback-induced mismatch. ## Trajectory Ensemble Run Log Script: ```bash python scripts/trajectory_ensemble.py --architectures 16,16 24,24 32,32 48,48 24,24,24 --samples 256 --steps 200 --lr 0.02 --eval-every 20 --feedback-runs 40 --data-seed 20 --init-seed 30 --feedback-seed-start 1000 --plot ``` Setup: - architectures: `16,16`, `24,24`, `32,32`, `48,48`, `24,24,24` - feedback seeds per architecture: `40` - total FA trajectories: `200` - synthetic full-batch regression - shared BP baseline per architecture Architecture-level results: | architecture | BP final loss | FA gap mean | FA gap std | gap min | gap max | final hidden cosine mean | final capacity mean | |---|---:|---:|---:|---:|---:|---:|---:| | `h16x16` | `0.549502` | `0.029880` | `0.054179` | `-0.111327` | `0.113590` | `0.305739` | `6.435878` | | `h24x24` | `0.560063` | `0.187094` | `0.064862` | `0.059168` | `0.316122` | `0.256078` | `7.749068` | | `h32x32` | `0.427190` | `0.068749` | `0.023786` | `0.029412` | `0.124503` | `0.190019` | `8.956942` | | `h48x48` | `0.349687` | `0.078056` | `0.018102` | `0.032782` | `0.113509` | `0.196928` | `15.784286` | | `h24x24x24` | `0.336629` | `0.066436` | `0.020885` | `0.010641` | `0.106398` | `0.008886` | `4.814671` | Pooled result: - total FA runs: `200` - final gap mean: `0.086042897` - final gap std: `0.067060324` - final gap min: `-0.11132673` - final gap max: `0.31612214` - negative-gap FA runs: `9/200`, all in `h16x16` Correlation takeaways: - Pooled correlations are weak because architecture-level differences dominate and confound the relationship between capacity proxies and gap. - Within architecture, final hidden-gradient alignment is consistently predictive: larger final hidden cosine corresponds to smaller final FA/BP gap. - Strongest within-architecture Spearman correlations with final gap: - `h16x16`: final hidden cosine, \(\rho=-0.4587\), \(p=0.00291\) - `h24x24`: final capacity, \(\rho=-0.4432\), \(p=0.00419\) - `h32x32`: final hidden cosine, \(\rho=-0.4477\), \(p=0.00377\) - `h48x48`: final hidden cosine, \(\rho=-0.5180\), \(p=0.000618\) - `h24x24x24`: final hidden cosine, \(\rho=-0.7235\), \(p=1.34e-7\) Interpretation: - Static \(Q\) and capacity proxies alone do not explain pooled loss gaps across architectures. - Same-architecture seed variation is much better explained by trajectory-level hidden-gradient alignment. - This supports the current paper framing: static distributional capacity bounds quantify the initial burden, while trajectory bridge metrics are needed to explain realized FA/BP loss gaps. ## Trajectory Gap Distribution Bridge Run Log Script for `h16x16`: ```bash python scripts/trajectory_gap_distribution.py --hidden-widths 16 16 --samples 128 --steps 60 --lr 0.02 --feedback-runs 1000 --data-seed 7 --init-seed 8 --feedback-seed-start 10000 --outdir outputs/trajectory_gap_distribution/h16x16_1000 --plot ``` Script for `h24x24`: ```bash python scripts/trajectory_gap_distribution.py --hidden-widths 24 24 --samples 128 --steps 60 --lr 0.02 --feedback-runs 500 --data-seed 7 --init-seed 8 --feedback-seed-start 20000 --outdir outputs/trajectory_gap_distribution/h24x24_500 --plot ``` Bridge predictor: \[ \widehat{\delta\theta}_T(B) = -\eta \sum_{t