2 files changed, 328 insertions, 0 deletions
diff --git a/notes/04_downstream_capacity_design.md b/notes/04_downstream_capacity_design.md
new file mode 100644
index 0000000..c2b9704
--- /dev/null
+++ b/notes/04_downstream_capacity_design.md
@@ -0,0 +1,250 @@
+# Downstream Capacity Experiment Design
+
+## Goal
+
+Validate the core downstream claim:
+
+> FA and BP should behave similarly while redundant parameter capacity can absorb the FA alignment burden, and FA should degrade faster once redundancy is exhausted.
+
+For random-label memorization, the primary performance metric is train MSE. The
+experiment should produce two performance curves:
+
+\[
+L_{\mathrm{BP}}(P)
+\quad \text{and} \quad
+L_{\mathrm{FA}}(P),
+\]
+
+where \(P\) is model parameter count. The target signature is a widening FA/BP
+train gap as \(P\) is reduced, with the widening centered near the predicted FA
+capacity margin.
+
+## Main Task Choice: Random-Label Memorization
+
+Use:
+
+\[
+x_i\sim \mathcal N(0,I_{n_0}),
+\qquad
+y_i\sim \mathcal N(0,I_{n_L}).
+\]
+
+The target is train-set memorization, not test generalization. This makes the
+task dimension directly measurable:
+
+\[
+d_{\mathrm{task}}\approx N n_L.
+\]
+
+For a two-hidden-layer MLP with no bias,
+
+\[
+P(n)=n_0 n+n^2+n n_L.
+\]
+
+For ordinary FA, the hard alignment-burden proxy is:
+
+\[
+K_{\mathrm{FA}}(n)
+=
+(n^2-1)+(n n_L-1).
+\]
+
+The predicted thresholds are:
+
+\[
+P(n)\gtrsim Nn_L
+\qquad \text{for BP},
+\]
+
+and
+
+\[
+P(n)-K_{\mathrm{FA}}(n)\gtrsim Nn_L
+\qquad \text{for FA}.
+\]
+
+This is the cleanest task for testing the redundancy-exhaustion story because
+the breakpoint is predicted before training.
+
+## Auxiliary Task Choice: Low-Rank Linear Teacher
+
+Use:
+
+\[
+x\sim \mathcal N(0,I_{n_0}),
+\]
+
+\[
+y=A x + \epsilon,
+\]
+
+where:
+
+\[
+A=U\Sigma V^\top,\qquad \operatorname{rank}(A)=r.
+\]
+
+Reasons:
+
+- Input/output dimensions are explicit.
+- Task rank is controlled.
+- Test loss is easy to measure.
+- It is simple enough that BP should establish a clear capacity baseline.
+
+Open risk:
+
+- The task may be too easy for ReLU MLPs, and FA may not show a clean transition unless width is very small.
+
+## Architecture
+
+Use a two-hidden-layer ReLU MLP:
+
+\[
+n_0 \to n \to n \to n_L.
+\]
+
+Sweep hidden width:
+
+\[
+n\in\{2,3,4,6,8,12,16,24,32,48,64\}.
+\]
+
+Parameter count without bias:
+
+\[
+P(n)=n_0 n+n^2+n n_L.
+\]
+
+Depth is fixed so that the main knob is capacity, not feedback path length.
+
+## Training
+
+For each width:
+
+- Train BP baseline.
+- Train FA with fixed random feedback matrices.
+- Use same initial forward weights for matched BP/FA comparisons.
+- Use multiple data/init seeds and multiple feedback seeds.
+
+Initial smoke sweep:
+
+- widths: \(\{2,4,8,16,32\}\)
+- init seeds: `3`
+- feedback seeds per init: `5`
+
+Scale-up sweep:
+
+- widths: `2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64`
+- init seeds: `10`
+- feedback seeds per init: `20`
+
+## Theoretical Quantities to Estimate
+
+### Parameter Count
+
+\[
+P=\sum_l n_l n_{l-1}.
+\]
+
+### Task Effective Dimension
+
+At the BP-trained endpoint, estimate:
+
+\[
+J=\frac{\partial f_\theta(X)}{\partial \theta}
+\]
+
+on a probe batch. Let:
+
+\[
+K=JJ^\top.
+\]
+
+Effective dimension:
+
+\[
+d_{\mathrm{eff}}(\lambda)
+=
+\operatorname{tr}\left[K(K+\lambda I)^{-1}\right].
+\]
+
+This estimates how many functional directions the task uses locally.
+
+### Hard FA Burden
+
+For the random-label memorization task, use the hard alignment-burden proxy:
+
+\[
+K_{\mathrm{FA}}=\sum_l(D_l-1).
+\]
+
+For ordinary FA in a two-hidden-layer MLP:
+
+\[
+D_1=n^2,\qquad D_2=n n_L,
+\]
+
+so:
+
+\[
+K_{\mathrm{FA}}=(n^2-1)+(n n_L-1).
+\]
+
+### Log-Volume FA Burden
+
+For matrix-level capacity validation, keep the log-volume quantity:
+
+\[
+k_{\mathrm{FA}}^{\mathrm{proxy}}
+=
+\sum_l C_l(q),
+\]
+
+where:
+
+\[
+C_l(q)=
+-\log P(Q_l\ge q),
+\qquad
+Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
+\]
+
+Use a fixed threshold \(q\) for the first sweep. This is not a hard constraint
+count; it is the matrix-level log-volume cost used in the direct capacity
+distribution experiments.
+
+### Redundancy Score
+
+Define:
+
+\[
+R_{\mathrm{eff}}
+=
+P-d_{\mathrm{eff}}-K_{\mathrm{FA}}.
+\]
+
+For random-label memorization, also use the direct theoretical margin:
+
+\[
+M_{\mathrm{FA}}=P-K_{\mathrm{FA}}-Nn_L.
+\]
+
+## Main Plots
+
+1. BP and FA train MSE vs hidden width or parameter count.
+2. FA/BP train MSE gap vs hidden width or parameter count.
+3. FA/BP train MSE gap vs predicted FA margin \(P-K_{\mathrm{FA}}-Nn_L\).
+4. Estimated \(d_{\mathrm{eff}}\), \(k_{\mathrm{FA}}^{\mathrm{proxy}}\), and \(P\) vs width.
+
+## Success Criterion
+
+The experiment supports the contribution if:
+
+- BP reaches low train MSE near the predicted \(P\gtrsim Nn_L\) threshold.
+- FA reaches low train MSE near the predicted \(P-K_{\mathrm{FA}}\gtrsim Nn_L\) threshold.
+- The FA/BP train gap is largest where BP has positive margin but FA has negative margin.
+- The transition aligns better with \(P-K_{\mathrm{FA}}-Nn_L\) than with arbitrary width.
+
+Teacher-regression tasks are useful only as auxiliary downstream checks because
+their task dimension is not known before training.
diff --git a/notes/05_downstream_random_capacity_results.md b/notes/05_downstream_random_capacity_results.md
new file mode 100644
index 0000000..b6b0e92
--- /dev/null
+++ b/notes/05_downstream_random_capacity_results.md
@@ -0,0 +1,78 @@
+# Downstream Random-Capacity Results
+
+## Main Run
+
+Output directory:
+
+`outputs/downstream_capacity_random_main_fast`
+
+Task:
+
+\[
+x_i\sim \mathcal N(0,I_{16}),
+\qquad
+y_i\sim \mathcal N(0,I_4),
+\qquad
+N=128.
+\]
+
+Task dimension:
+
+\[
+N n_L=512.
+\]
+
+Model:
+
+\[
+16\to n\to n\to 4.
+\]
+
+Theory:
+
+\[
+P(n)=16n+n^2+4n,
+\]
+
+\[
+K_{\mathrm{FA}}(n)=(n^2-1)+(4n-1),
+\]
+
+\[
+M_{\mathrm{BP}}=P-Nn_L,
+\qquad
+M_{\mathrm{FA}}=P-K_{\mathrm{FA}}-Nn_L.
+\]
+
+Prediction:
+
+- BP threshold at width \(16\).
+- FA threshold at width \(32\).
+
+Trajectory count:
+
+- BP: \(4\) seeds per width.
+- FA: \(4\times 12=48\) trajectories per width.
+- Widths: \(8,12,16,24,32,48,64,96\).
+
+## Summary
+
+| width | BP train | FA train | gap | BP margin | FA margin |
+|---:|---:|---:|---:|---:|---:|
+| 8 | 0.7088 | 1.064 | 0.3555 | -288 | -382 |
+| 12 | 0.2252 | 0.6141 | 0.3889 | -128 | -318 |
+| 16 | 0.00743 | 0.2948 | 0.2873 | 64 | -254 |
+| 24 | 0.000129 | 0.0346 | 0.0345 | 544 | -126 |
+| 32 | 0.000007 | 0.0108 | 0.0108 | 1152 | 2 |
+| 48 | 0.000007 | 0.00392 | 0.00391 | 2752 | 258 |
+| 64 | 0.000046 | 0.00189 | 0.00185 | 4864 | 514 |
+| 96 | 0.000250 | 0.00231 | 0.00206 | 10624 | 1026 |
+
+## Main Figures
+
+- `outputs/downstream_capacity_random_main_fast/train_mse_trajectory_distribution.png`
+- `outputs/downstream_capacity_random_main_fast/train_gap_trajectory_distribution_vs_fa_margin.png`
+- `outputs/downstream_capacity_random_main_fast/bp_fa_train_mse_vs_width.png`
+- `outputs/downstream_capacity_random_main_fast/train_gap_vs_fa_capacity_margin.png`
+
+The cleanest figure is `train_gap_trajectory_distribution_vs_fa_margin.png`: the FA/BP train gap is large when \(M_{\mathrm{FA}}<0\), drops sharply near \(M_{\mathrm{FA}}=0\), and remains small for positive FA margin.