1 files changed, 250 insertions, 0 deletions
diff --git a/notes/04_downstream_capacity_design.md b/notes/04_downstream_capacity_design.md
new file mode 100644
index 0000000..c2b9704
--- /dev/null
+++ b/notes/04_downstream_capacity_design.md
@@ -0,0 +1,250 @@
+# Downstream Capacity Experiment Design
+
+## Goal
+
+Validate the core downstream claim:
+
+> FA and BP should behave similarly while redundant parameter capacity can absorb the FA alignment burden, and FA should degrade faster once redundancy is exhausted.
+
+For random-label memorization, the primary performance metric is train MSE. The
+experiment should produce two performance curves:
+
+\[
+L_{\mathrm{BP}}(P)
+\quad \text{and} \quad
+L_{\mathrm{FA}}(P),
+\]
+
+where \(P\) is model parameter count. The target signature is a widening FA/BP
+train gap as \(P\) is reduced, with the widening centered near the predicted FA
+capacity margin.
+
+## Main Task Choice: Random-Label Memorization
+
+Use:
+
+\[
+x_i\sim \mathcal N(0,I_{n_0}),
+\qquad
+y_i\sim \mathcal N(0,I_{n_L}).
+\]
+
+The target is train-set memorization, not test generalization. This makes the
+task dimension directly measurable:
+
+\[
+d_{\mathrm{task}}\approx N n_L.
+\]
+
+For a two-hidden-layer MLP with no bias,
+
+\[
+P(n)=n_0 n+n^2+n n_L.
+\]
+
+For ordinary FA, the hard alignment-burden proxy is:
+
+\[
+K_{\mathrm{FA}}(n)
+=
+(n^2-1)+(n n_L-1).
+\]
+
+The predicted thresholds are:
+
+\[
+P(n)\gtrsim Nn_L
+\qquad \text{for BP},
+\]
+
+and
+
+\[
+P(n)-K_{\mathrm{FA}}(n)\gtrsim Nn_L
+\qquad \text{for FA}.
+\]
+
+This is the cleanest task for testing the redundancy-exhaustion story because
+the breakpoint is predicted before training.
+
+## Auxiliary Task Choice: Low-Rank Linear Teacher
+
+Use:
+
+\[
+x\sim \mathcal N(0,I_{n_0}),
+\]
+
+\[
+y=A x + \epsilon,
+\]
+
+where:
+
+\[
+A=U\Sigma V^\top,\qquad \operatorname{rank}(A)=r.
+\]
+
+Reasons:
+
+- Input/output dimensions are explicit.
+- Task rank is controlled.
+- Test loss is easy to measure.
+- It is simple enough that BP should establish a clear capacity baseline.
+
+Open risk:
+
+- The task may be too easy for ReLU MLPs, and FA may not show a clean transition unless width is very small.
+
+## Architecture
+
+Use a two-hidden-layer ReLU MLP:
+
+\[
+n_0 \to n \to n \to n_L.
+\]
+
+Sweep hidden width:
+
+\[
+n\in\{2,3,4,6,8,12,16,24,32,48,64\}.
+\]
+
+Parameter count without bias:
+
+\[
+P(n)=n_0 n+n^2+n n_L.
+\]
+
+Depth is fixed so that the main knob is capacity, not feedback path length.
+
+## Training
+
+For each width:
+
+- Train BP baseline.
+- Train FA with fixed random feedback matrices.
+- Use same initial forward weights for matched BP/FA comparisons.
+- Use multiple data/init seeds and multiple feedback seeds.
+
+Initial smoke sweep:
+
+- widths: \(\{2,4,8,16,32\}\)
+- init seeds: `3`
+- feedback seeds per init: `5`
+
+Scale-up sweep:
+
+- widths: `2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64`
+- init seeds: `10`
+- feedback seeds per init: `20`
+
+## Theoretical Quantities to Estimate
+
+### Parameter Count
+
+\[
+P=\sum_l n_l n_{l-1}.
+\]
+
+### Task Effective Dimension
+
+At the BP-trained endpoint, estimate:
+
+\[
+J=\frac{\partial f_\theta(X)}{\partial \theta}
+\]
+
+on a probe batch. Let:
+
+\[
+K=JJ^\top.
+\]
+
+Effective dimension:
+
+\[
+d_{\mathrm{eff}}(\lambda)
+=
+\operatorname{tr}\left[K(K+\lambda I)^{-1}\right].
+\]
+
+This estimates how many functional directions the task uses locally.
+
+### Hard FA Burden
+
+For the random-label memorization task, use the hard alignment-burden proxy:
+
+\[
+K_{\mathrm{FA}}=\sum_l(D_l-1).
+\]
+
+For ordinary FA in a two-hidden-layer MLP:
+
+\[
+D_1=n^2,\qquad D_2=n n_L,
+\]
+
+so:
+
+\[
+K_{\mathrm{FA}}=(n^2-1)+(n n_L-1).
+\]
+
+### Log-Volume FA Burden
+
+For matrix-level capacity validation, keep the log-volume quantity:
+
+\[
+k_{\mathrm{FA}}^{\mathrm{proxy}}
+=
+\sum_l C_l(q),
+\]
+
+where:
+
+\[
+C_l(q)=
+-\log P(Q_l\ge q),
+\qquad
+Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
+\]
+
+Use a fixed threshold \(q\) for the first sweep. This is not a hard constraint
+count; it is the matrix-level log-volume cost used in the direct capacity
+distribution experiments.
+
+### Redundancy Score
+
+Define:
+
+\[
+R_{\mathrm{eff}}
+=
+P-d_{\mathrm{eff}}-K_{\mathrm{FA}}.
+\]
+
+For random-label memorization, also use the direct theoretical margin:
+
+\[
+M_{\mathrm{FA}}=P-K_{\mathrm{FA}}-Nn_L.
+\]
+
+## Main Plots
+
+1. BP and FA train MSE vs hidden width or parameter count.
+2. FA/BP train MSE gap vs hidden width or parameter count.
+3. FA/BP train MSE gap vs predicted FA margin \(P-K_{\mathrm{FA}}-Nn_L\).
+4. Estimated \(d_{\mathrm{eff}}\), \(k_{\mathrm{FA}}^{\mathrm{proxy}}\), and \(P\) vs width.
+
+## Success Criterion
+
+The experiment supports the contribution if:
+
+- BP reaches low train MSE near the predicted \(P\gtrsim Nn_L\) threshold.
+- FA reaches low train MSE near the predicted \(P-K_{\mathrm{FA}}\gtrsim Nn_L\) threshold.
+- The FA/BP train gap is largest where BP has positive margin but FA has negative margin.
+- The transition aligns better with \(P-K_{\mathrm{FA}}-Nn_L\) than with arbitrary width.
+
+Teacher-regression tasks are useful only as auxiliary downstream checks because
+their task dimension is not known before training.