Initialize project notes

1 files changed, 356 insertions, 0 deletions

diff --git a/notes/01_theory_notes.md b/notes/01_theory_notes.md
new file mode 100644
index 0000000..c0ac8fe
--- /dev/null
+++ b/notes/01_theory_notes.md
@@ -0,0 +1,356 @@
+# Theory Notes
+
+## Notation
+
+MLP widths:
+
+\[
+n_0,n_1,\dots,n_L.
+\]
+
+Forward weights:
+
+\[
+W_l\in \mathbb R^{n_l\times n_{l-1}}.
+\]
+
+For ordinary FA, feedback matrix for layer \(l\):
+
+\[
+B_l\in \mathbb R^{n_l\times n_{l+1}},
+\]
+
+which replaces:
+
+\[
+W_{l+1}^{\top}\in \mathbb R^{n_l\times n_{l+1}}.
+\]
+
+Matrix direction dimension:
+
+\[
+D_l=n_l n_{l+1}.
+\]
+
+## Candidate Theorem 1: Static Alignment Distribution
+
+Assume:
+
+- \(A_l=W_{l+1}^{\top}\).
+- \(A_l/\|A_l\|_F\) and \(B_l/\|B_l\|_F\) are independent isotropic directions in \(\mathbb R^{D_l}\).
+
+Define:
+
+\[
+Q_l =
+\frac{
+\langle A_l,B_l\rangle_F^2
+}{
+\|A_l\|_F^2\|B_l\|_F^2
+}.
+\]
+
+Then:
+
+\[
+Q_l\sim
+\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
+\]
+
+Consequences:
+
+\[
+\mathbb E[Q_l]=\frac1{D_l}.
+\]
+
+\[
+D_lQ_l \Rightarrow \chi_1^2
+\quad
+\text{as } D_l\to\infty.
+\]
+
+Tail:
+
+\[
+\Pr(Q_l\ge q)
+=
+1-I_q\left(\frac12,\frac{D_l-1}{2}\right).
+\]
+
+High-dimensional approximation to refine:
+
+\[
+\Pr(Q_l\ge q)
+\lesssim
+2\exp\left[-\frac{(D_l-1)q}{2}\right].
+\]
+
+## Capacity Definition
+
+For a required alignment threshold \(q\):
+
+\[
+C_l(q)
+=
+-\log \Pr(Q_l\ge q).
+\]
+
+Under the beta law:
+
+\[
+C_l(q)
+=
+-\log
+\left[
+1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
+\right].
+\]
+
+High-dimensional approximation:
+
+\[
+C_l(q)
+\approx
+\frac{D_l-1}{2}\log\frac1{1-q}.
+\]
+
+Small-\(q\) approximation:
+
+\[
+C_l(q)\approx \frac{D_l q}{2}.
+\]
+
+Open detail: decide whether logs are natural logs or base-2 bits. Use natural logs in theorem statements unless the paper explicitly wants bit units.
+
+## Candidate Corollary 1: Multilayer Scaling
+
+If layerwise feedback matrices are independent and act on distinct matrix blocks, then:
+
+\[
+p_{\mathrm{all}}
+=
+\prod_l
+\Pr(Q_l\ge q_l),
+\]
+
+and:
+
+\[
+C_{\mathrm{all}}
+=
+\sum_l C_l(q_l).
+\]
+
+Interpretation:
+
+- Log-capacity cost is additive.
+- Raw feasible volume is multiplicative.
+
+Equal-width case:
+
+\[
+D_l\approx n^2.
+\]
+
+Fixed threshold \(q>0\):
+
+\[
+C_{\mathrm{all}}=\Theta(Ln^2).
+\]
+
+Chance-level threshold \(q=c/D_l\):
+
+\[
+C_l\approx c/2,
+\qquad
+C_{\mathrm{all}}=\Theta(L).
+\]
+
+## Candidate Theorem 2: Prior-Free Minimax Bound
+
+Let:
+
+\[
+\hat b=\frac{\operatorname{vec}(B)}{\|B\|_F}\in\mathbb S^{D-1}.
+\]
+
+Any feedback initialization distribution \(\mu\) induces:
+
+\[
+M_\mu=\mathbb E_\mu[\hat b\hat b^\top],
+\qquad
+\operatorname{tr}M_\mu=1.
+\]
+
+For target direction \(a\in\mathbb S^{D-1}\):
+
+\[
+\mathbb E_\mu[(a^\top \hat b)^2]=a^\top M_\mu a.
+\]
+
+Since:
+
+\[
+\lambda_{\min}(M_\mu)\le \frac{\operatorname{tr}M_\mu}{D}=\frac1D,
+\]
+
+we have:
+
+\[
+\inf_{\|a\|=1}
+\mathbb E_\mu[(a^\top \hat b)^2]
+\le
+\frac1D.
+\]
+
+Thus:
+
+\[
+\sup_\mu
+\inf_{\|a\|=1}
+\mathbb E_\mu[(a^\top \hat b)^2]
+=
+\frac1D.
+\]
+
+Isotropic initialization attains this bound.
+
+## Prior-Aware Corollary
+
+If target directions have prior covariance:
+
+\[
+\Sigma_A=\mathbb E[aa^\top],
+\]
+
+then:
+
+\[
+\mathbb E_{a,B}[(a^\top \hat b)^2]
+=
+\operatorname{tr}(\Sigma_A M_\mu).
+\]
+
+Therefore the optimal structured feedback distribution depends on the eigenspectrum of \(\Sigma_A\). Without prior information, isotropy is minimax; with prior information, top-eigenspace feedback can be better.
+
+## Functional Capacity
+
+Let total parameter dimension be:
+
+\[
+P=\sum_l n_l(n_{l-1}+1).
+\]
+
+Let task Jacobian be:
+
+\[
+J=\frac{\partial f_\theta(X)}{\partial \theta}.
+\]
+
+Local task rank:
+
+\[
+d=\operatorname{rank}(J).
+\]
+
+If alignment constraints remove a generic \(k\)-dimensional parameter subspace, then hard local function rank after constraints is:
+
+\[
+d_{\mathrm{hard}}=\min(d,P-k).
+\]
+
+Hard functional loss:
+
+\[
+\Delta d_{\mathrm{hard}}
+=
+d-d_{\mathrm{hard}}
+=
+\max(0,k-(P-d)).
+\]
+
+Soft overlap model. Let \(E\) be the alignment constraint subspace and \(S\) be the task-sensitive subspace:
+
+\[
+T_k=\operatorname{tr}(P_E P_S).
+\]
+
+For random subspaces:
+
+\[
+\mathbb E[T_k]=\frac{kd}{P}.
+\]
+
+Variance:
+
+\[
+\operatorname{Var}(T_k)
+=
+\frac{
+2kd(P-k)(P-d)
+}{
+P^2(P-1)(P+2)
+}.
+\]
+
+Interpretation:
+
+- Hard rank loss has a redundancy-exhaustion threshold.
+- Soft conditioning loss can grow approximately linearly from the start.
+
+## Trajectory Bridge
+
+BP gradient:
+
+\[
+g_t=\nabla_\theta L(\theta_t).
+\]
+
+FA surrogate gradient:
+
+\[
+\tilde g_t(B).
+\]
+
+Mismatch:
+
+\[
+\epsilon_t(B)=\tilde g_t(B)-g_t.
+\]
+
+Linearized deviation from the BP trajectory:
+
+\[
+\delta\theta_T(B)
+\approx
+-\eta
+\sum_{t<T}
+\Phi_{T,t+1}\epsilon_t(B),
+\]
+
+where:
+
+\[
+\Phi_{T,t+1}
+=
+\prod_{s=t+1}^{T-1}(I-\eta H_s).
+\]
+
+Second-order excess loss:
+
+\[
+\Delta L_T(B)
+\approx
+\frac12
+\delta\theta_T(B)^\top H_T\delta\theta_T(B).
+\]
+
+If \(\epsilon_t(B)\) is approximately Gaussian under random \(B\), then:
+
+\[
+\Delta L_T(B)
+\approx
+\frac12\sum_i \lambda_i\chi_i^2.
+\]
+
+This should be treated as a bridge approximation, not as a primary architecture-only theorem.