# Theory Notes ## Notation MLP widths: \[ n_0,n_1,\dots,n_L. \] Forward weights: \[ W_l\in \mathbb R^{n_l\times n_{l-1}}. \] For ordinary FA, feedback matrix for layer \(l\): \[ B_l\in \mathbb R^{n_l\times n_{l+1}}, \] which replaces: \[ W_{l+1}^{\top}\in \mathbb R^{n_l\times n_{l+1}}. \] Matrix direction dimension: \[ D_l=n_l n_{l+1}. \] ## Candidate Theorem 1: Static Alignment Distribution Assume: - \(A_l=W_{l+1}^{\top}\). - \(A_l/\|A_l\|_F\) and \(B_l/\|B_l\|_F\) are independent isotropic directions in \(\mathbb R^{D_l}\). Define: \[ Q_l = \frac{ \langle A_l,B_l\rangle_F^2 }{ \|A_l\|_F^2\|B_l\|_F^2 }. \] Then: \[ Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right). \] Consequences: \[ \mathbb E[Q_l]=\frac1{D_l}. \] \[ D_lQ_l \Rightarrow \chi_1^2 \quad \text{as } D_l\to\infty. \] Tail: \[ \Pr(Q_l\ge q) = 1-I_q\left(\frac12,\frac{D_l-1}{2}\right). \] High-dimensional approximation to refine: \[ \Pr(Q_l\ge q) \lesssim 2\exp\left[-\frac{(D_l-1)q}{2}\right]. \] ## Capacity Definition For a required alignment threshold \(q\): \[ C_l(q) = -\log \Pr(Q_l\ge q). \] Under the beta law: \[ C_l(q) = -\log \left[ 1-I_q\left(\frac12,\frac{D_l-1}{2}\right) \right]. \] High-dimensional approximation: \[ C_l(q) \approx \frac{D_l-1}{2}\log\frac1{1-q}. \] Small-\(q\) approximation: \[ C_l(q)\approx \frac{D_l q}{2}. \] Open detail: decide whether logs are natural logs or base-2 bits. Use natural logs in theorem statements unless the paper explicitly wants bit units. ## Candidate Corollary 1: Multilayer Scaling If layerwise feedback matrices are independent and act on distinct matrix blocks, then: \[ p_{\mathrm{all}} = \prod_l \Pr(Q_l\ge q_l), \] and: \[ C_{\mathrm{all}} = \sum_l C_l(q_l). \] Interpretation: - Log-capacity cost is additive. - Raw feasible volume is multiplicative. Equal-width case: \[ D_l\approx n^2. \] Fixed threshold \(q>0\): \[ C_{\mathrm{all}}=\Theta(Ln^2). \] Chance-level threshold \(q=c/D_l\): \[ C_l\approx c/2, \qquad C_{\mathrm{all}}=\Theta(L). \] ## Candidate Theorem 2: Prior-Free Minimax Bound Let: \[ \hat b=\frac{\operatorname{vec}(B)}{\|B\|_F}\in\mathbb S^{D-1}. \] Any feedback initialization distribution \(\mu\) induces: \[ M_\mu=\mathbb E_\mu[\hat b\hat b^\top], \qquad \operatorname{tr}M_\mu=1. \] For target direction \(a\in\mathbb S^{D-1}\): \[ \mathbb E_\mu[(a^\top \hat b)^2]=a^\top M_\mu a. \] Since: \[ \lambda_{\min}(M_\mu)\le \frac{\operatorname{tr}M_\mu}{D}=\frac1D, \] we have: \[ \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] \le \frac1D. \] Thus: \[ \sup_\mu \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] = \frac1D. \] Isotropic initialization attains this bound. ## Prior-Aware Corollary If target directions have prior covariance: \[ \Sigma_A=\mathbb E[aa^\top], \] then: \[ \mathbb E_{a,B}[(a^\top \hat b)^2] = \operatorname{tr}(\Sigma_A M_\mu). \] Therefore the optimal structured feedback distribution depends on the eigenspectrum of \(\Sigma_A\). Without prior information, isotropy is minimax; with prior information, top-eigenspace feedback can be better. ## Functional Capacity Let total parameter dimension be: \[ P=\sum_l n_l(n_{l-1}+1). \] Let task Jacobian be: \[ J=\frac{\partial f_\theta(X)}{\partial \theta}. \] Local task rank: \[ d=\operatorname{rank}(J). \] If alignment constraints remove a generic \(k\)-dimensional parameter subspace, then hard local function rank after constraints is: \[ d_{\mathrm{hard}}=\min(d,P-k). \] Hard functional loss: \[ \Delta d_{\mathrm{hard}} = d-d_{\mathrm{hard}} = \max(0,k-(P-d)). \] Soft overlap model. Let \(E\) be the alignment constraint subspace and \(S\) be the task-sensitive subspace: \[ T_k=\operatorname{tr}(P_E P_S). \] For random subspaces: \[ \mathbb E[T_k]=\frac{kd}{P}. \] Variance: \[ \operatorname{Var}(T_k) = \frac{ 2kd(P-k)(P-d) }{ P^2(P-1)(P+2) }. \] Interpretation: - Hard rank loss has a redundancy-exhaustion threshold. - Soft conditioning loss can grow approximately linearly from the start. ## Trajectory Bridge BP gradient: \[ g_t=\nabla_\theta L(\theta_t). \] FA surrogate gradient: \[ \tilde g_t(B). \] Mismatch: \[ \epsilon_t(B)=\tilde g_t(B)-g_t. \] Linearized deviation from the BP trajectory: \[ \delta\theta_T(B) \approx -\eta \sum_{t