# Theory Notes ## Notation MLP widths: \[ n_0,n_1,\dots,n_L. \] Forward weights: \[ W_l\in \mathbb R^{n_l\times n_{l-1}}. \] For ordinary FA, feedback matrix for layer \(l\): \[ B_l\in \mathbb R^{n_l\times n_{l+1}}, \] which replaces: \[ W_{l+1}^{\top}\in \mathbb R^{n_l\times n_{l+1}}. \] Matrix direction dimension: \[ D_l=n_l n_{l+1}. \] ## Candidate Theorem 1: Static Alignment Distribution Assume: - \(A_l=W_{l+1}^{\top}\). - \(A_l/\|A_l\|_F\) and \(B_l/\|B_l\|_F\) are independent isotropic directions in \(\mathbb R^{D_l}\). Define: \[ Q_l = \frac{ \langle A_l,B_l\rangle_F^2 }{ \|A_l\|_F^2\|B_l\|_F^2 }. \] Then: \[ Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right). \] Consequences: \[ \mathbb E[Q_l]=\frac1{D_l}. \] \[ D_lQ_l \Rightarrow \chi_1^2 \quad \text{as } D_l\to\infty. \] Tail: \[ \Pr(Q_l\ge q) = 1-I_q\left(\frac12,\frac{D_l-1}{2}\right). \] High-dimensional approximation to refine: \[ \Pr(Q_l\ge q) \lesssim 2\exp\left[-\frac{(D_l-1)q}{2}\right]. \] ## Capacity Definition For a required alignment threshold \(q\): \[ C_l(q) = -\log \Pr(Q_l\ge q). \] Under the beta law: \[ C_l(q) = -\log \left[ 1-I_q\left(\frac12,\frac{D_l-1}{2}\right) \right]. \] High-dimensional approximation: \[ C_l(q) \approx \frac{D_l-1}{2}\log\frac1{1-q}. \] Small-\(q\) approximation: \[ C_l(q)\approx \frac{D_l q}{2}. \] Open detail: decide whether logs are natural logs or base-2 bits. Use natural logs in theorem statements unless the paper explicitly wants bit units. ## Candidate Corollary 1: Multilayer Scaling If layerwise feedback matrices are independent and act on distinct matrix blocks, then: \[ p_{\mathrm{all}} = \prod_l \Pr(Q_l\ge q_l), \] and: \[ C_{\mathrm{all}} = \sum_l C_l(q_l). \] Interpretation: - Log-capacity cost is additive. - Raw feasible volume is multiplicative. Equal-width case: \[ D_l\approx n^2. \] Fixed threshold \(q>0\): \[ C_{\mathrm{all}}=\Theta(Ln^2). \] Chance-level threshold \(q=c/D_l\): \[ C_l\approx c/2, \qquad C_{\mathrm{all}}=\Theta(L). \] ## Candidate Corollary 2: Observed Capacity Distribution The threshold cost \(C_l(q)\) is deterministic once \(q\) is chosen. For empirical distribution matching, define the observed alignment surprisal: \[ S_l = -\log \Pr(Q_l'\ge Q_l), \] where \(Q_l'\) is an independent draw from the same beta law as \(Q_l\). If: \[ Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right), \] and the survival function is continuous, then by the probability integral transform: \[ U_l=\Pr(Q_l'\ge Q_l)\sim \mathrm{Uniform}(0,1), \] so: \[ S_l=-\log U_l\sim \mathrm{Exp}(1). \] For \(L\) independent feedback-aligned layers, the observed total capacity surprisal is: \[ S_{1:L}=\sum_{l=1}^L S_l. \] If the layerwise feedback matrices are independent, then: \[ S_{1:L}\sim \mathrm{Gamma}(L,1). \] This is useful because it predicts a full distribution rather than only a mean or bound. It also shows that after the beta-tail transform, the null distribution of observed capacity surprisal is dimension-free; dimension enters through the raw \(Q_l\) scale and the threshold cost \(C_l(q)\), not through the transformed surprisal law. ## Candidate Theorem 2: Prior-Free Minimax Bound Let: \[ \hat b=\frac{\operatorname{vec}(B)}{\|B\|_F}\in\mathbb S^{D-1}. \] Any feedback initialization distribution \(\mu\) induces: \[ M_\mu=\mathbb E_\mu[\hat b\hat b^\top], \qquad \operatorname{tr}M_\mu=1. \] For target direction \(a\in\mathbb S^{D-1}\): \[ \mathbb E_\mu[(a^\top \hat b)^2]=a^\top M_\mu a. \] Since: \[ \lambda_{\min}(M_\mu)\le \frac{\operatorname{tr}M_\mu}{D}=\frac1D, \] we have: \[ \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] \le \frac1D. \] Thus: \[ \sup_\mu \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] = \frac1D. \] Isotropic initialization attains this bound. ### Proof Sketch The proof only needs second moments. For any feedback initialization \(\mu\), define: \[ M_\mu=\mathbb E_\mu[\hat b\hat b^\top]. \] Since \(\|\hat b\|=1\): \[ \operatorname{tr}M_\mu = \mathbb E_\mu[\operatorname{tr}(\hat b\hat b^\top)] = \mathbb E_\mu[\|\hat b\|^2] =1. \] For any fixed target direction \(a\): \[ \mathbb E_\mu[(a^\top \hat b)^2] = \mathbb E_\mu[a^\top \hat b\hat b^\top a] = a^\top M_\mu a. \] The worst-case target direction is the minimum-eigenvalue direction of \(M_\mu\): \[ \inf_{\|a\|=1} a^\top M_\mu a = \lambda_{\min}(M_\mu). \] Because the average eigenvalue is \(1/D\): \[ \lambda_{\min}(M_\mu) \le \frac1D. \] Hence: \[ \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] \le \frac1D \] for every initialization \(\mu\). If \(\mu\) is isotropic, then: \[ M_\mu=\frac1D I, \] so every direction has: \[ a^\top M_\mu a=\frac1D. \] Thus: \[ \sup_\mu \inf_{\|a\|=1} \mathbb E_\mu[(a^\top \hat b)^2] = \frac1D. \] This is a worst-case theorem. If the target direction is itself uniformly random, then: \[ \mathbb E_a[a^\top M_\mu a] = \frac{\operatorname{tr}M_\mu}{D} = \frac1D \] for every \(\mu\). Therefore prior-free average-case alignment over uniform targets cannot distinguish initializations by their mean; the distinction is worst-case coverage, tail behavior, and conditioning. ## Prior-Aware Corollary If target directions have prior covariance: \[ \Sigma_A=\mathbb E[aa^\top], \] then: \[ \mathbb E_{a,B}[(a^\top \hat b)^2] = \operatorname{tr}(\Sigma_A M_\mu). \] Therefore the optimal structured feedback distribution depends on the eigenspectrum of \(\Sigma_A\). Without prior information, isotropy is minimax; with prior information, top-eigenspace feedback can be better. ## Functional Capacity Let total parameter dimension be: \[ P=\sum_l n_l(n_{l-1}+1). \] Let task Jacobian be: \[ J=\frac{\partial f_\theta(X)}{\partial \theta}. \] Local task rank: \[ d=\operatorname{rank}(J). \] If alignment constraints remove a generic \(k\)-dimensional parameter subspace, then hard local function rank after constraints is: \[ d_{\mathrm{hard}}=\min(d,P-k). \] Hard functional loss: \[ \Delta d_{\mathrm{hard}} = d-d_{\mathrm{hard}} = \max(0,k-(P-d)). \] Interpretation: - \(P-d\) is the local redundant parameter dimension. - If \(k\le P-d\), generic alignment constraints can be absorbed by redundant directions without reducing hard local function rank. - If \(k>P-d\), every additional generic constraint reduces hard local function rank one-for-one. This gives the proposed FA/BP functional-gap onset: parameter-volume cost can grow before the model loses hard function rank, but once the alignment burden exhausts redundancy, FA should separate from BP more sharply. Soft overlap model. Let \(E\) be the alignment constraint subspace and \(S\) be the task-sensitive subspace: \[ T_k=\operatorname{tr}(P_E P_S). \] For random subspaces: \[ \mathbb E[T_k]=\frac{kd}{P}. \] Variance: \[ \operatorname{Var}(T_k) = \frac{ 2kd(P-k)(P-d) }{ P^2(P-1)(P+2) }. \] Interpretation: - Hard rank loss has a redundancy-exhaustion threshold. - Soft conditioning loss can grow approximately linearly from the start. ## Trajectory Bridge BP gradient: \[ g_t=\nabla_\theta L(\theta_t). \] FA surrogate gradient: \[ \tilde g_t(B). \] Mismatch: \[ \epsilon_t(B)=\tilde g_t(B)-g_t. \] Linearized deviation from the BP trajectory: \[ \delta\theta_T(B) \approx -\eta \sum_{t