# Actual FA Initial Operator Moment Theorem This note records the first successful attempt to replace the hard-k random-subspace proxy with a theorem about actual FA. ## Problem The hard random-subspace null model says: ```text E[e] = k/P ``` but experiments showed it overpredicts real FA gaps. The reason is that actual FA does not delete a hard `k`-dimensional subspace. It changes the backward operator. We need a quantity closer to real FA. ## Setup Squared loss with residual vector `r`. For each parameter block/layer `l`: ```text g_l^BP = BP gradient block g_l^FA = FA pseudo-gradient block ``` The BP one-step learning speed is: ```text speed_BP = sum_l ||g_l^BP||^2 ``` The FA first-order loss decrease uses the mixed inner product: ```text speed_FA = sum_l ``` Equivalently: ```text speed_BP = r^T K_BP r speed_FA = r^T K_FA r ``` where: ```text K_BP = J J^T K_FA = J J_tilde^T ``` The local operator erosion is: ```text e_0(B) = 1 - speed_FA / speed_BP ``` ## Theorem: Exact Conditional Mean at Initialization Assume: ```text 1. forward weights W are fixed; 2. residual r is fixed; 3. feedback matrices B are independent of W and r; 4. each feedback matrix has zero-mean entries or, more generally, zero first moment in every hidden backward direction; 5. the output layer is trained with the true output gradient, as in standard FA. ``` Then, at initialization: ```text E_B[speed_FA | W, r] = ||g_output^BP||^2 ``` Therefore: ```text E_B[ speed_FA / speed_BP | W, r ] = ||g_output^BP||^2 / sum_l ||g_l^BP||^2 ``` and: ```text E_B[e_0 | W, r] = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2 ``` So the initial effective FA burden is not raw hard constraint count. It is the fraction of BP learning speed carried by hidden-layer gradient blocks: ```text e_0_mean = hidden BP speed share ``` If we want to express it as an effective dimension: ```text k_eff,0 = P * e_0_mean ``` but the more natural object is the dimensionless erosion `e_0_mean`. ## Proof Sketch The output-layer FA gradient equals the BP gradient: ```text g_output^FA = g_output^BP ``` so its mixed contribution is: ```text = ||g_output^BP||^2 ``` Every hidden FA pseudo-gradient contains at least one zero-mean independent feedback matrix. Conditional on fixed forward weights and residuals: ```text E_B[g_l^FA | W, r] = 0 ``` for hidden layers `l`. Thus: ```text E_B[ | W, r] = 0 ``` for every hidden layer. Adding the layer contributions gives the result. ## Relation to Song, Xu, and Lafferty For a two-layer scalar-output network, Song/Xu/Lafferty decompose FA dynamics as: ```text K_FA = G + H_FA ``` where `G` is the top-layer kernel and `H_FA` is the feedback-dependent hidden term. This theorem is the residual-direction version of that decomposition at initialization: ```text E_B[H_FA | W, r] = 0 ``` so: ```text E_B[K_FA | W] = K_output ``` not: ```text E_B[K_FA | W] = (1 - k/P) K_BP ``` This explains why the hard-k random-subspace theory was too pessimistic. ## Distribution, Not Just Mean For one-hidden-layer scalar-output FA with Gaussian feedback, the hidden mixed speed is exactly Gaussian conditional on `W` and `r`. It can be written as: ```text speed_FA = output_speed + Z ``` where: ```text E_B[Z | W, r] = 0 Var_B[Z | W, r] = feedback_variance * ||C(W,r)||^2 ``` for an explicit coefficient tensor `C(W,r)` determined by activations, gates, residuals, and BP hidden gradients. Therefore: ```text e_0(B) = 1 - (output_speed + Z) / speed_BP ``` has an explicit conditional Gaussian distribution in the one-hidden-layer Gaussian-feedback case. For deeper FA, hidden terms contain products of feedback matrices. The exact mean above still holds, but the finite-width distribution is no longer exactly Gaussian. A CLT/Wick expansion should give a computable approximation. ## Infinite-Width Interpretation At infinite width, layerwise BP speed shares should converge to deterministic NTK/NNGP quantities: ```text output_share = r^T K_output r / r^T K_BP r hidden_share = 1 - output_share ``` Thus an architecture-level prediction may be obtained from layerwise NTK recursions: ```text E_B[e_0] -> hidden_share_NTK ``` This gives a principled route to an architecture/data-level prediction without post-hoc scaling. ## Time-Varying Extension At later time: ```text speed_FA,t = output_speed_t + hidden_alignment_gain_t ``` where: ```text hidden_alignment_gain_t = sum_hidden ``` At initialization: ```text E_B[hidden_alignment_gain_0 | W,r] = 0 ``` During training, forward weights become dependent on feedback matrices, so this term can become positive. This is actual feedback alignment. Therefore: ```text e_t = 1 - (output_speed_t + hidden_alignment_gain_t) / speed_BP,t ``` The current tangent-operator estimator works because it measures early movement of this alignment gain. A future theorem should bound or approximate: ```text hidden_alignment_gain_t ``` from the early dynamics, possibly using the perturbative `H_FA` analysis style from Song/Xu/Lafferty. ## Empirical Validation Script: ```text scripts/actual_fa_initial_operator_moments.py ``` Run: ```text python scripts/actual_fa_initial_operator_moments.py \ --widths 16 24 32 48 64 96 \ --init-seeds 6 \ --feedback-samples 512 \ --torch-threads 8 \ --outdir outputs/actual_fa_initial_operator_moments ``` Outputs: ```text outputs/actual_fa_initial_operator_moments/initial_operator_moment_rows.csv outputs/actual_fa_initial_operator_moments/predicted_vs_empirical_initial_erosion_mean.png outputs/actual_fa_initial_operator_moments/initial_erosion_mean_error_by_width.png outputs/actual_fa_initial_operator_moments/initial_erosion_histograms.png ``` Representative results: | width | init | theory mean erosion | empirical mean erosion | error | |---:|---:|---:|---:|---:| | 16 | 0 | 0.451931 | 0.444348 | -0.00758 | | 24 | 0 | 0.312016 | 0.312286 | 0.00027 | | 32 | 0 | 0.236058 | 0.239714 | 0.00366 | | 48 | 0 | 0.261471 | 0.259858 | -0.00161 | | 64 | 0 | 0.209125 | 0.211261 | 0.00214 | | 96 | 0 | 0.259765 | 0.259147 | -0.00062 | Across all tested widths and initializations, the no-fit mean prediction is accurate to Monte Carlo error. ## Consequence for Contributions This improves contribution 2: ```text hard-k random-subspace erosion = clean null model actual FA initial erosion = hidden BP speed share theorem ``` Contribution 4 still handles finite-time gap: ```text initial theorem gives e_0 operator estimator tracks e_t and converts it into finite-time gap ``` This is a stronger and cleaner story than trying to force raw capacity margin to predict the full trajectory gap.