Derive actual FA initial operator moment

3 files changed, 393 insertions, 6 deletions

diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md
index be4d775..aa26caf 100644
--- a/notes/18_contribution_roadmap.md
+++ b/notes/18_contribution_roadmap.md
@@ -93,6 +93,33 @@ conditioning loss, or effective-dimension loss.
 
 This is the soft capacity-erosion contribution.
 
+There is now a cleaner actual-FA initialization theorem that should be used
+before the random-subspace null model. For fixed forward weights and residuals,
+with independent zero-mean feedback matrices:
+
+```text
+E_B[speed_FA | W, r] = ||g_output^BP||^2
+```
+
+where:
+
+```text
+speed_FA = sum_l <g_l^BP, g_l^FA>
+speed_BP = sum_l ||g_l^BP||^2
+```
+
+Therefore:
+
+```text
+E_B[e_0 | W, r]
+  = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2
+```
+
+So the true initial FA burden is the hidden-layer BP speed share, not the raw
+hard feedback-matrix constraint count. The hard-k random-subspace model remains
+a useful null model and capacity sanity check, but it should not be presented
+as the exact FA gap theorem.
+
 ## 3. Prior-Free Minimax Initialization Bound
 
 Let `b` be the normalized feedback direction in `D` dimensions and `a` be the
@@ -169,11 +196,16 @@ Experiments should validate three levels:
    ```
 2. capacity and soft erosion:
    FA/BP gap distribution changes smoothly with hard FA capacity margin;
-3. tangent-operator trajectory distributions:
+3. actual-FA initial operator moments:
+   ```text
+   E_B[e_0 | W,r] = hidden BP speed share
+   ```
+4. tangent-operator trajectory distributions:
    predicted gap distributions overlap empirical trajectory distributions.
 
 Current strong evidence:
 
+- initial actual-FA operator moment validation over widths and feedback seeds;
 - 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`;
 - stress grid over depth, width, and horizon;
 - systematic low-bias analysis showing residual error is kernel-path curvature,
diff --git a/notes/28_contribution_status_proof_vs_fit.md b/notes/28_contribution_status_proof_vs_fit.md
index 40c1118..5507de7 100644
--- a/notes/28_contribution_status_proof_vs_fit.md
+++ b/notes/28_contribution_status_proof_vs_fit.md
@@ -17,7 +17,7 @@ Our current project should avoid calling every component a theorem.
 | contribution | current status | fitted? | safe main-paper phrasing |
 |---|---|---:|---|
 | 1. capacity formalization | definition + exact random-geometry theorem | no | exact under isotropic feedback assumptions |
-| 2. scaling and soft erosion | exact scaling + exact random-subspace null model; real FA extension not yet proved | no, but partly only a null model | capacity cost scales exactly; random-subspace erosion gives a baseline, while real FA needs `k_eff` |
+| 2. scaling and soft erosion | exact scaling + exact random-subspace null model + actual-FA initial moment theorem | no, but finite-time FA still needs dynamics | capacity cost scales exactly; actual FA has a no-fit `e_0` theorem, while `t>0` needs operator dynamics |
 | 3. prior-free minimax bound | exact theorem | no | minimax optimality for prior-free angular alignment |
 | 4. tangent-operator estimator | first-order tangent derivation + conditional predictor | no scalar fit, but uses early observed operators | estimator for finite-time gap, not architecture-only theorem |
 | 5. empirical distribution validation | experiment | no | validates which theoretical distributions and estimators match trajectories |
@@ -132,6 +132,31 @@ hard-k random-subspace formula exactly predicts real FA/BP gap
 
 This is false in our current experiments.
 
+We now have a stronger actual-FA initialization theorem:
+
+```text
+speed_BP = sum_l ||g_l^BP||^2
+speed_FA = sum_l <g_l^BP, g_l^FA>
+e_0 = 1 - speed_FA / speed_BP
+```
+
+For fixed forward weights and residuals, with independent zero-mean feedback:
+
+```text
+E_B[speed_FA | W,r] = ||g_output^BP||^2
+```
+
+Therefore:
+
+```text
+E_B[e_0 | W,r]
+  = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2
+```
+
+This is actual FA, not a random-subspace proxy. It says the initial effective
+burden is the hidden-layer BP speed share. It does not yet solve the full
+finite-time problem because training makes `W_t` depend on `B`.
+
 ## 3. Prior-Free Minimax Initialization Bound
 
 This is proof-level.
@@ -305,9 +330,12 @@ The paper can honestly claim:
 The current unresolved theoretical gap:
 
 ```text
-derive k_eff or e_t distribution for actual FA directly from architecture and
-feedback initialization, rather than measuring early tangent operators.
+derive the t>0 distribution of k_eff(t) or e_t for actual FA directly from
+architecture and feedback initialization, rather than measuring early tangent
+operators.
 ```
 
-Until that is solved, contribution 2 should not overpromise exact FA gap
-prediction, and contribution 4 should be the quantitative gap tool.
+The t=0 conditional mean is now solved by the actual-FA initial moment theorem.
+Until the t>0 dynamics are solved, contribution 2 should not overpromise exact
+finite-time FA gap prediction, and contribution 4 should remain the
+quantitative finite-time gap tool.
diff --git a/notes/29_actual_fa_initial_operator_moment_theorem.md b/notes/29_actual_fa_initial_operator_moment_theorem.md
new file mode 100644
index 0000000..5814f06
--- /dev/null
+++ b/notes/29_actual_fa_initial_operator_moment_theorem.md
@@ -0,0 +1,327 @@
+# Actual FA Initial Operator Moment Theorem
+
+This note records the first successful attempt to replace the hard-k
+random-subspace proxy with a theorem about actual FA.
+
+## Problem
+
+The hard random-subspace null model says:
+
+```text
+E[e] = k/P
+```
+
+but experiments showed it overpredicts real FA gaps. The reason is that actual
+FA does not delete a hard `k`-dimensional subspace. It changes the backward
+operator.
+
+We need a quantity closer to real FA.
+
+## Setup
+
+Squared loss with residual vector `r`.
+
+For each parameter block/layer `l`:
+
+```text
+g_l^BP = BP gradient block
+g_l^FA = FA pseudo-gradient block
+```
+
+The BP one-step learning speed is:
+
+```text
+speed_BP = sum_l ||g_l^BP||^2
+```
+
+The FA first-order loss decrease uses the mixed inner product:
+
+```text
+speed_FA = sum_l <g_l^BP, g_l^FA>
+```
+
+Equivalently:
+
+```text
+speed_BP = r^T K_BP r
+speed_FA = r^T K_FA r
+```
+
+where:
+
+```text
+K_BP = J J^T
+K_FA = J J_tilde^T
+```
+
+The local operator erosion is:
+
+```text
+e_0(B) = 1 - speed_FA / speed_BP
+```
+
+## Theorem: Exact Conditional Mean at Initialization
+
+Assume:
+
+```text
+1. forward weights W are fixed;
+2. residual r is fixed;
+3. feedback matrices B are independent of W and r;
+4. each feedback matrix has zero-mean entries or, more generally, zero
+   first moment in every hidden backward direction;
+5. the output layer is trained with the true output gradient, as in standard FA.
+```
+
+Then, at initialization:
+
+```text
+E_B[speed_FA | W, r] = ||g_output^BP||^2
+```
+
+Therefore:
+
+```text
+E_B[ speed_FA / speed_BP | W, r ]
+  = ||g_output^BP||^2 / sum_l ||g_l^BP||^2
+```
+
+and:
+
+```text
+E_B[e_0 | W, r]
+  = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2
+```
+
+So the initial effective FA burden is not raw hard constraint count. It is the
+fraction of BP learning speed carried by hidden-layer gradient blocks:
+
+```text
+e_0_mean = hidden BP speed share
+```
+
+If we want to express it as an effective dimension:
+
+```text
+k_eff,0 = P * e_0_mean
+```
+
+but the more natural object is the dimensionless erosion `e_0_mean`.
+
+## Proof Sketch
+
+The output-layer FA gradient equals the BP gradient:
+
+```text
+g_output^FA = g_output^BP
+```
+
+so its mixed contribution is:
+
+```text
+<g_output^BP, g_output^FA> = ||g_output^BP||^2
+```
+
+Every hidden FA pseudo-gradient contains at least one zero-mean independent
+feedback matrix. Conditional on fixed forward weights and residuals:
+
+```text
+E_B[g_l^FA | W, r] = 0
+```
+
+for hidden layers `l`.
+
+Thus:
+
+```text
+E_B[<g_l^BP, g_l^FA> | W, r] = 0
+```
+
+for every hidden layer.
+
+Adding the layer contributions gives the result.
+
+## Relation to Song, Xu, and Lafferty
+
+For a two-layer scalar-output network, Song/Xu/Lafferty decompose FA dynamics
+as:
+
+```text
+K_FA = G + H_FA
+```
+
+where `G` is the top-layer kernel and `H_FA` is the feedback-dependent hidden
+term. This theorem is the residual-direction version of that decomposition at
+initialization:
+
+```text
+E_B[H_FA | W, r] = 0
+```
+
+so:
+
+```text
+E_B[K_FA | W] = K_output
+```
+
+not:
+
+```text
+E_B[K_FA | W] = (1 - k/P) K_BP
+```
+
+This explains why the hard-k random-subspace theory was too pessimistic.
+
+## Distribution, Not Just Mean
+
+For one-hidden-layer scalar-output FA with Gaussian feedback, the hidden mixed
+speed is exactly Gaussian conditional on `W` and `r`.
+
+It can be written as:
+
+```text
+speed_FA = output_speed + Z
+```
+
+where:
+
+```text
+E_B[Z | W, r] = 0
+Var_B[Z | W, r] = feedback_variance * ||C(W,r)||^2
+```
+
+for an explicit coefficient tensor `C(W,r)` determined by activations, gates,
+residuals, and BP hidden gradients.
+
+Therefore:
+
+```text
+e_0(B) = 1 - (output_speed + Z) / speed_BP
+```
+
+has an explicit conditional Gaussian distribution in the one-hidden-layer
+Gaussian-feedback case.
+
+For deeper FA, hidden terms contain products of feedback matrices. The exact
+mean above still holds, but the finite-width distribution is no longer exactly
+Gaussian. A CLT/Wick expansion should give a computable approximation.
+
+## Infinite-Width Interpretation
+
+At infinite width, layerwise BP speed shares should converge to deterministic
+NTK/NNGP quantities:
+
+```text
+output_share = r^T K_output r / r^T K_BP r
+hidden_share = 1 - output_share
+```
+
+Thus an architecture-level prediction may be obtained from layerwise NTK
+recursions:
+
+```text
+E_B[e_0] -> hidden_share_NTK
+```
+
+This gives a principled route to an architecture/data-level prediction without
+post-hoc scaling.
+
+## Time-Varying Extension
+
+At later time:
+
+```text
+speed_FA,t = output_speed_t + hidden_alignment_gain_t
+```
+
+where:
+
+```text
+hidden_alignment_gain_t = sum_hidden <g_l^BP(t), g_l^FA(t)>
+```
+
+At initialization:
+
+```text
+E_B[hidden_alignment_gain_0 | W,r] = 0
+```
+
+During training, forward weights become dependent on feedback matrices, so this
+term can become positive. This is actual feedback alignment.
+
+Therefore:
+
+```text
+e_t = 1 - (output_speed_t + hidden_alignment_gain_t) / speed_BP,t
+```
+
+The current tangent-operator estimator works because it measures early movement
+of this alignment gain. A future theorem should bound or approximate:
+
+```text
+hidden_alignment_gain_t
+```
+
+from the early dynamics, possibly using the perturbative `H_FA` analysis style
+from Song/Xu/Lafferty.
+
+## Empirical Validation
+
+Script:
+
+```text
+scripts/actual_fa_initial_operator_moments.py
+```
+
+Run:
+
+```text
+python scripts/actual_fa_initial_operator_moments.py \
+  --widths 16 24 32 48 64 96 \
+  --init-seeds 6 \
+  --feedback-samples 512 \
+  --torch-threads 8 \
+  --outdir outputs/actual_fa_initial_operator_moments
+```
+
+Outputs:
+
+```text
+outputs/actual_fa_initial_operator_moments/initial_operator_moment_rows.csv
+outputs/actual_fa_initial_operator_moments/predicted_vs_empirical_initial_erosion_mean.png
+outputs/actual_fa_initial_operator_moments/initial_erosion_mean_error_by_width.png
+outputs/actual_fa_initial_operator_moments/initial_erosion_histograms.png
+```
+
+Representative results:
+
+| width | init | theory mean erosion | empirical mean erosion | error |
+|---:|---:|---:|---:|---:|
+| 16 | 0 | 0.451931 | 0.444348 | -0.00758 |
+| 24 | 0 | 0.312016 | 0.312286 | 0.00027 |
+| 32 | 0 | 0.236058 | 0.239714 | 0.00366 |
+| 48 | 0 | 0.261471 | 0.259858 | -0.00161 |
+| 64 | 0 | 0.209125 | 0.211261 | 0.00214 |
+| 96 | 0 | 0.259765 | 0.259147 | -0.00062 |
+
+Across all tested widths and initializations, the no-fit mean prediction is
+accurate to Monte Carlo error.
+
+## Consequence for Contributions
+
+This improves contribution 2:
+
+```text
+hard-k random-subspace erosion = clean null model
+actual FA initial erosion = hidden BP speed share theorem
+```
+
+Contribution 4 still handles finite-time gap:
+
+```text
+initial theorem gives e_0
+operator estimator tracks e_t and converts it into finite-time gap
+```
+
+This is a stronger and cleaner story than trying to force raw capacity margin
+to predict the full trajectory gap.