Compare soft erosion theory distributions

1 files changed, 214 insertions, 0 deletions

diff --git a/notes/25_soft_erosion_theory_distribution_validation.md b/notes/25_soft_erosion_theory_distribution_validation.md
new file mode 100644
index 0000000..1b078d0
--- /dev/null
+++ b/notes/25_soft_erosion_theory_distribution_validation.md
@@ -0,0 +1,214 @@
+# Soft-Erosion Theory Distribution Validation Attempt
+
+We tested whether the revised soft-erosion theory can predict the empirical
+FA/BP train-gap distribution from the dense `T=30000` sweep.
+
+## Data
+
+Empirical run:
+
+```text
+outputs/phase_transition_dense_T30000_352traj
+```
+
+Validation output:
+
+```text
+outputs/soft_erosion_theory_vs_empirical
+```
+
+Main plots:
+
+```text
+outputs/soft_erosion_theory_vs_empirical/gap_distribution_theory_vs_empirical.png
+outputs/soft_erosion_theory_vs_empirical/local_erosion_distribution_theory_vs_empirical.png
+outputs/soft_erosion_theory_vs_empirical/local_operator_scalar_gap_theory_vs_empirical.png
+```
+
+Script:
+
+```text
+scripts/soft_erosion_theory_vs_empirical.py
+```
+
+## Theory 1: Hard-k Random-Subspace Gap Distribution
+
+This test uses the null model:
+
+```text
+P = parameter count
+k = hard FA constraint count
+rho ~ Beta((P-k)/2, k/2)
+K_FA ≈ rho K_BP
+```
+
+Then it predicts finite-time loss from the BP initial tangent spectrum:
+
+```text
+r_T_BP = exp(-eta T K_BP / N) r_0
+r_T_FA = exp(-eta T rho K_BP / N) r_0
+gap_T = (||r_T_FA||^2 - ||r_T_BP||^2) / (2N)
+```
+
+This is not fitted. The only random quantity is the random-subspace theory
+distribution for `rho`.
+
+Result:
+
+```text
+gap_distribution_theory_vs_empirical.png
+```
+
+The hard-k random-subspace theory strongly overpredicts the empirical gap,
+especially on the positive-margin side.
+
+Examples:
+
+| width | margin | hard-k theory mean | empirical mean |
+|---:|---:|---:|---:|
+| 40 | 130 | 0.040374 | 0.000162 |
+| 36 | 66 | 0.050133 | 0.000791 |
+| 32 | 2 | 0.064534 | 0.003729 |
+| 28 | -62 | 0.103630 | 0.014195 |
+| 24 | -126 | 0.121998 | 0.046464 |
+| 20 | -190 | 0.151140 | 0.146803 |
+
+So the hard constraint count `k` is not the effective burden. It is too
+pessimistic.
+
+## Theory 2: Local One-Step Erosion
+
+We measured the actual initial local operator erosion for each feedback seed:
+
+```text
+speed_ratio = <g_BP, g_FA> / <g_BP, g_BP>
+erosion = 1 - speed_ratio
+```
+
+This directly estimates:
+
+```text
+1 - (r^T S_FA r) / (r^T K_BP r)
+```
+
+where:
+
+```text
+S_FA = (K_FA + K_FA^T) / 2
+```
+
+The comparison:
+
+```text
+local_erosion_distribution_theory_vs_empirical.png
+```
+
+shows that real FA local erosion is much smaller than the hard-k Beta null.
+
+Examples:
+
+| width | empirical local erosion mean | hard-k Beta erosion mean |
+|---:|---:|---:|
+| 24 | 0.3907 | about 0.634 |
+| 32 | 0.2338 | about 0.691 |
+| 40 | 0.3381 | about 0.733 |
+
+Again, hard `k` overcounts effective burden.
+
+## Theory 3: Local-Operator Scalar Gap Prediction
+
+We then used the measured initial speed ratio as a no-fit scalar operator
+prediction:
+
+```text
+K_FA ≈ speed_ratio(B) K_BP
+```
+
+and propagated the BP tangent spectrum for `T=30000`.
+
+Result:
+
+```text
+local_operator_scalar_gap_theory_vs_empirical.png
+```
+
+This improves over hard-k random subspace but is still not sufficient.
+
+Examples:
+
+| width | margin | local-operator theory mean | empirical mean |
+|---:|---:|---:|---:|
+| 40 | 130 | 0.002701 | 0.000162 |
+| 36 | 66 | 0.004612 | 0.000791 |
+| 32 | 2 | 0.005276 | 0.003729 |
+| 28 | -62 | 0.026043 | 0.014195 |
+| 24 | -126 | 0.047863 | 0.046464 |
+| 22 | -158 | 0.037249 | 0.100526 |
+| 20 | -190 | 0.071783 | 0.146803 |
+
+It matches around `width=24` and is roughly right near `width=32`, but:
+
+1. it overpredicts the positive-margin tail;
+2. it underpredicts the more negative-margin regime;
+3. it is not monotone enough because a single initial scalar cannot capture
+   training-time alignment and feature drift.
+
+## Conclusion
+
+The current result is not a successful validation of the hard-k random-subspace
+theory.
+
+It gives a sharper diagnosis:
+
+```text
+hard FA constraint count k is not k_eff.
+```
+
+The empirical gap is controlled by an effective operator erosion that evolves
+during training:
+
+```text
+K_FA,t = J_t J_tilde_t^T
+```
+
+not by a fixed random deletion of a `k`-dimensional subspace.
+
+## Revised Theory Target
+
+The next theory should predict a distribution over:
+
+```text
+e_t(B) = 1 - (r_t^T S_FA,t r_t) / (r_t^T K_BP,t r_t)
+```
+
+or its cumulative finite-time form:
+
+```text
+r_T_FA = product_t (I - eta K_FA,t / N) r_0
+r_T_BP = product_t (I - eta K_BP,t / N) r_0
+```
+
+The right effective capacity is therefore:
+
+```text
+k_eff(B,t) = P * e_t(B)
+```
+
+or a spectral version of this quantity, not the raw hard matrix constraint
+count.
+
+## Relation to Song, Xu, and Lafferty
+
+`Convergence and Alignment of Gradient Descent with Random Backpropagation
+Weights` is highly relevant and must be read carefully next.
+
+Current hypothesis:
+
+```text
+Their overparameterized convergence theorem is compatible with our findings.
+It shows FA can converge; it does not imply BP-equivalent finite-time tangent
+capacity.
+```
+
+We need to check whether their alignment dynamics imply a decay law for
+`e_t(B)` or `k_eff(B,t)`.