diff options
Diffstat (limited to 'notes/25_soft_erosion_theory_distribution_validation.md')
| -rw-r--r-- | notes/25_soft_erosion_theory_distribution_validation.md | 214 |
1 files changed, 214 insertions, 0 deletions
diff --git a/notes/25_soft_erosion_theory_distribution_validation.md b/notes/25_soft_erosion_theory_distribution_validation.md new file mode 100644 index 0000000..1b078d0 --- /dev/null +++ b/notes/25_soft_erosion_theory_distribution_validation.md @@ -0,0 +1,214 @@ +# Soft-Erosion Theory Distribution Validation Attempt + +We tested whether the revised soft-erosion theory can predict the empirical +FA/BP train-gap distribution from the dense `T=30000` sweep. + +## Data + +Empirical run: + +```text +outputs/phase_transition_dense_T30000_352traj +``` + +Validation output: + +```text +outputs/soft_erosion_theory_vs_empirical +``` + +Main plots: + +```text +outputs/soft_erosion_theory_vs_empirical/gap_distribution_theory_vs_empirical.png +outputs/soft_erosion_theory_vs_empirical/local_erosion_distribution_theory_vs_empirical.png +outputs/soft_erosion_theory_vs_empirical/local_operator_scalar_gap_theory_vs_empirical.png +``` + +Script: + +```text +scripts/soft_erosion_theory_vs_empirical.py +``` + +## Theory 1: Hard-k Random-Subspace Gap Distribution + +This test uses the null model: + +```text +P = parameter count +k = hard FA constraint count +rho ~ Beta((P-k)/2, k/2) +K_FA ≈ rho K_BP +``` + +Then it predicts finite-time loss from the BP initial tangent spectrum: + +```text +r_T_BP = exp(-eta T K_BP / N) r_0 +r_T_FA = exp(-eta T rho K_BP / N) r_0 +gap_T = (||r_T_FA||^2 - ||r_T_BP||^2) / (2N) +``` + +This is not fitted. The only random quantity is the random-subspace theory +distribution for `rho`. + +Result: + +```text +gap_distribution_theory_vs_empirical.png +``` + +The hard-k random-subspace theory strongly overpredicts the empirical gap, +especially on the positive-margin side. + +Examples: + +| width | margin | hard-k theory mean | empirical mean | +|---:|---:|---:|---:| +| 40 | 130 | 0.040374 | 0.000162 | +| 36 | 66 | 0.050133 | 0.000791 | +| 32 | 2 | 0.064534 | 0.003729 | +| 28 | -62 | 0.103630 | 0.014195 | +| 24 | -126 | 0.121998 | 0.046464 | +| 20 | -190 | 0.151140 | 0.146803 | + +So the hard constraint count `k` is not the effective burden. It is too +pessimistic. + +## Theory 2: Local One-Step Erosion + +We measured the actual initial local operator erosion for each feedback seed: + +```text +speed_ratio = <g_BP, g_FA> / <g_BP, g_BP> +erosion = 1 - speed_ratio +``` + +This directly estimates: + +```text +1 - (r^T S_FA r) / (r^T K_BP r) +``` + +where: + +```text +S_FA = (K_FA + K_FA^T) / 2 +``` + +The comparison: + +```text +local_erosion_distribution_theory_vs_empirical.png +``` + +shows that real FA local erosion is much smaller than the hard-k Beta null. + +Examples: + +| width | empirical local erosion mean | hard-k Beta erosion mean | +|---:|---:|---:| +| 24 | 0.3907 | about 0.634 | +| 32 | 0.2338 | about 0.691 | +| 40 | 0.3381 | about 0.733 | + +Again, hard `k` overcounts effective burden. + +## Theory 3: Local-Operator Scalar Gap Prediction + +We then used the measured initial speed ratio as a no-fit scalar operator +prediction: + +```text +K_FA ≈ speed_ratio(B) K_BP +``` + +and propagated the BP tangent spectrum for `T=30000`. + +Result: + +```text +local_operator_scalar_gap_theory_vs_empirical.png +``` + +This improves over hard-k random subspace but is still not sufficient. + +Examples: + +| width | margin | local-operator theory mean | empirical mean | +|---:|---:|---:|---:| +| 40 | 130 | 0.002701 | 0.000162 | +| 36 | 66 | 0.004612 | 0.000791 | +| 32 | 2 | 0.005276 | 0.003729 | +| 28 | -62 | 0.026043 | 0.014195 | +| 24 | -126 | 0.047863 | 0.046464 | +| 22 | -158 | 0.037249 | 0.100526 | +| 20 | -190 | 0.071783 | 0.146803 | + +It matches around `width=24` and is roughly right near `width=32`, but: + +1. it overpredicts the positive-margin tail; +2. it underpredicts the more negative-margin regime; +3. it is not monotone enough because a single initial scalar cannot capture + training-time alignment and feature drift. + +## Conclusion + +The current result is not a successful validation of the hard-k random-subspace +theory. + +It gives a sharper diagnosis: + +```text +hard FA constraint count k is not k_eff. +``` + +The empirical gap is controlled by an effective operator erosion that evolves +during training: + +```text +K_FA,t = J_t J_tilde_t^T +``` + +not by a fixed random deletion of a `k`-dimensional subspace. + +## Revised Theory Target + +The next theory should predict a distribution over: + +```text +e_t(B) = 1 - (r_t^T S_FA,t r_t) / (r_t^T K_BP,t r_t) +``` + +or its cumulative finite-time form: + +```text +r_T_FA = product_t (I - eta K_FA,t / N) r_0 +r_T_BP = product_t (I - eta K_BP,t / N) r_0 +``` + +The right effective capacity is therefore: + +```text +k_eff(B,t) = P * e_t(B) +``` + +or a spectral version of this quantity, not the raw hard matrix constraint +count. + +## Relation to Song, Xu, and Lafferty + +`Convergence and Alignment of Gradient Descent with Random Backpropagation +Weights` is highly relevant and must be read carefully next. + +Current hypothesis: + +```text +Their overparameterized convergence theorem is compatible with our findings. +It shows FA can converge; it does not imply BP-equivalent finite-time tangent +capacity. +``` + +We need to check whether their alignment dynamics imply a decay law for +`e_t(B)` or `k_eff(B,t)`. |