Revise capacity theory to soft erosion

3 files changed, 233 insertions, 12 deletions

diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md
index 179bd5d..4805a19 100644
--- a/notes/18_contribution_roadmap.md
+++ b/notes/18_contribution_roadmap.md
@@ -33,7 +33,7 @@ C_l(q_l) = -log P(Q_l >= q_l)
 This formalizes what it means for feedback alignment to consume parameter
 direction volume.
 
-## 2. Scaling Law and Redundancy Exhaustion
+## 2. Scaling Law and Soft Capacity Erosion
 
 Across independent layers, raw feasible volume multiplies while log-capacity
 cost adds:
@@ -50,17 +50,37 @@ C_all = Θ(L n^2)
 p_all = exp[-Θ(L n^2)]
 ```
 
-Functional loss does not need to appear immediately. If total parameter
-dimension is `P`, task dimension is `d`, and alignment imposes `k` generic
-constraints, then hard functional rank loss is:
+Functional loss should be modeled as soft erosion, not a hard step. If total
+parameter dimension is `P`, task-sensitive dimension is `d`, and alignment
+imposes a generic `k`-dimensional constraint subspace, then its expected overlap
+with the task subspace is:
+
+```text
+E[tr(P_E P_S)] = k d / P
+```
+
+Equivalently, if `J` is the training-set output Jacobian and a random
+`k`-dimensional constraint leaves projection `Q`, then:
+
+```text
+K_k = J Q J^T
+E[K_k | J] = (1 - k/P) K_BP
+```
+
+So the FA/BP gap need not wait until redundant directions are fully exhausted.
+Redundancy suppresses the expected task-relevant alignment burden, but does not
+make it exactly zero.
+
+Hard functional rank loss remains a limiting sanity check:
 
 ```text
 Δd_hard = max(0, k - (P - d))
 ```
 
-So the FA/BP gap should open when redundant directions are exhausted.
+This threshold describes exact rank collapse, not finite-time loss,
+conditioning loss, or effective-dimension loss.
 
-This is the phase-transition contribution.
+This is the soft capacity-erosion contribution.
 
 ## 3. Prior-Free Minimax Initialization Bound
 
@@ -136,8 +156,8 @@ Experiments should validate three levels:
    ```text
    Q_l ~ Beta(1/2, (D_l - 1)/2)
    ```
-2. capacity and redundancy transition:
-   FA/BP gap opens near the hard FA capacity margin crossing;
+2. capacity and soft erosion:
+   FA/BP gap distribution changes smoothly with hard FA capacity margin;
 3. tangent-operator trajectory distributions:
    predicted gap distributions overlap empirical trajectory distributions.
 
@@ -167,16 +187,19 @@ This separation avoids overclaiming while still giving a coherent theory chain.
 
 ## Next Priority
 
-The weakest current visual is the phase-transition contribution.
+The weakest current visual was the phase-transition contribution; the dense
+long-training sweep now shows that the correct phenomenon is a soft ramp, not a
+hard kink.
 
 The desired figure should show:
 
 ```text
-capacity margin decreases -> redundant directions exhausted -> FA/BP gap opens
-and then grows
+capacity margin decreases -> task-relevant alignment overlap increases ->
+FA/BP gap distribution grows smoothly
 ```
 
 The plot should use hard FA capacity margin on the x-axis, place the zero
-margin as a vertical reference, and show BP/FA train loss or FA-BP train gap.
+margin as a vertical reference, and show the FA-BP train-gap distribution on a
+log scale.
 The most direct target is a width sweep on a random-label task, because random
 labels make the task dimension controllable and force memorization capacity.
diff --git a/notes/19_phase_transition_visualization.md b/notes/19_phase_transition_visualization.md
index a2672c2..e5d0589 100644
--- a/notes/19_phase_transition_visualization.md
+++ b/notes/19_phase_transition_visualization.md
@@ -1,5 +1,12 @@
 # Phase Transition Visualization
 
+Status: superseded by the dense long-training sweep in
+`notes/22_dense_phase_transition_soft_ramp.md` and the theory revision in
+`notes/23_soft_capacity_theory_revision.md`.
+
+This note records an earlier hard-exhaustion framing. The current interpretation
+is soft capacity erosion, not a sharp phase transition.
+
 This note records the first clean phase-transition visualization for the
 capacity-exhaustion contribution.
 
diff --git a/notes/23_soft_capacity_theory_revision.md b/notes/23_soft_capacity_theory_revision.md
new file mode 100644
index 0000000..bc5ff3c
--- /dev/null
+++ b/notes/23_soft_capacity_theory_revision.md
@@ -0,0 +1,191 @@
+# Soft Capacity Erosion Revision
+
+The dense long-training sweep rules out the clean hard-exhaustion story as the
+main theory.
+
+The previous hard-rank claim was:
+
+```text
+Δd_hard = max(0, k - (P - d))
+```
+
+This is still a useful sanity check for exact rank collapse, but it is too
+coarse for FA/BP train-gap prediction. It predicts no functional loss before
+redundant directions are exhausted. The dense sweep shows the gap changes
+smoothly with capacity margin instead.
+
+## Correct Interpretation
+
+Capacity is eroded continuously.
+
+Alignment constraints do not wait until all redundant directions are exhausted.
+They project into task-relevant directions with nonzero expected overlap from
+the beginning. Overparameterization reduces the overlap, but does not make it
+zero.
+
+So the right statement is not:
+
+```text
+enough redundant capacity -> no FA/BP bias
+```
+
+The right statement is:
+
+```text
+more redundant capacity -> smaller expected FA/BP bias
+```
+
+## Random-Subspace Soft Erosion
+
+Let:
+
+```text
+P = parameter dimension
+d = task-sensitive dimension
+k = alignment constraint dimension
+S = task-sensitive parameter subspace
+E = alignment constraint subspace
+P_S = projection onto S
+P_E = projection onto E
+```
+
+The soft overlap is:
+
+```text
+T_k = tr(P_E P_S)
+```
+
+For a random `k`-dimensional alignment subspace:
+
+```text
+E[T_k] = k d / P
+```
+
+This is nonzero for every `k > 0`. Therefore, at the effective-capacity level,
+there is no exact zero-loss region before hard exhaustion. There is only a
+small-overlap region when `P` is large.
+
+The hard-rank loss:
+
+```text
+max(0, k - (P - d))
+```
+
+is a threshold for exact rank collapse, not a threshold for finite-time loss,
+conditioning loss, or effective-dimension loss.
+
+## Tangent-Kernel Version
+
+Let `J` be the training-set output Jacobian. BP uses:
+
+```text
+K_BP = J J^T
+```
+
+If alignment removes a random `k`-dimensional parameter subspace and leaves
+projection `Q`, then the restricted tangent kernel is:
+
+```text
+K_k = J Q J^T
+```
+
+For a random retained subspace of dimension `P-k`:
+
+```text
+E[Q] = (1 - k/P) I
+```
+
+Therefore:
+
+```text
+E[K_k | J] = (1 - k/P) K_BP
+```
+
+This is the cleanest correction. Before any hard rank collapse, the expected
+tangent operator is already weakened by a factor proportional to `k/P`.
+
+For squared-loss gradient descent with residual `r_t`, each BP eigenmode with
+eigenvalue `λ_i` contracts roughly as:
+
+```text
+BP:        (1 - η λ_i / N)^T
+soft FA:  (1 - η (1-k/P) λ_i / N)^T
+```
+
+So finite-time gap is positive for any `k > 0`, although it shrinks as `k/P`
+shrinks.
+
+## Penalized Alignment Version
+
+If alignment is a soft penalty rather than a hard projection:
+
+```text
+H_align = C^T C
+K_λ = J (M + λ H_align)^(-1) J^T
+```
+
+For small `λ`:
+
+```text
+K_λ = K_0 - λ J M^(-1) H_align M^(-1) J^T + higher-order terms
+```
+
+Again, there is no hard threshold. Any nonzero alignment pressure reduces
+effective tangent capacity in task-relevant directions unless it is exactly
+orthogonal to the task subspace.
+
+## Revised Contribution 2
+
+Old wording:
+
+```text
+FA/BP gap opens when redundant directions are exhausted.
+```
+
+New wording:
+
+```text
+Feedback alignment induces a soft capacity erosion. The expected task-relevant
+overlap of alignment constraints scales as k d / P, so redundancy suppresses but
+does not eliminate the FA/BP gap. Hard rank exhaustion remains a limiting
+boundary for exact rank collapse, while empirical train gaps follow a smooth
+capacity-controlled ramp.
+```
+
+## What The Dense Experiment Supports
+
+Dense `T=30000` sweep:
+
+```text
+margin -190 -> gap 0.1468
+margin -158 -> gap 0.1005
+margin -126 -> gap 0.0465
+margin  -94 -> gap 0.0297
+margin  -62 -> gap 0.0142
+margin  -30 -> gap 0.0075
+margin    2 -> gap 0.0037
+margin   34 -> gap 0.0021
+margin   66 -> gap 0.0008
+margin   98 -> gap 0.0006
+margin  130 -> gap 0.0002
+```
+
+This is exactly the qualitative shape expected from soft erosion: monotone,
+smooth, and close to log-linear over the tested range.
+
+## Paper Consequence
+
+Contribution 2 should be renamed from:
+
+```text
+Scaling Law and Redundancy Exhaustion
+```
+
+to something like:
+
+```text
+Scaling Law and Soft Capacity Erosion
+```
+
+Hard exhaustion should appear as a corollary or limiting case, not the main
+claim.