From 29b92fb3146f57f4d0ff7cb76f073b387a0d4879 Mon Sep 17 00:00:00 2001
From: YurenHao0426 <Blackhao0426@gmail.com>
Date: Fri, 5 Jun 2026 14:52:14 -0500
Subject: Revise capacity theory to soft erosion

---
 notes/23_soft_capacity_theory_revision.md | 191 ++++++++++++++++++++++++++++++
 1 file changed, 191 insertions(+)
 create mode 100644 notes/23_soft_capacity_theory_revision.md

(limited to 'notes/23_soft_capacity_theory_revision.md')

diff --git a/notes/23_soft_capacity_theory_revision.md b/notes/23_soft_capacity_theory_revision.md
new file mode 100644
index 0000000..bc5ff3c
--- /dev/null
+++ b/notes/23_soft_capacity_theory_revision.md
@@ -0,0 +1,191 @@
+# Soft Capacity Erosion Revision
+
+The dense long-training sweep rules out the clean hard-exhaustion story as the
+main theory.
+
+The previous hard-rank claim was:
+
+```text
+Δd_hard = max(0, k - (P - d))
+```
+
+This is still a useful sanity check for exact rank collapse, but it is too
+coarse for FA/BP train-gap prediction. It predicts no functional loss before
+redundant directions are exhausted. The dense sweep shows the gap changes
+smoothly with capacity margin instead.
+
+## Correct Interpretation
+
+Capacity is eroded continuously.
+
+Alignment constraints do not wait until all redundant directions are exhausted.
+They project into task-relevant directions with nonzero expected overlap from
+the beginning. Overparameterization reduces the overlap, but does not make it
+zero.
+
+So the right statement is not:
+
+```text
+enough redundant capacity -> no FA/BP bias
+```
+
+The right statement is:
+
+```text
+more redundant capacity -> smaller expected FA/BP bias
+```
+
+## Random-Subspace Soft Erosion
+
+Let:
+
+```text
+P = parameter dimension
+d = task-sensitive dimension
+k = alignment constraint dimension
+S = task-sensitive parameter subspace
+E = alignment constraint subspace
+P_S = projection onto S
+P_E = projection onto E
+```
+
+The soft overlap is:
+
+```text
+T_k = tr(P_E P_S)
+```
+
+For a random `k`-dimensional alignment subspace:
+
+```text
+E[T_k] = k d / P
+```
+
+This is nonzero for every `k > 0`. Therefore, at the effective-capacity level,
+there is no exact zero-loss region before hard exhaustion. There is only a
+small-overlap region when `P` is large.
+
+The hard-rank loss:
+
+```text
+max(0, k - (P - d))
+```
+
+is a threshold for exact rank collapse, not a threshold for finite-time loss,
+conditioning loss, or effective-dimension loss.
+
+## Tangent-Kernel Version
+
+Let `J` be the training-set output Jacobian. BP uses:
+
+```text
+K_BP = J J^T
+```
+
+If alignment removes a random `k`-dimensional parameter subspace and leaves
+projection `Q`, then the restricted tangent kernel is:
+
+```text
+K_k = J Q J^T
+```
+
+For a random retained subspace of dimension `P-k`:
+
+```text
+E[Q] = (1 - k/P) I
+```
+
+Therefore:
+
+```text
+E[K_k | J] = (1 - k/P) K_BP
+```
+
+This is the cleanest correction. Before any hard rank collapse, the expected
+tangent operator is already weakened by a factor proportional to `k/P`.
+
+For squared-loss gradient descent with residual `r_t`, each BP eigenmode with
+eigenvalue `λ_i` contracts roughly as:
+
+```text
+BP:        (1 - η λ_i / N)^T
+soft FA:  (1 - η (1-k/P) λ_i / N)^T
+```
+
+So finite-time gap is positive for any `k > 0`, although it shrinks as `k/P`
+shrinks.
+
+## Penalized Alignment Version
+
+If alignment is a soft penalty rather than a hard projection:
+
+```text
+H_align = C^T C
+K_λ = J (M + λ H_align)^(-1) J^T
+```
+
+For small `λ`:
+
+```text
+K_λ = K_0 - λ J M^(-1) H_align M^(-1) J^T + higher-order terms
+```
+
+Again, there is no hard threshold. Any nonzero alignment pressure reduces
+effective tangent capacity in task-relevant directions unless it is exactly
+orthogonal to the task subspace.
+
+## Revised Contribution 2
+
+Old wording:
+
+```text
+FA/BP gap opens when redundant directions are exhausted.
+```
+
+New wording:
+
+```text
+Feedback alignment induces a soft capacity erosion. The expected task-relevant
+overlap of alignment constraints scales as k d / P, so redundancy suppresses but
+does not eliminate the FA/BP gap. Hard rank exhaustion remains a limiting
+boundary for exact rank collapse, while empirical train gaps follow a smooth
+capacity-controlled ramp.
+```
+
+## What The Dense Experiment Supports
+
+Dense `T=30000` sweep:
+
+```text
+margin -190 -> gap 0.1468
+margin -158 -> gap 0.1005
+margin -126 -> gap 0.0465
+margin  -94 -> gap 0.0297
+margin  -62 -> gap 0.0142
+margin  -30 -> gap 0.0075
+margin    2 -> gap 0.0037
+margin   34 -> gap 0.0021
+margin   66 -> gap 0.0008
+margin   98 -> gap 0.0006
+margin  130 -> gap 0.0002
+```
+
+This is exactly the qualitative shape expected from soft erosion: monotone,
+smooth, and close to log-linear over the tested range.
+
+## Paper Consequence
+
+Contribution 2 should be renamed from:
+
+```text
+Scaling Law and Redundancy Exhaustion
+```
+
+to something like:
+
+```text
+Scaling Law and Soft Capacity Erosion
+```
+
+Hard exhaustion should appear as a corollary or limiting case, not the main
+claim.
-- 
cgit v1.2.3