# Soft Capacity Erosion Revision

The dense long-training sweep rules out the clean hard-exhaustion story as the
main theory.

The previous hard-rank claim was:

```text
Δd_hard = max(0, k - (P - d))
```

This is still a useful sanity check for exact rank collapse, but it is too
coarse for FA/BP train-gap prediction. It predicts no functional loss before
redundant directions are exhausted. The dense sweep shows the gap changes
smoothly with capacity margin instead.

## Correct Interpretation

Capacity is eroded continuously.

Alignment constraints do not wait until all redundant directions are exhausted.
They project into task-relevant directions with nonzero expected overlap from
the beginning. Overparameterization reduces the overlap, but does not make it
zero.

So the right statement is not:

```text
enough redundant capacity -> no FA/BP bias
```

The right statement is:

```text
more redundant capacity -> smaller expected FA/BP bias
```

## Random-Subspace Soft Erosion

Let:

```text
P = parameter dimension
d = task-sensitive dimension
k = alignment constraint dimension
S = task-sensitive parameter subspace
E = alignment constraint subspace
P_S = projection onto S
P_E = projection onto E
```

The soft overlap is:

```text
T_k = tr(P_E P_S)
```

For a random `k`-dimensional alignment subspace:

```text
E[T_k] = k d / P
```

This is nonzero for every `k > 0`. Therefore, at the effective-capacity level,
there is no exact zero-loss region before hard exhaustion. There is only a
small-overlap region when `P` is large.

The hard-rank loss:

```text
max(0, k - (P - d))
```

is a threshold for exact rank collapse, not a threshold for finite-time loss,
conditioning loss, or effective-dimension loss.

## Tangent-Kernel Version

Let `J` be the training-set output Jacobian. BP uses:

```text
K_BP = J J^T
```

If alignment removes a random `k`-dimensional parameter subspace and leaves
projection `Q`, then the restricted tangent kernel is:

```text
K_k = J Q J^T
```

For a random retained subspace of dimension `P-k`:

```text
E[Q] = (1 - k/P) I
```

Therefore:

```text
E[K_k | J] = (1 - k/P) K_BP
```

This is the cleanest correction. Before any hard rank collapse, the expected
tangent operator is already weakened by a factor proportional to `k/P`.

For squared-loss gradient descent with residual `r_t`, each BP eigenmode with
eigenvalue `λ_i` contracts roughly as:

```text
BP:        (1 - η λ_i / N)^T
soft FA:  (1 - η (1-k/P) λ_i / N)^T
```

So finite-time gap is positive for any `k > 0`, although it shrinks as `k/P`
shrinks.

## Penalized Alignment Version

If alignment is a soft penalty rather than a hard projection:

```text
H_align = C^T C
K_λ = J (M + λ H_align)^(-1) J^T
```

For small `λ`:

```text
K_λ = K_0 - λ J M^(-1) H_align M^(-1) J^T + higher-order terms
```

Again, there is no hard threshold. Any nonzero alignment pressure reduces
effective tangent capacity in task-relevant directions unless it is exactly
orthogonal to the task subspace.

## Revised Contribution 2

Old wording:

```text
FA/BP gap opens when redundant directions are exhausted.
```

New wording:

```text
Feedback alignment induces a soft capacity erosion. The expected task-relevant
overlap of alignment constraints scales as k d / P, so redundancy suppresses but
does not eliminate the FA/BP gap. Hard rank exhaustion remains a limiting
boundary for exact rank collapse, while empirical train gaps follow a smooth
capacity-controlled ramp.
```

## What The Dense Experiment Supports

Dense `T=30000` sweep:

```text
margin -190 -> gap 0.1468
margin -158 -> gap 0.1005
margin -126 -> gap 0.0465
margin  -94 -> gap 0.0297
margin  -62 -> gap 0.0142
margin  -30 -> gap 0.0075
margin    2 -> gap 0.0037
margin   34 -> gap 0.0021
margin   66 -> gap 0.0008
margin   98 -> gap 0.0006
margin  130 -> gap 0.0002
```

This is exactly the qualitative shape expected from soft erosion: monotone,
smooth, and close to log-linear over the tested range.

## Paper Consequence

Contribution 2 should be renamed from:

```text
Scaling Law and Redundancy Exhaustion
```

to something like:

```text
Scaling Law and Soft Capacity Erosion
```

Hard exhaustion should appear as a corollary or limiting case, not the main
claim.