path: root/notes/23_soft_capacity_theory_revision.md

# Soft Capacity Erosion Revision

The dense long-training sweep rules out the clean hard-exhaustion story as the
main theory.

The previous hard-rank claim was:

```text
Δd_hard = max(0, k - (P - d))
```

This is still a useful sanity check for exact rank collapse, but it is too
coarse for FA/BP train-gap prediction. It predicts no functional loss before
redundant directions are exhausted. The dense sweep shows the gap changes
smoothly with capacity margin instead.

## Correct Interpretation

Capacity is eroded continuously.

Alignment constraints do not wait until all redundant directions are exhausted.
They project into task-relevant directions with nonzero expected overlap from
the beginning. Overparameterization reduces the overlap, but does not make it
zero.

So the right statement is not:

```text
enough redundant capacity -> no FA/BP bias
```

The right statement is:

```text
more redundant capacity -> smaller expected FA/BP bias
```

## Random-Subspace Soft Erosion

Let:

```text
P = parameter dimension
d = task-sensitive dimension
k = alignment constraint dimension
S = task-sensitive parameter subspace
E = alignment constraint subspace
P_S = projection onto S
P_E = projection onto E
```

The soft overlap is:

```text
T_k = tr(P_E P_S)
```

For a random `k`-dimensional alignment subspace:

```text
E[T_k] = k d / P
```

This is nonzero for every `k > 0`. Therefore, at the effective-capacity level,
there is no exact zero-loss region before hard exhaustion. There is only a
small-overlap region when `P` is large.

The hard-rank loss:

```text
max(0, k - (P - d))
```

is a threshold for exact rank collapse, not a threshold for finite-time loss,
conditioning loss, or effective-dimension loss.

## Tangent-Kernel Version

Let `J` be the training-set output Jacobian. BP uses:

```text
K_BP = J J^T
```

If alignment removes a random `k`-dimensional parameter subspace and leaves
projection `Q`, then the restricted tangent kernel is:

```text
K_k = J Q J^T
```

For a random retained subspace of dimension `P-k`:

```text
E[Q] = (1 - k/P) I
```

Therefore:

```text
E[K_k | J] = (1 - k/P) K_BP
```

This is the cleanest correction. Before any hard rank collapse, the expected
tangent operator is already weakened by a factor proportional to `k/P`.

For squared-loss gradient descent with residual `r_t`, each BP eigenmode with
eigenvalue `λ_i` contracts roughly as:

```text
BP:        (1 - η λ_i / N)^T
soft FA:  (1 - η (1-k/P) λ_i / N)^T
```

So finite-time gap is positive for any `k > 0`, although it shrinks as `k/P`
shrinks.

## Penalized Alignment Version

If alignment is a soft penalty rather than a hard projection:

```text
H_align = C^T C
K_λ = J (M + λ H_align)^(-1) J^T
```

For small `λ`:

```text
K_λ = K_0 - λ J M^(-1) H_align M^(-1) J^T + higher-order terms
```

Again, there is no hard threshold. Any nonzero alignment pressure reduces
effective tangent capacity in task-relevant directions unless it is exactly
orthogonal to the task subspace.

## Revised Contribution 2

Old wording:

```text
FA/BP gap opens when redundant directions are exhausted.
```

New wording:

```text
Feedback alignment induces a soft capacity erosion. The expected task-relevant
overlap of alignment constraints scales as k d / P, so redundancy suppresses but
does not eliminate the FA/BP gap. Hard rank exhaustion remains a limiting
boundary for exact rank collapse, while empirical train gaps follow a smooth
capacity-controlled ramp.
```

## What The Dense Experiment Supports

Dense `T=30000` sweep:

```text
margin -190 -> gap 0.1468
margin -158 -> gap 0.1005
margin -126 -> gap 0.0465
margin  -94 -> gap 0.0297
margin  -62 -> gap 0.0142
margin  -30 -> gap 0.0075
margin    2 -> gap 0.0037
margin   34 -> gap 0.0021
margin   66 -> gap 0.0008
margin   98 -> gap 0.0006
margin  130 -> gap 0.0002
```

This is exactly the qualitative shape expected from soft erosion: monotone,
smooth, and close to log-linear over the tested range.

## Paper Consequence

Contribution 2 should be renamed from:

```text
Scaling Law and Redundancy Exhaustion
```

to something like:

```text
Scaling Law and Soft Capacity Erosion
```

Hard exhaustion should appear as a corollary or limiting case, not the main
claim.

## Selling Point

The natural intuition is:

```text
if the model is sufficiently overparameterized, it should absorb the random
feedback-alignment burden and behave like BP.
```

Existing FA/DFA work often supports the softer intuition that the network can
learn to make random feedback useful, and overparameterized two-layer analyses
show convergence in settings where random feedback is used. Our result should
not deny those convergence results. Instead, it sharpens the question:

```text
does overparameterization make the alignment burden exactly free?
```

The soft-erosion answer is no in general.

Overparameterization dilutes the burden only through the ratio `k/P`:

```text
E[K_k | J] = (1 - k/P) K_BP
```

So if `k` is fixed and `P -> infinity`, then:

```text
k/P -> 0
```

and the soft capacity loss vanishes.

But FA does not usually keep `k` fixed while scaling the model. In an equal-width
MLP, parameter dimension scales like:

```text
P = Θ(L n^2)
```

The feedback-alignment matrices also scale with the layer matrix dimensions:

```text
D_l = n_l n_{l+1}
sum_l D_l = Θ(L n^2)
```

If the effective alignment burden is proportional to these feedback matrix
dimensions, then:

```text
k = α P
```

for some effective burden fraction `α`. Then:

```text
k/P = α
E[K_k | J] = (1 - α) K_BP
```

The relative tangent-capacity reduction does not vanish just because width grows.

This gives the clean scaling message:

```text
large models can make the absolute performance strong;
but if the feedback-alignment burden scales with model size, the relative
capacity erosion can remain nonzero.
```

This also explains why empirical gaps can look small in large models:

1. BP and FA losses may both be very low, so the absolute task metric gap is
   visually small.
2. The residual or test metric can be insensitive once both methods are in a
   high-performance regime.
3. The correct theoretical object is the fractional erosion `k/P` or the
   operator reduction `K_FA` versus `K_BP`, not only the final raw loss gap.

Therefore, the main claim should be:

```text
capacity does not make feedback alignment free; it only controls the erosion
rate. The gap vanishes only under scalings where the effective alignment burden
grows sublinearly relative to parameter dimension.
```