path: root/notes/18_contribution_roadmap.md

# Contribution Roadmap

Working title:

`Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons`

The current paper should use five contributions. The fifth contribution is not
extra decoration: it separates structural capacity theory from finite-time
loss-gap prediction.

## 1. Capacity Formalization

We define the capacity burden induced by feedback alignment.

Layerwise matrix alignment:

```text
Q_l = cos^2(W_{l+1}^T, B_l)
```

For isotropic random feedback in dimension `D_l`:

```text
Q_l ~ Beta(1/2, (D_l - 1)/2)
```

Alignment threshold `q_l` gives a log-volume capacity cost:

```text
C_l(q_l) = -log P(Q_l >= q_l)
```

This formalizes what it means for feedback alignment to consume parameter
direction volume.

## 2. Scaling Law and Soft Capacity Erosion

Across independent layers, raw feasible volume multiplies while log-capacity
cost adds:

```text
p_all = product_l p_l
C_all = sum_l C_l
```

For an equal-width MLP with width `n`, depth `L`, and fixed alignment threshold:

```text
C_all = Θ(L n^2)
p_all = exp[-Θ(L n^2)]
```

Functional loss should be modeled as soft erosion, not a hard step. If total
parameter dimension is `P`, task-sensitive dimension is `d`, and alignment
imposes a generic `k`-dimensional constraint subspace, then its expected overlap
with the task subspace is:

```text
E[tr(P_E P_S)] = k d / P
```

Equivalently, if `J` is the training-set output Jacobian and a random
`k`-dimensional constraint leaves projection `Q`, then:

```text
K_k = J Q J^T
E[K_k | J] = (1 - k/P) K_BP
```

So the FA/BP gap need not wait until redundant directions are fully exhausted.
Redundancy suppresses the expected task-relevant alignment burden, but does not
make it exactly zero.

The scaling distinction is:

```text
k fixed, P grows        -> k/P -> 0, erosion vanishes
k proportional to P     -> k/P stays constant, relative erosion persists
```

For equal-width MLPs, both parameter count and feedback matrix dimensions scale
like `Θ(L n^2)`, so a fixed per-matrix alignment burden can remain a constant
fraction of capacity under width scaling.

Hard functional rank loss remains a limiting sanity check:

```text
Δd_hard = max(0, k - (P - d))
```

This threshold describes exact rank collapse, not finite-time loss,
conditioning loss, or effective-dimension loss.

This is the soft capacity-erosion contribution.

## 3. Prior-Free Minimax Initialization Bound

Let `b` be the normalized feedback direction in `D` dimensions and `a` be the
unknown normalized target backward direction.

For any feedback initialization distribution `μ`, define:

```text
M_μ = E[b b^T]
trace(M_μ) = 1
```

Then:

```text
inf_a E_μ[(a^T b)^2] = λ_min(M_μ) <= 1/D
```

Therefore:

```text
sup_μ inf_a E_μ[(a^T b)^2] = 1/D
```

Isotropic random feedback reaches the bound. Without a prior over `W` or task
directions, no initialization can beat isotropic random feedback in worst-case
expected squared alignment.

## 4. Tangent-Operator Loss/Gap Estimator

This is the bridge from capacity regime to finite-time loss gap.

For BP squared loss, standard tangent-kernel residual dynamics gives:

```text
r_{t+1}^{BP} ≈ (I - η K_t^{BP}/N) r_t
K_t^{BP} = J_t J_t^T
```

For FA, the parameter update uses a surrogate backward Jacobian `J_tilde_t`,
but output change is still measured by the true forward Jacobian `J_t`:

```text
r_{t+1}^{FA} ≈ (I - η K_t^{FA}/N) r_t
K_t^{FA} = J_t J_tilde_t^T
```

`K_t^{FA}` is better called a tangent operator, not a PSD kernel.

The local estimator freezes `K_0`. The finite-time estimator uses an early
operator velocity:

```text
K_hat_t = K_0 + t (K_s - K_0) / s
```

Then:

```text
L_hat_T = ||r_hat_T||^2 / (2N)
gap_hat_T = L_hat_T^FA - L_hat_T^BP
```

This contribution is conditional on early operator observations `(K_0, K_s)`.
It is not an architecture-only theorem.

## 5. Distributional Empirical Validation

Experiments should validate three levels:

1. static alignment distributions:
   ```text
   Q_l ~ Beta(1/2, (D_l - 1)/2)
   ```
2. capacity and soft erosion:
   FA/BP gap distribution changes smoothly with hard FA capacity margin;
3. tangent-operator trajectory distributions:
   predicted gap distributions overlap empirical trajectory distributions.

Current strong evidence:

- 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`;
- stress grid over depth, width, and horizon;
- systematic low-bias analysis showing residual error is kernel-path curvature,
  not normalization.

## Important Separation

Do not claim:

```text
capacity alone predicts exact loss gap
```

Correct claim:

```text
capacity bounds identify the structural regime;
tangent-operator dynamics quantify the finite-time gap within that regime.
```

This separation avoids overclaiming while still giving a coherent theory chain.

## Next Priority

The weakest current visual was the phase-transition contribution; the dense
long-training sweep now shows that the correct phenomenon is a soft ramp, not a
hard kink.

The desired figure should show:

```text
capacity margin decreases -> task-relevant alignment overlap increases ->
FA/BP gap distribution grows smoothly
```

The plot should use hard FA capacity margin on the x-axis, place the zero
margin as a vertical reference, and show the FA-BP train-gap distribution on a
log scale.
The most direct target is a width sweep on a random-label task, because random
labels make the task dimension controllable and force memorization capacity.