# Contribution Roadmap

Working title:

`Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons`

The current paper should use five contributions. The fifth contribution is not
extra decoration: it separates structural capacity theory from finite-time
loss-gap prediction.

## 1. Capacity Formalization

We define the capacity burden induced by feedback alignment.

Layerwise matrix alignment:

```text
Q_l = cos^2(W_{l+1}^T, B_l)
```

For isotropic random feedback in dimension `D_l`:

```text
Q_l ~ Beta(1/2, (D_l - 1)/2)
```

Alignment threshold `q_l` gives a log-volume capacity cost:

```text
C_l(q_l) = -log P(Q_l >= q_l)
```

This formalizes what it means for feedback alignment to consume parameter
direction volume.

## 2. Scaling Law and Soft Capacity Erosion

Across independent layers, raw feasible volume multiplies while log-capacity
cost adds:

```text
p_all = product_l p_l
C_all = sum_l C_l
```

For an equal-width MLP with width `n`, depth `L`, and fixed alignment threshold:

```text
C_all = Θ(L n^2)
p_all = exp[-Θ(L n^2)]
```

Functional loss should be modeled as soft erosion, not a hard step. If total
parameter dimension is `P`, task-sensitive dimension is `d`, and alignment
imposes a generic `k`-dimensional constraint subspace, then its expected overlap
with the task subspace is:

```text
E[tr(P_E P_S)] = k d / P
```

Equivalently, if `J` is the training-set output Jacobian and a random
`k`-dimensional constraint leaves projection `Q`, then:

```text
K_k = J Q J^T
E[K_k | J] = (1 - k/P) K_BP
```

So the FA/BP gap need not wait until redundant directions are fully exhausted.
Redundancy suppresses the expected task-relevant alignment burden, but does not
make it exactly zero.

The scaling distinction is:

```text
k fixed, P grows        -> k/P -> 0, erosion vanishes
k proportional to P     -> k/P stays constant, relative erosion persists
```

For equal-width MLPs, both parameter count and feedback matrix dimensions scale
like `Θ(L n^2)`, so a fixed per-matrix alignment burden can remain a constant
fraction of capacity under width scaling.

Hard functional rank loss remains a limiting sanity check:

```text
Δd_hard = max(0, k - (P - d))
```

This threshold describes exact rank collapse, not finite-time loss,
conditioning loss, or effective-dimension loss.

This is the soft capacity-erosion contribution.

There is now a cleaner actual-FA initialization theorem that should be used
before the random-subspace null model. For fixed forward weights and residuals,
with independent zero-mean feedback matrices:

```text
E_B[speed_FA | W, r] = ||g_output^BP||^2
```

where:

```text
speed_FA = sum_l <g_l^BP, g_l^FA>
speed_BP = sum_l ||g_l^BP||^2
```

Therefore:

```text
E_B[e_0 | W, r]
  = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2
```

So the true initial FA burden is the hidden-layer BP speed share, not the raw
hard feedback-matrix constraint count. The hard-k random-subspace model remains
a useful null model and capacity sanity check, but it should not be presented
as the exact FA gap theorem.

## 3. Prior-Free Minimax Initialization Bound

Let `b` be the normalized feedback direction in `D` dimensions and `a` be the
unknown normalized target backward direction.

For any feedback initialization distribution `μ`, define:

```text
M_μ = E[b b^T]
trace(M_μ) = 1
```

Then:

```text
inf_a E_μ[(a^T b)^2] = λ_min(M_μ) <= 1/D
```

Therefore:

```text
sup_μ inf_a E_μ[(a^T b)^2] = 1/D
```

Isotropic random feedback reaches the bound. Without a prior over `W` or task
directions, no initialization can beat isotropic random feedback in worst-case
expected squared alignment.

## 4. Tangent-Operator Loss/Gap Estimator

This is the bridge from capacity regime to finite-time loss gap.

For BP squared loss, standard tangent-kernel residual dynamics gives:

```text
r_{t+1}^{BP} ≈ (I - η K_t^{BP}/N) r_t
K_t^{BP} = J_t J_t^T
```

For FA, the parameter update uses a surrogate backward Jacobian `J_tilde_t`,
but output change is still measured by the true forward Jacobian `J_t`:

```text
r_{t+1}^{FA} ≈ (I - η K_t^{FA}/N) r_t
K_t^{FA} = J_t J_tilde_t^T
```

`K_t^{FA}` is better called a tangent operator, not a PSD kernel.

The local estimator freezes `K_0`. The finite-time estimator uses an early
operator velocity:

```text
K_hat_t = K_0 + t (K_s - K_0) / s
```

Then:

```text
L_hat_T = ||r_hat_T||^2 / (2N)
gap_hat_T = L_hat_T^FA - L_hat_T^BP
```

This contribution is conditional on early operator observations `(K_0, K_s)`.
It is not an architecture-only theorem.

## 5. Distributional Empirical Validation

Experiments should validate three levels:

1. static alignment distributions:
   ```text
   Q_l ~ Beta(1/2, (D_l - 1)/2)
   ```
2. capacity and soft erosion:
   FA/BP gap distribution changes smoothly with hard FA capacity margin;
3. actual-FA initial operator moments:
   ```text
   E_B[e_0 | W,r] = hidden BP speed share
   ```
4. tangent-operator trajectory distributions:
   predicted gap distributions overlap empirical trajectory distributions.

Current strong evidence:

- initial actual-FA operator moment validation over widths and feedback seeds;
- 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`;
- stress grid over depth, width, and horizon;
- systematic low-bias analysis showing residual error is kernel-path curvature,
  not normalization.

## Important Separation

Do not claim:

```text
capacity alone predicts exact loss gap
```

Correct claim:

```text
capacity bounds identify the structural regime;
tangent-operator dynamics quantify the finite-time gap within that regime.
```

This separation avoids overclaiming while still giving a coherent theory chain.

## Next Priority

The weakest current theory gap is not the static capacity law anymore. It is
the time-dependent operator dynamics:

```text
e_0 is solved by the actual-FA initial moment theorem
e_t for t > 0 still needs alignment-gain dynamics
```

The literature review in `notes/30_literature_methods_for_fa_operator_dynamics.md`
suggests the most principled route is an FA tangent hierarchy:

```text
K_FA,t = J_t J_tilde_t^T
dK_FA,t/dt = dJ_t/dt J_tilde_t^T + J_t dJ_tilde_t^T/dt
```

with a short-time approximation:

```text
K_FA,t ≈ K_FA,0 + t dot_K_FA,0
```

This formalizes the successful time-varying operator estimator. A lower-risk
full-time theorem can be developed in deep linear FA or in a two-layer
teacher-student order-parameter setting.

The weakest current visual was the old phase-transition contribution; the dense
long-training sweep now shows that the correct phenomenon is a soft ramp, not a
hard kink.

The desired figure should show:

```text
capacity margin decreases -> task-relevant alignment overlap increases ->
FA/BP gap distribution grows smoothly
```

The plot should use hard FA capacity margin on the x-axis, place the zero
margin as a vertical reference, and show the FA-BP train-gap distribution on a
log scale.
The most direct target is a width sweep on a random-label task, because random
labels make the task dimension controllable and force memorization capacity.