path: root/notes/01_theory_notes.md

# Theory Notes

## Notation

MLP widths:

\[
n_0,n_1,\dots,n_L.
\]

Forward weights:

\[
W_l\in \mathbb R^{n_l\times n_{l-1}}.
\]

For ordinary FA, feedback matrix for layer \(l\):

\[
B_l\in \mathbb R^{n_l\times n_{l+1}},
\]

which replaces:

\[
W_{l+1}^{\top}\in \mathbb R^{n_l\times n_{l+1}}.
\]

Matrix direction dimension:

\[
D_l=n_l n_{l+1}.
\]

## Candidate Theorem 1: Static Alignment Distribution

Assume:

- \(A_l=W_{l+1}^{\top}\).
- \(A_l/\|A_l\|_F\) and \(B_l/\|B_l\|_F\) are independent isotropic directions in \(\mathbb R^{D_l}\).

Define:

\[
Q_l =
\frac{
\langle A_l,B_l\rangle_F^2
}{
\|A_l\|_F^2\|B_l\|_F^2
}.
\]

Then:

\[
Q_l\sim
\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]

Consequences:

\[
\mathbb E[Q_l]=\frac1{D_l}.
\]

\[
D_lQ_l \Rightarrow \chi_1^2
\quad
\text{as } D_l\to\infty.
\]

Tail:

\[
\Pr(Q_l\ge q)
=
1-I_q\left(\frac12,\frac{D_l-1}{2}\right).
\]

High-dimensional approximation to refine:

\[
\Pr(Q_l\ge q)
\lesssim
2\exp\left[-\frac{(D_l-1)q}{2}\right].
\]

## Capacity Definition

For a required alignment threshold \(q\):

\[
C_l(q)
=
-\log \Pr(Q_l\ge q).
\]

Under the beta law:

\[
C_l(q)
=
-\log
\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]

High-dimensional approximation:

\[
C_l(q)
\approx
\frac{D_l-1}{2}\log\frac1{1-q}.
\]

Small-\(q\) approximation:

\[
C_l(q)\approx \frac{D_l q}{2}.
\]

Open detail: decide whether logs are natural logs or base-2 bits. Use natural logs in theorem statements unless the paper explicitly wants bit units.

## Candidate Corollary 1: Multilayer Scaling

If layerwise feedback matrices are independent and act on distinct matrix blocks, then:

\[
p_{\mathrm{all}}
=
\prod_l
\Pr(Q_l\ge q_l),
\]

and:

\[
C_{\mathrm{all}}
=
\sum_l C_l(q_l).
\]

Interpretation:

- Log-capacity cost is additive.
- Raw feasible volume is multiplicative.

Equal-width case:

\[
D_l\approx n^2.
\]

Fixed threshold \(q>0\):

\[
C_{\mathrm{all}}=\Theta(Ln^2).
\]

Chance-level threshold \(q=c/D_l\):

\[
C_l\approx c/2,
\qquad
C_{\mathrm{all}}=\Theta(L).
\]

## Candidate Theorem 2: Prior-Free Minimax Bound

Let:

\[
\hat b=\frac{\operatorname{vec}(B)}{\|B\|_F}\in\mathbb S^{D-1}.
\]

Any feedback initialization distribution \(\mu\) induces:

\[
M_\mu=\mathbb E_\mu[\hat b\hat b^\top],
\qquad
\operatorname{tr}M_\mu=1.
\]

For target direction \(a\in\mathbb S^{D-1}\):

\[
\mathbb E_\mu[(a^\top \hat b)^2]=a^\top M_\mu a.
\]

Since:

\[
\lambda_{\min}(M_\mu)\le \frac{\operatorname{tr}M_\mu}{D}=\frac1D,
\]

we have:

\[
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
\le
\frac1D.
\]

Thus:

\[
\sup_\mu
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]

Isotropic initialization attains this bound.

### Proof Sketch

The proof only needs second moments. For any feedback initialization \(\mu\), define:

\[
M_\mu=\mathbb E_\mu[\hat b\hat b^\top].
\]

Since \(\|\hat b\|=1\):

\[
\operatorname{tr}M_\mu
=
\mathbb E_\mu[\operatorname{tr}(\hat b\hat b^\top)]
=
\mathbb E_\mu[\|\hat b\|^2]
=1.
\]

For any fixed target direction \(a\):

\[
\mathbb E_\mu[(a^\top \hat b)^2]
=
\mathbb E_\mu[a^\top \hat b\hat b^\top a]
=
a^\top M_\mu a.
\]

The worst-case target direction is the minimum-eigenvalue direction of \(M_\mu\):

\[
\inf_{\|a\|=1} a^\top M_\mu a
=
\lambda_{\min}(M_\mu).
\]

Because the average eigenvalue is \(1/D\):

\[
\lambda_{\min}(M_\mu)
\le
\frac1D.
\]

Hence:

\[
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
\le
\frac1D
\]

for every initialization \(\mu\). If \(\mu\) is isotropic, then:

\[
M_\mu=\frac1D I,
\]

so every direction has:

\[
a^\top M_\mu a=\frac1D.
\]

Thus:

\[
\sup_\mu
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]

This is a worst-case theorem. If the target direction is itself uniformly random, then:

\[
\mathbb E_a[a^\top M_\mu a]
=
\frac{\operatorname{tr}M_\mu}{D}
=
\frac1D
\]

for every \(\mu\). Therefore prior-free average-case alignment over uniform targets cannot distinguish initializations by their mean; the distinction is worst-case coverage, tail behavior, and conditioning.

## Prior-Aware Corollary

If target directions have prior covariance:

\[
\Sigma_A=\mathbb E[aa^\top],
\]

then:

\[
\mathbb E_{a,B}[(a^\top \hat b)^2]
=
\operatorname{tr}(\Sigma_A M_\mu).
\]

Therefore the optimal structured feedback distribution depends on the eigenspectrum of \(\Sigma_A\). Without prior information, isotropy is minimax; with prior information, top-eigenspace feedback can be better.

## Functional Capacity

Let total parameter dimension be:

\[
P=\sum_l n_l(n_{l-1}+1).
\]

Let task Jacobian be:

\[
J=\frac{\partial f_\theta(X)}{\partial \theta}.
\]

Local task rank:

\[
d=\operatorname{rank}(J).
\]

If alignment constraints remove a generic \(k\)-dimensional parameter subspace, then hard local function rank after constraints is:

\[
d_{\mathrm{hard}}=\min(d,P-k).
\]

Hard functional loss:

\[
\Delta d_{\mathrm{hard}}
=
d-d_{\mathrm{hard}}
=
\max(0,k-(P-d)).
\]

Soft overlap model. Let \(E\) be the alignment constraint subspace and \(S\) be the task-sensitive subspace:

\[
T_k=\operatorname{tr}(P_E P_S).
\]

For random subspaces:

\[
\mathbb E[T_k]=\frac{kd}{P}.
\]

Variance:

\[
\operatorname{Var}(T_k)
=
\frac{
2kd(P-k)(P-d)
}{
P^2(P-1)(P+2)
}.
\]

Interpretation:

- Hard rank loss has a redundancy-exhaustion threshold.
- Soft conditioning loss can grow approximately linearly from the start.

## Trajectory Bridge

BP gradient:

\[
g_t=\nabla_\theta L(\theta_t).
\]

FA surrogate gradient:

\[
\tilde g_t(B).
\]

Mismatch:

\[
\epsilon_t(B)=\tilde g_t(B)-g_t.
\]

Linearized deviation from the BP trajectory:

\[
\delta\theta_T(B)
\approx
-\eta
\sum_{t<T}
\Phi_{T,t+1}\epsilon_t(B),
\]

where:

\[
\Phi_{T,t+1}
=
\prod_{s=t+1}^{T-1}(I-\eta H_s).
\]

Second-order excess loss:

\[
\Delta L_T(B)
\approx
\frac12
\delta\theta_T(B)^\top H_T\delta\theta_T(B).
\]

If \(\epsilon_t(B)\) is approximately Gaussian under random \(B\), then:

\[
\Delta L_T(B)
\approx
\frac12\sum_i \lambda_i\chi_i^2.
\]

This should be treated as a bridge approximation, not as a primary architecture-only theorem.