path: root/notes/01_theory_notes.md

# Theory Notes

## Notation

MLP widths:

\[
n_0,n_1,\dots,n_L.
\]

Forward weights:

\[
W_l\in \mathbb R^{n_l\times n_{l-1}}.
\]

For ordinary FA, feedback matrix for layer \(l\):

\[
B_l\in \mathbb R^{n_l\times n_{l+1}},
\]

which replaces:

\[
W_{l+1}^{\top}\in \mathbb R^{n_l\times n_{l+1}}.
\]

Matrix direction dimension:

\[
D_l=n_l n_{l+1}.
\]

## Candidate Theorem 1: Static Alignment Distribution

Assume:

- \(A_l=W_{l+1}^{\top}\).
- \(A_l/\|A_l\|_F\) and \(B_l/\|B_l\|_F\) are independent isotropic directions in \(\mathbb R^{D_l}\).

Define:

\[
Q_l =
\frac{
\langle A_l,B_l\rangle_F^2
}{
\|A_l\|_F^2\|B_l\|_F^2
}.
\]

Then:

\[
Q_l\sim
\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]

Consequences:

\[
\mathbb E[Q_l]=\frac1{D_l}.
\]

\[
D_lQ_l \Rightarrow \chi_1^2
\quad
\text{as } D_l\to\infty.
\]

Tail:

\[
\Pr(Q_l\ge q)
=
1-I_q\left(\frac12,\frac{D_l-1}{2}\right).
\]

High-dimensional approximation to refine:

\[
\Pr(Q_l\ge q)
\lesssim
2\exp\left[-\frac{(D_l-1)q}{2}\right].
\]

## Capacity Definition

For a required alignment threshold \(q\):

\[
C_l(q)
=
-\log \Pr(Q_l\ge q).
\]

Under the beta law:

\[
C_l(q)
=
-\log
\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]

High-dimensional approximation:

\[
C_l(q)
\approx
\frac{D_l-1}{2}\log\frac1{1-q}.
\]

Small-\(q\) approximation:

\[
C_l(q)\approx \frac{D_l q}{2}.
\]

Open detail: decide whether logs are natural logs or base-2 bits. Use natural logs in theorem statements unless the paper explicitly wants bit units.

## Candidate Corollary 1: Multilayer Scaling

If layerwise feedback matrices are independent and act on distinct matrix blocks, then:

\[
p_{\mathrm{all}}
=
\prod_l
\Pr(Q_l\ge q_l),
\]

and:

\[
C_{\mathrm{all}}
=
\sum_l C_l(q_l).
\]

Interpretation:

- Log-capacity cost is additive.
- Raw feasible volume is multiplicative.

Equal-width case:

\[
D_l\approx n^2.
\]

Fixed threshold \(q>0\):

\[
C_{\mathrm{all}}=\Theta(Ln^2).
\]

Chance-level threshold \(q=c/D_l\):

\[
C_l\approx c/2,
\qquad
C_{\mathrm{all}}=\Theta(L).
\]

## Candidate Corollary 2: Observed Capacity Distribution

The threshold cost \(C_l(q)\) is deterministic once \(q\) is chosen. For empirical distribution matching, define the observed alignment surprisal:

\[
S_l
=
-\log \Pr(Q_l'\ge Q_l),
\]

where \(Q_l'\) is an independent draw from the same beta law as \(Q_l\). If:

\[
Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right),
\]

and the survival function is continuous, then by the probability integral transform:

\[
U_l=\Pr(Q_l'\ge Q_l)\sim \mathrm{Uniform}(0,1),
\]

so:

\[
S_l=-\log U_l\sim \mathrm{Exp}(1).
\]

For \(L\) independent feedback-aligned layers, the observed total capacity surprisal is:

\[
S_{1:L}=\sum_{l=1}^L S_l.
\]

If the layerwise feedback matrices are independent, then:

\[
S_{1:L}\sim \mathrm{Gamma}(L,1).
\]

This is useful because it predicts a full distribution rather than only a mean or bound. It also shows that after the beta-tail transform, the null distribution of observed capacity surprisal is dimension-free; dimension enters through the raw \(Q_l\) scale and the threshold cost \(C_l(q)\), not through the transformed surprisal law.

## Candidate Theorem 2: Prior-Free Minimax Bound

Let:

\[
\hat b=\frac{\operatorname{vec}(B)}{\|B\|_F}\in\mathbb S^{D-1}.
\]

Any feedback initialization distribution \(\mu\) induces:

\[
M_\mu=\mathbb E_\mu[\hat b\hat b^\top],
\qquad
\operatorname{tr}M_\mu=1.
\]

For target direction \(a\in\mathbb S^{D-1}\):

\[
\mathbb E_\mu[(a^\top \hat b)^2]=a^\top M_\mu a.
\]

Since:

\[
\lambda_{\min}(M_\mu)\le \frac{\operatorname{tr}M_\mu}{D}=\frac1D,
\]

we have:

\[
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
\le
\frac1D.
\]

Thus:

\[
\sup_\mu
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]

Isotropic initialization attains this bound.

### Proof Sketch

The proof only needs second moments. For any feedback initialization \(\mu\), define:

\[
M_\mu=\mathbb E_\mu[\hat b\hat b^\top].
\]

Since \(\|\hat b\|=1\):

\[
\operatorname{tr}M_\mu
=
\mathbb E_\mu[\operatorname{tr}(\hat b\hat b^\top)]
=
\mathbb E_\mu[\|\hat b\|^2]
=1.
\]

For any fixed target direction \(a\):

\[
\mathbb E_\mu[(a^\top \hat b)^2]
=
\mathbb E_\mu[a^\top \hat b\hat b^\top a]
=
a^\top M_\mu a.
\]

The worst-case target direction is the minimum-eigenvalue direction of \(M_\mu\):

\[
\inf_{\|a\|=1} a^\top M_\mu a
=
\lambda_{\min}(M_\mu).
\]

Because the average eigenvalue is \(1/D\):

\[
\lambda_{\min}(M_\mu)
\le
\frac1D.
\]

Hence:

\[
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
\le
\frac1D
\]

for every initialization \(\mu\). If \(\mu\) is isotropic, then:

\[
M_\mu=\frac1D I,
\]

so every direction has:

\[
a^\top M_\mu a=\frac1D.
\]

Thus:

\[
\sup_\mu
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]

This is a worst-case theorem. If the target direction is itself uniformly random, then:

\[
\mathbb E_a[a^\top M_\mu a]
=
\frac{\operatorname{tr}M_\mu}{D}
=
\frac1D
\]

for every \(\mu\). Therefore prior-free average-case alignment over uniform targets cannot distinguish initializations by their mean; the distinction is worst-case coverage, tail behavior, and conditioning.

## Prior-Aware Corollary

If target directions have prior covariance:

\[
\Sigma_A=\mathbb E[aa^\top],
\]

then:

\[
\mathbb E_{a,B}[(a^\top \hat b)^2]
=
\operatorname{tr}(\Sigma_A M_\mu).
\]

Therefore the optimal structured feedback distribution depends on the eigenspectrum of \(\Sigma_A\). Without prior information, isotropy is minimax; with prior information, top-eigenspace feedback can be better.

## Functional Capacity

Let total parameter dimension be:

\[
P=\sum_l n_l(n_{l-1}+1).
\]

Let task Jacobian be:

\[
J=\frac{\partial f_\theta(X)}{\partial \theta}.
\]

Local task rank:

\[
d=\operatorname{rank}(J).
\]

If alignment constraints remove a generic \(k\)-dimensional parameter subspace, then hard local function rank after constraints is:

\[
d_{\mathrm{hard}}=\min(d,P-k).
\]

Hard functional loss:

\[
\Delta d_{\mathrm{hard}}
=
d-d_{\mathrm{hard}}
=
\max(0,k-(P-d)).
\]

Interpretation:

- \(P-d\) is the local redundant parameter dimension.
- If \(k\le P-d\), generic alignment constraints can be absorbed by redundant directions without reducing hard local function rank.
- If \(k>P-d\), every additional generic constraint reduces hard local function rank one-for-one.

This gives the proposed FA/BP functional-gap onset: parameter-volume cost can grow before the model loses hard function rank, but once the alignment burden exhausts redundancy, FA should separate from BP more sharply.

Soft overlap model. Let \(E\) be the alignment constraint subspace and \(S\) be the task-sensitive subspace:

\[
T_k=\operatorname{tr}(P_E P_S).
\]

For random subspaces:

\[
\mathbb E[T_k]=\frac{kd}{P}.
\]

Variance:

\[
\operatorname{Var}(T_k)
=
\frac{
2kd(P-k)(P-d)
}{
P^2(P-1)(P+2)
}.
\]

Interpretation:

- Hard rank loss has a redundancy-exhaustion threshold.
- Soft conditioning loss can grow approximately linearly from the start.

## Trajectory Bridge

BP gradient:

\[
g_t=\nabla_\theta L(\theta_t).
\]

FA surrogate gradient:

\[
\tilde g_t(B).
\]

Mismatch:

\[
\epsilon_t(B)=\tilde g_t(B)-g_t.
\]

Linearized deviation from the BP trajectory:

\[
\delta\theta_T(B)
\approx
-\eta
\sum_{t<T}
\Phi_{T,t+1}\epsilon_t(B),
\]

where:

\[
\Phi_{T,t+1}
=
\prod_{s=t+1}^{T-1}(I-\eta H_s).
\]

Second-order excess loss:

\[
\Delta L_T(B)
\approx
\frac12
\delta\theta_T(B)^\top H_T\delta\theta_T(B).
\]

If \(\epsilon_t(B)\) is approximately Gaussian under random \(B\), then:

\[
\Delta L_T(B)
\approx
\frac12\sum_i \lambda_i\chi_i^2.
\]

This should be treated as a bridge approximation, not as a primary architecture-only theorem.

### First Empirical Bridge Variant

For a concrete first trajectory-distribution predictor, ignore the Hessian propagation factors and integrate the FA/BP gradient mismatch along the BP path:

\[
\widehat{\delta\theta}_T(B)
=
-\eta
\sum_{t<T}
\left[
g_t^{\mathrm{FA}}(\theta_t^{\mathrm{BP}};B)
-
g_t^{\mathrm{BP}}(\theta_t^{\mathrm{BP}})
\right].
\]

Then predict the final FA/BP loss gap by evaluating the loss at the BP endpoint plus this bridge deviation:

\[
\widehat{\Delta L}_T(B)
=
L(\theta_T^{\mathrm{BP}}+\widehat{\delta\theta}_T(B))
-
L(\theta_T^{\mathrm{BP}}).
\]

This is expected to be misscaled when curvature and contraction along the BP path are important. It is still useful as a distributional shape test: if the standardized \(\widehat{\Delta L}_T(B)\) distribution matches the standardized empirical \(\Delta L_T(B)\) distribution, then the bridge captures the random-feedback shape but not yet the absolute scale. A stronger next theorem should include a curvature or response operator replacing the identity propagation used here.