path: root/notes/07_bp_capacity_theory.md

# BP-Capacity Theory Correction

The previous hard-count transition model is not acceptable as a main theory:

\[
M_{\mathrm{FA}}=P-K_{\mathrm{FA}}-Nn_L
\]

is only a crude diagnostic. It treats BP capacity as raw parameter count and
treats FA alignment as full hard constraints. The main paper should instead use
a capacity definition grounded in the training-set Jacobian / NTK.

## Literature-Supported BP Capacity

For squared loss on a finite training set, the mature local object is the output
Jacobian

\[
J_{\mathrm{BP}}(\theta, X)
=
\frac{\partial f_\theta(X)}{\partial \theta}
\in
\mathbb R^{Nn_L\times P}.
\]

Karhadkar et al. prove the key local lemma: at a differentiable critical point,
if the training-set Jacobian has full row rank, then the squared-loss critical
point is a global minimizer. Their deep-ReLU results further show that many
activation regions have full-rank Jacobian under explicit activation-pattern
rank conditions.

Therefore the hard BP interpolation capacity should be

\[
r_{\mathrm{BP}}(\theta)
=
\operatorname{rank} J_{\mathrm{BP}}(\theta, X),
\]

not \(P\) and not an empirical fitted multiple of \(Nn_L\).

For finite-time or ill-conditioned training, use the empirical NTK

\[
K_{\mathrm{BP}}(\theta)
=
J_{\mathrm{BP}}(\theta,X)J_{\mathrm{BP}}(\theta,X)^\top
\]

and its effective dimension

\[
d_{\mathrm{BP}}(\sigma^2)
=
\operatorname{tr}
\left[
K_{\mathrm{BP}}
\left(K_{\mathrm{BP}}+\sigma^2 I\right)^{-1}
\right].
\]

This follows the NTK view: training progress on finite data is governed by the
kernel Gram spectrum, with fast convergence in large-eigenvalue directions and
slow convergence in small-eigenvalue directions.

## FA Capacity Occupation

There are two separate FA capacity notions.

1. Matrix-alignment capacity: how much parameter-direction volume is occupied by
   making forward matrices align with feedback matrices.
2. Downstream training capacity: how much of BP's local training kernel survives
   when BP gradients are replaced by FA surrogate gradients.

The downstream experiment should use the second object as its main predictor.

## FA Surrogate-Gradient Kernel

Let \(m=Nn_L\). For each output coordinate basis vector \(e_i\in\mathbb R^m\),
define the FA pseudo-gradient direction

\[
\tilde g_i^{\mathrm{FA}}(B,\theta)
\in
\mathbb R^P
\]

obtained by running the FA backward rule with output residual \(e_i\). Stack
these rows into

\[
\tilde J_{\mathrm{FA}}(B,\theta)
\in
\mathbb R^{m\times P}.
\]

For squared loss with residual \(r=f_\theta(X)-y\), the BP update is

\[
\Delta\theta_{\mathrm{BP}}
=
-\eta J_{\mathrm{BP}}^\top r,
\]

and the FA update is

\[
\Delta\theta_{\mathrm{FA}}
=
-\eta \tilde J_{\mathrm{FA}}^\top r.
\]

The induced first-order output dynamics are therefore

\[
\Delta f_{\mathrm{BP}}
=
-\eta
K_{\mathrm{BP}}r,
\qquad
K_{\mathrm{BP}}=J_{\mathrm{BP}}J_{\mathrm{BP}}^\top,
\]

and

\[
\Delta f_{\mathrm{FA}}
=
-\eta
K_{\mathrm{FA}}r,
\qquad
K_{\mathrm{FA}}
=
J_{\mathrm{BP}}\tilde J_{\mathrm{FA}}^\top.
\]

This is the correct local object for downstream FA/BP gap. It is also the
natural analogue of the surrogate-gradient NTK: the forward Jacobian controls
how parameter motion changes outputs, while the surrogate Jacobian controls the
parameter motion chosen by the learning rule.

Since \(K_{\mathrm{FA}}\) can be non-symmetric, hard local solvability is governed
by

\[
\operatorname{rank}(K_{\mathrm{FA}}),
\]

while stable finite-time learning should be measured by the spectrum of the
symmetrized or positive-real part of the operator. A conservative symmetric
capacity proxy is

\[
H_{\mathrm{FA}}
=
\frac{K_{\mathrm{FA}}+K_{\mathrm{FA}}^\top}{2},
\]

with positive eigenvalues \(\lambda_i^+\). Then

\[
d_{\mathrm{FA}}^{\mathrm{dyn}}(\sigma^2)
=
\sum_i
\frac{\max(\lambda_i,0)}
{\max(\lambda_i,0)+\sigma^2}.
\]

This is not a fitted scale. It is computed directly from the local FA learning
operator.

## Alignment-Constraint Projection

Let \(C_{\mathrm{FA}}\) be the local Jacobian of the FA alignment constraints
with respect to parameters. The hard constraint projector is

\[
\Pi_C
=
I
-
C_{\mathrm{FA}}^\top
\left(C_{\mathrm{FA}}C_{\mathrm{FA}}^\top\right)^+
C_{\mathrm{FA}}.
\]

The hard FA-accessible task rank is

\[
r_{\mathrm{FA}}
=
\operatorname{rank}
\left(
J_{\mathrm{BP}}\Pi_C J_{\mathrm{BP}}^\top
\right).
\]

So the capacity loss is

\[
\Delta r
=
r_{\mathrm{BP}}-r_{\mathrm{FA}}.
\]

For a soft alignment burden, use

\[
K_{\mathrm{FA},\lambda}
=
J_{\mathrm{BP}}
\left(M+\lambda C_{\mathrm{FA}}^\top C_{\mathrm{FA}}\right)^{-1}
J_{\mathrm{BP}}^\top,
\]

and

\[
d_{\mathrm{FA},\lambda}(\sigma^2)
=
\operatorname{tr}
\left[
K_{\mathrm{FA},\lambda}
\left(K_{\mathrm{FA},\lambda}+\sigma^2 I\right)^{-1}
\right].
\]

The soft capacity loss is

\[
\Delta d_\lambda
=
d_{\mathrm{BP}}(\sigma^2)
-
d_{\mathrm{FA},\lambda}(\sigma^2).
\]

No fitted scalar is allowed here. Every quantity is computed from \(J\), \(C\),
and a declared metric \(M\).

## Correct Transition Statement

For downstream memorization experiments, the FA/BP transition should first be
predicted by the FA surrogate-gradient kernel:

\[
\operatorname{rank}(K_{\mathrm{FA}})\approx Nn_L
\]

or, for finite-time training,

\[
d_{\mathrm{FA}}^{\mathrm{dyn}}(\sigma^2)\approx Nn_L.
\]

For the matrix-alignment capacity theory, the transition can also be written as

\[
r_{\mathrm{FA}}\approx Nn_L
\]

in the hard projected-rank model, or by

\[
d_{\mathrm{FA},\lambda}(\sigma^2)\approx Nn_L
\]

in the finite-time/effective-rank model.

Equivalently: the breakpoint occurs when FA-projected tangent capacity can no
longer span the training output constraints.

This is the clean version of "redundancy exhaustion":

\[
\operatorname{range}(J_{\mathrm{BP}}^\top)
\]

is the BP task-sensitive subspace. FA alignment only hurts downstream capacity
to the extent that its constraint directions overlap this subspace.

Under a generic random-subspace approximation with

\[
r_{\mathrm{BP}}=\operatorname{rank}J_{\mathrm{BP}},
\qquad
k=\operatorname{rank}C_{\mathrm{FA}},
\]

the expected soft overlap is

\[
\mathbb E\operatorname{tr}(P_C P_J)
=
\frac{kr_{\mathrm{BP}}}{P}.
\]

The hard generic rank after projection is

\[
r_{\mathrm{FA}}
=
\min(r_{\mathrm{BP}}, P-k).
\]

This gives a simple theorem without post-hoc parameters, but the experimentally
plotted prediction should prefer the directly measured Jacobian/NTK quantities.

## What To Change In Experiments

The next downstream-capacity plot should not use only
\(P-K_{\mathrm{FA}}-Nn_L\). It should compute, per architecture and seed:

1. \(r_{\mathrm{BP}}\) or \(d_{\mathrm{BP}}(\sigma^2)\) on the full training set.
2. \(\operatorname{rank}(K_{\mathrm{FA}})\) and
   \(d_{\mathrm{FA}}^{\mathrm{dyn}}(\sigma^2)\) from the FA pseudo-gradient
   kernel.
3. \(r_{\mathrm{FA}}\) or \(d_{\mathrm{FA},\lambda}(\sigma^2)\) after applying
   the FA alignment projector / penalty as the matrix-capacity ablation.
4. A predicted distribution of train-gap transition from these computed spectra,
   not from a fitted shift.

The old hard margin may remain as an ablation or sanity-check axis, not as the
main theorem.

## Current Diagnostic

The script

`scripts/fa_tangent_kernel_capacity.py`

computes \(J_{\mathrm{BP}}\), \(\tilde J_{\mathrm{FA}}\),
\(K_{\mathrm{BP}}\), and \(K_{\mathrm{FA}}\) on the full training set for the
existing downstream sweep.

A smoke/probe run on the current random-label Adam experiment shows that the
initial \(K_{\mathrm{FA}}\) often has a large negative symmetric spectrum. The
continuous-time initial-kernel prediction therefore greatly overestimates FA
failure, while the actual Adam FA trajectories recover through nonlinear
alignment dynamics.

This means:

1. the FA tangent-kernel object is the right local capacity object;
2. the current Adam random-label experiment is not a valid test of a fixed
   initialization-kernel final-gap theorem;
3. a quantitative final-gap theory must either use an experiment in the
   SGD/lazy-kernel regime, or model the time-varying FA alignment trajectory.

Do not present the initial-kernel prediction as a main downstream theorem for
the existing Adam trajectories.

## References To Cite

- Karhadkar et al., *Mildly Overparameterized ReLU Networks Have a Favorable
  Loss Landscape*: full-rank training-set Jacobian implies squared-loss critical
  points are global minimizers; deep-ReLU activation regions can have full-rank
  Jacobians under explicit rank conditions.
- Jacot et al., *Neural Tangent Kernel*: finite-data training dynamics are
  governed by the NTK Gram spectrum; convergence follows kernel principal
  components.
- Eilers et al., *A Generalized Neural Tangent Kernel for Surrogate Gradient
  Learning*: surrogate-gradient methods can be analyzed by a generalized NTK;
  FA should be treated as a surrogate-gradient learning rule for downstream
  dynamics.
- Yun et al., *Small ReLU Networks Are Powerful Memorizers*: ReLU memorization
  capacity scales with hidden-layer parameter products, supporting the need to
  treat memorization capacity separately from raw parameter count.
- Bartlett et al., *Nearly-Tight VC-Dimension and Pseudodimension Bounds for
  Piecewise Linear Neural Networks*: worst-case expressivity scales differently
  from local trainability; useful as a background upper-bound reference, not the
  exact transition predictor.