path: root/paper/body.tex

\begin{abstract}
Feedback alignment (FA) trains a neural network with \emph{fixed random}
matrices in place of the transposed weights that backpropagation (BP) uses
to send errors backward. A decade of work has established the surprising
fact that this \emph{works}. It has not established what it \emph{costs}.
We settle that question for multilayer perceptrons, exactly and without a
single fitted parameter. First, the geometric mismatch between a random
feedback matrix and the matrix BP would have used obeys an exact
$\Beta(\tfrac12,\tfrac{D-1}{2})$ law, which we turn into a capacity cost
that adds up to $\Theta(Ln^2)$ across an $L$-layer width-$n$ network.
Second, this cost cannot be engineered away: with no prior on the task, no
feedback initialization beats isotropic random feedback. Third---and this
is the crux---the resulting FA/BP optimization gap is \emph{always
strictly positive}, equals \emph{exactly} the hidden-layer share of BP's
learning speed at initialization, and, contrary to the natural guess,
never undergoes a phase transition: it is a \emph{soft ramp} that we derive
in closed form and predict to log-space correlation $0.98$ from
initialization alone, and to correlation $0.999$ with a no-fit early-time
estimator. The one-line summary is that random feedback is not free: its
cost is irreducible, but soft, and now exactly computable.
\end{abstract}

\section{Introduction}

Backpropagation rests on a hidden symmetry: to send the error backward
through a layer, it reuses the \emph{transpose of that layer's forward
weights}. Nothing in a brain or in a piece of analog hardware ties the
backward pathway to the forward one this way, which is why
\citet{lillicrap2016} asked what happens if you simply cut the tie---if you
replace each backward matrix $W^{\top}$ by a \emph{fixed random} matrix $B$
drawn once and never touched again. The answer was unexpected: the network
still learns, and the forward weights quietly rotate until they partly
align with the random feedback. This is \emph{feedback alignment} (FA), and
it and its variants train deep networks to low error~\cite{nokland2016}.

A decade later, the field has thoroughly answered one question---\emph{can}
FA learn?---and left the more useful one untouched: \emph{what does it
cost?} ``FA eventually reaches low error'' is perfectly compatible with FA
paying a steady, measurable price relative to BP at every step, a price one
might hope a wide enough network simply absorbs. Whether it does is not a
matter of opinion; it is a quantity, and in this paper we compute it.

Our message is simple and, we will argue, complete: \textbf{random feedback
imposes a cost that is irreducible but soft, and we give its exact ledger.}
We mean three things by this, and the paper is organized around them.

\begin{itemize}
\item \textbf{The static cost is a clean law (Sec.~\ref{sec:cap}).} How
badly does a random feedback matrix match the one BP wants? Exactly as
badly as the geometry forces: their squared cosine is
$\Beta(\tfrac12,\tfrac{D-1}{2})$-distributed, with mean $1/D$. Reading this
as a ``log-volume cost'' makes it \emph{add} across layers to
$\Theta(Ln^2)$.

\item \textbf{You cannot cheat the cost (Sec.~\ref{sec:minimax}).} Maybe a
cleverer fixed feedback would help? Not without knowing the task: the best
worst-case alignment any initialization can guarantee is $1/D$, and plain
isotropic random feedback already achieves it.

\item \textbf{The optimization gap is positive, soft, and computable
(Sec.~\ref{sec:gap}).} The interesting cost is the actual FA/BP training
gap. We prove it is always nonzero; that at initialization its expected
size is \emph{exactly} the fraction of BP's learning speed carried by the
hidden layers; and that over training it grows as a smooth ramp---not a
phase transition---which we write in closed form and confirm against dense
simulation. A natural hypothesis (the gap appears suddenly when redundant
capacity runs out) turns out to be wrong, and we can say precisely why.
\end{itemize}

We keep every model and theorem exact; what we work hard at is making them
\emph{legible}, because the value here is not a single inequality but a
coherent, quantitative picture of a phenomenon that has so far been
described only qualitatively.

\section{Setup: BP, FA, and ``Learning Speed''}\label{sec:setup}

Fix an MLP with widths $n_0,\dots,n_L$, forward weights
$W_l\in\R^{n_l\times n_{l-1}}$, and ReLU activations
$h_l=\phi(W_l h_{l-1})$. Train on $N$ examples with squared loss
$\mathcal{L}=\tfrac{1}{2N}\|r\|^2$, where $r=f_\theta(X)-Y$ stacks all
residuals.

The one thing that distinguishes FA from BP lives entirely in the backward
pass. BP propagates the error using $A_l=W_{l+1}^{\top}$; FA uses a fixed
random $B_l\in\R^{n_l\times n_{l+1}}$ instead. Write $g_l^{\mathrm{BP}}$ and
$g_l^{\mathrm{FA}}$ for the resulting gradient of layer $l$. One fact does
all the work later and is worth stating now: \emph{the output layer is
identical under both rules}, because no backward matrix sits above it, so
$g_L^{\mathrm{FA}}=g_L^{\mathrm{BP}}$. The flattened backward matrices live
in dimension $D_l=n_l n_{l+1}$.

To talk about ``cost'' we need a yardstick, and the natural one is how fast
the loss goes down. A step $-\eta g$ decreases the loss, to first order, by
$\eta\langle g^{\mathrm{BP}},g\rangle$---the true gradient dotted with the
step you actually take. So define
\begin{align}
\mathrm{speed}_{\mathrm{BP}} &= \textstyle\sum_l \|g_l^{\mathrm{BP}}\|^2,\\
\mathrm{speed}_{\mathrm{FA}} &= \textstyle\sum_l \langle g_l^{\mathrm{BP}},g_l^{\mathrm{FA}}\rangle .
\end{align}
The first is how fast BP descends; the second is how fast the FA step
descends the \emph{true} loss. Their normalized difference,
\begin{equation}\label{eq:e0}
e_0 = 1-\frac{\mathrm{speed}_{\mathrm{FA}}}{\mathrm{speed}_{\mathrm{BP}}},
\end{equation}
is the fraction of BP's learning speed that FA throws away in one step.
$e_0=0$ means FA is as good as BP right now; $e_0=1$ means FA makes no
progress at all. Everything in Section~\ref{sec:gap} is about how large
$e_0$ is, and what it compounds to over training.

Finally, two operators. Let $J=\partial f_\theta(X)/\partial\theta$ be the
Jacobian of all outputs. BP's tangent kernel (the NTK) is $\KBP=JJ^{\top}$,
and the residual moves as $r_{t+1}\approx(I-\tfrac{\eta}{N}\KBP)r_t$---this
is just gradient descent written in output space, and we use it as our
lens on training. FA swaps the update Jacobian for a surrogate $\tilde J$,
giving a \emph{non-symmetric} operator $\KFA=J\tilde J^{\top}$ whose
symmetric part $\SFA=\tfrac12(\KFA+\KFA^{\top})$ is all that affects the
loss to first order. Keep $\KBP$ (``what BP does'') and $\SFA$ (``what FA
does'') in mind; the whole story is a comparison of these two.

\section{The Static Cost Is a Clean Law}\label{sec:cap}

Start with the most basic question: at initialization, how far is a random
feedback matrix $B$ from the matrix $A=W^{\top}$ that BP would use? Measure
``far'' by the squared cosine between them. The answer is not approximate
or asymptotic---it is an exact classical distribution.

\begin{theorem}[Alignment law]\label{thm:beta}
Let $a=\vecop(A)/\|A\|_F$ and $b=\vecop(B)/\|B\|_F$ be independent isotropic
unit directions in $\R^D$, and $Q=(a^{\top}b)^2$. Then
\begin{equation}
Q\sim\Beta\!\left(\tfrac12,\tfrac{D-1}{2}\right),\qquad \E[Q]=\tfrac1D .
\end{equation}
\end{theorem}
\begin{proof}[Proof]
Rotational invariance lets us fix $a=e_1$, so $Q=b_1^2$ with $b$ uniform on
$\mathbb{S}^{D-1}$. The squared first coordinate of a uniform sphere vector
is exactly $\Beta(\tfrac12,\tfrac{D-1}{2})$.
\end{proof}

Read off the consequence: the typical cosine is about $1/\sqrt D$, so in
any realistic layer (with $D=n^2$ in the thousands) \emph{a random feedback
matrix is essentially perpendicular to the one BP wants}. That near-orthogonality is
the entire geometric source of FA's difficulty, and Theorem~\ref{thm:beta}
says exactly how perpendicular, with no slack.

It is convenient to price this. For a target alignment $q$, define the
\emph{log-volume cost} $C(q)=-\log\Pr(Q\ge q)$---the number of nats by
which reaching alignment $q$ is improbable by chance, equivalently the log
of how small a sliver of feedback space clears the bar. For fixed $q$ in
high dimension, $C(q)\approx\tfrac{D-1}{2}\log\tfrac1{1-q}=\Theta(D)$. The
reason to bother defining it is that it behaves like a true cost---it adds.

\begin{proposition}[Multilayer scaling]\label{prop:scaling}
For independent layer-wise feedback blocks the alignment events are
independent, so their probabilities multiply and the costs add:
$C_{\mathrm{all}}=\sum_l C_l(q_l)$. For an equal-width MLP ($D_l\approx
n^2$) at fixed $q$,
\begin{equation}
C_{\mathrm{all}}=\Theta(Ln^2),\qquad
\Pr(\text{all }q_l)=e^{-\Theta(Ln^2)} .
\end{equation}
\end{proposition}

So the ``bits of luck'' required to make random feedback align grow
linearly in the number of weights, and the probability of getting that luck
shrinks exponentially in it. Both depth and width make raw alignment
exponentially harder---a clean, if unsurprising, fact that fixes the units
for everything that follows.

\section{You Cannot Cheat the Cost}\label{sec:minimax}

A reader's first instinct is to fight the geometry: surely some carefully
chosen, still-fixed feedback aligns better than dumb isotropic noise? The
honest answer is that you can---but only if you already know which backward
direction the task will need, which at initialization you do not.

\begin{theorem}[Prior-free minimax]\label{thm:minimax}
Let $b\in\mathbb{S}^{D-1}$ be drawn from any distribution $\mu$ and let
$a\in\mathbb{S}^{D-1}$ be an unknown target. With $M_\mu=\E_\mu[bb^{\top}]$
(so $\mathrm{tr}\,M_\mu=1$),
\begin{equation}
\sup_{\mu}\ \inf_{\|a\|=1}\ \E_\mu[(a^{\top}b)^2]
=\sup_\mu \lambda_{\min}(M_\mu)=\tfrac1D,
\end{equation}
attained by isotropic feedback ($M_\mu=\tfrac1D I$).
\end{theorem}
\begin{proof}[Proof]
$\E_\mu[(a^{\top}b)^2]=a^{\top}M_\mu a$; its worst case over $a$ is
$\lambda_{\min}(M_\mu)\le\mathrm{tr}\,M_\mu/D=1/D$. Isotropic $\mu$ makes
every direction equal to $1/D$.
\end{proof}

The point of stating it as a minimax is to forestall a class of ``fixes.''
Any anisotropic feedback that aligns better with some directions does so by
aligning \emph{worse} with others, and an adversarially chosen task lands
on the bad ones. Sparsity, orthogonality, or low rank can change
conditioning, signal norm, or hardware cost---but they cannot beat $1/D$ in
the worst case. Isotropic random feedback is not a lazy default; it is
\emph{provably the best prior-free choice}. The cost in
Section~\ref{sec:cap} is therefore not an artifact of using noise; it is a
floor.

\section{The Optimization Gap: Positive, Soft, Computable}\label{sec:gap}

Static geometry is the setup; the payoff is the actual FA/BP training gap.
We make three claims, in increasing strength: the gap is never zero; at
initialization its size is \emph{exactly} a quantity you can read off the
forward pass; and over training it follows a closed-form \emph{soft ramp}.

\subsection{The gap is never zero, and never sudden}

There is a tempting story---``a wide network has spare directions, so it
absorbs the random-feedback constraint for free until the spares run out,
then the gap snaps open.'' It is wrong, and a one-line null model shows why.
Suppose alignment simply deleted a uniformly random $k$-dimensional
subspace $E$ of the $P$ parameters, leaving the projected kernel
$JQ_E J^{\top}$ with $Q_E=I-P_E$. Writing $v=J^{\top}r$, the one-step
erosion is $\|P_E v\|^2/\|v\|^2$, and for Haar-random $E$,
\begin{equation}
e(r)\sim\Beta\!\left(\tfrac k2,\tfrac{P-k}{2}\right),\quad
\E[e(r)]=\tfrac kP,\quad \Pr(e=0)=0 .
\end{equation}
For \emph{any} positive burden $k$, the erosion is almost surely nonzero,
with mean $k/P$. Over-parameterization (large $P$) shrinks the gap through
the ratio $k/P$, but it never reaches zero, and nothing special happens at
any threshold. There is no phase transition to wait for.

\takeaway{Capacity dilutes the FA burden; it does not delete it. The gap is
soft.}

This null model is deliberately crude---it over-predicts the real gap
(Sec.~\ref{sec:exp})---but it isolates the mechanism. The exact statement
for genuine FA is next, and it is sharper than $k/P$.

\subsection{At initialization, the gap is the hidden speed share}

Here is the result we would put on the board first. It says the effective
initial cost of FA is not some opaque function of the random matrices; it
is a number already sitting in the BP gradients.

\begin{theorem}[Initial erosion = hidden speed share]\label{thm:e0}
Fix the forward weights $W$, data, and residual $r$. Assume the feedback
matrices $B_l$ are independent of $W,r$ with zero mean, and that the output
layer uses the true gradient. Then
$\E_B[\mathrm{speed}_{\mathrm{FA}}\mid W,r]=\|g_L^{\mathrm{BP}}\|^2$, so
\begin{equation}\label{eq:hidden-share}
\E_B[e_0\mid W,r]
= 1-\frac{\|g_L^{\mathrm{BP}}\|^2}{\sum_l \|g_l^{\mathrm{BP}}\|^2}
\;=:\;1-\rho .
\end{equation}
\end{theorem}
\begin{proof}[Proof]
The output FA gradient equals the BP gradient, contributing
$\|g_L^{\mathrm{BP}}\|^2$. Every hidden FA gradient is multilinear in
zero-mean independent feedback matrices, so its conditional expectation
vanishes; hence each hidden term
$\langle g_l^{\mathrm{BP}},g_l^{\mathrm{FA}}\rangle$ has zero mean. Summing
leaves only the output term.
\end{proof}

Unpack what this says, because it is more intuitive than it looks. On the
first step, random feedback points the hidden updates in random directions,
so \emph{on average they accomplish nothing}; FA gets only the output
layer, which it computes correctly, ``for free.'' The expected cost is
therefore precisely the share of the work the hidden layers were doing---a
number, $1-\rho$, that you get from one BP backward pass. We will call the
surviving fraction $\rho=\mathrm{output\ share}\in(0,1)$; it is the single
parameter that drives the rest of the paper. Note also what this does to
the kernel: in the $\KFA=G+H_{\mathrm{FA}}$ language of \citet{song2021},
it says $\E_B[H_{\mathrm{FA}}]=0$, i.e.\ at initialization FA's expected
operator is exactly the top-layer kernel.

For the simplest case we get the whole distribution, not just its mean.

\begin{proposition}[Exact Gaussian erosion]\label{prop:gauss}
For a one-hidden-layer ReLU MLP with i.i.d.\ Gaussian feedback
$B\sim\mathcal{N}(0,\sigma_B^2)$, the hidden mixed term is linear in $B$:
$\langle g_1^{\mathrm{BP}},g_1^{\mathrm{FA}}(B)\rangle=\langle C(W,r),B\rangle$
for an explicit $C(W,r)$. Hence
$e_0\sim\mathcal{N}\!\big(1-\rho,\ \sigma_B^2\|C\|_F^2/\mathrm{speed}_{\mathrm{BP}}^2\big)$,
conditionally on $(W,r)$.
\end{proposition}

\subsection{Over training, the gap is a closed-form soft ramp}

The number $1-\rho$ is a single step's loss; what reaches the reviewer's
plots is its cumulative effect over $T$ steps. To get there we make two
approximations, and we state them plainly rather than hide them, because
knowing exactly where the model is idealized is the point.

\begin{assumption}[Lazy operators]\label{ass:lazy}
The tangent operators are approximately frozen at initialization, so
$r_t^{\mathrm{BP}}=(I-\tfrac{\eta}{N}\KBP)^t r_0$.
\end{assumption}
\begin{assumption}[Scalar erosion]\label{ass:scalar}
$\SFA\approx\rho\,\KBP$. This is \emph{exact in the residual direction} by
Theorem~\ref{thm:e0} ($r^{\top}\E_B[\SFA]r=\rho\,r^{\top}\KBP r$); the
assumption is that the same slowdown $\rho$ applies in every eigendirection.
\end{assumption}

With these, FA is simply BP slowed by a factor $\rho$ in every mode, and the
gap drops out in one line.

\begin{theorem}[Closed-form gap]\label{thm:gap}
Under Assumptions~\ref{ass:lazy}--\ref{ass:scalar}, in the eigenbasis
$\KBP v_i=\lambda_i v_i$ with $c_i=v_i^{\top}r_0$,
\begin{equation}\label{eq:gap}
\mathrm{gap}_T=\frac{1}{2N}\sum_i c_i^2\Big[
\big(1-\tfrac{\eta\rho\lambda_i}{N}\big)^{2T}
-\big(1-\tfrac{\eta\lambda_i}{N}\big)^{2T}\Big].
\end{equation}
\end{theorem}
\begin{proof}[Proof]
Expand $r_0=\sum_i c_i v_i$. Assumption~\ref{ass:lazy} gives
$\|r_T^{\mathrm{BP}}\|^2=\sum_i c_i^2(1-\tfrac{\eta\lambda_i}{N})^{2T}$;
Assumption~\ref{ass:scalar} makes $\SFA$ share eigenvectors with eigenvalues
$\rho\lambda_i$, giving the FA term. Subtract, divide by $2N$.
\end{proof}

Now read the formula, because it settles the phase-transition question on
sight. Every bracket is nonnegative---$\rho<1$ makes each FA mode decay
slower than its BP counterpart, so each contributes a little extra residual.
And the whole sum is \emph{analytic} in $\rho$, the spectrum, and $T$. An
analytic function has no kinks; deform the capacity smoothly and the gap
moves smoothly. The only way to manufacture a sudden jump would be to send
$\rho\to0$ (FA stalling completely), which never happens because
$\rho\ge\mathrm{output\ share}>0$. The soft ramp is not an empirical
accident---it is forced by the algebra.

\begin{corollary}[Soft, not a phase transition]\label{cor:soft}
$\mathrm{gap}_T$ is real-analytic in $(\rho,\{\lambda_i\},T)$ and strictly
positive whenever $\rho<1$; it has no threshold. A discontinuity would
require $\rho\to0$.
\end{corollary}

The same formula tells you \emph{where} the ramp comes from. Once BP has
fit the data, the gap is just the error FA still has in the single slowest
direction:

\begin{corollary}[The ramp law]\label{cor:ramp}
When $\|r_T^{\mathrm{BP}}\|\approx0$,
\begin{equation}
\mathrm{gap}_T\approx\frac{c_{\min}^2}{2N}\,e^{-2\eta\rho\lambda_{\min}T/N},
\quad
\log\mathrm{gap}_T\approx\text{const}-\tfrac{2\eta T}{N}\,\rho\,\lambda_{\min}.
\end{equation}
\end{corollary}

In words: the FA/BP gap is the leftover error in the slowest-learning
direction, where FA crawls at a fraction $\rho$ of BP's rate. As you add
capacity, the smallest kernel eigenvalue $\lambda_{\min}$ grows smoothly, so
$\log\mathrm{gap}$ slides down linearly. That is the ramp, and
Section~\ref{sec:exp} shows the data falls exactly on this line.

\paragraph{Mechanism vs.\ magnitude.} We are upfront about the one thing the
closed form does not nail. Freezing the operators (Assumption~\ref{ass:lazy})
captures the \emph{shape} of the ramp but underestimates its travel, because
during training the forward weights start to align with $B$ and the
operators drift---this drift is the very ``alignment'' phenomenon that
makes FA work at all. We recover the missing magnitude with a no-fit
\emph{early-velocity estimator}: read the kernels $K_0$ and $K_s$ at one
early step $s$, extrapolate $\hat K_t=K_0+\tfrac ts(K_s-K_0)$, and roll out
the residual recursion for both rules. No scalar is fit; the only inputs are
two early snapshots. The closed form tells you \emph{why} the ramp is soft;
the estimator tells you \emph{how big} it is. Together they are the
complete finite-time answer.

\section{Experiments}\label{sec:exp}

Everything below uses full-batch training of MLPs on synthetic data, and we
emphasize the headline up front: \textbf{no quantity in this paper is
fitted.} Every prediction is computed from initialization or from two early
kernel snapshots.

\paragraph{The static law holds (Thm.~\ref{thm:beta}).} Sampling matrix
pairs, the empirical squared cosine matches
$\Beta(\tfrac12,\tfrac{D-1}{2})$ across dimensions (Kolmogorov--Smirnov
$p>0.3$). The geometry is as predicted, exactly.

\paragraph{The initial gap is the hidden share (Thm.~\ref{thm:e0}).}
Running the real FA backward pass over many feedback draws on a
two-hidden-layer network, the empirical mean erosion lands on the no-fit
prediction \eqref{eq:hidden-share}: $\E_B[e_0]=0.4378\pm0.008$ versus the
predicted hidden share $0.4403$. Proposition~\ref{prop:gauss}'s Gaussian
shape passes KS tests across widths. We did not tune anything; the number
comes straight from one BP backward pass.

\paragraph{The ramp is soft, and the closed form explains it
(Cor.~\ref{cor:soft}--\ref{cor:ramp}).} Sweeping width on a random-label
task to $T=30{,}000$ steps, the FA/BP train gap slides smoothly from
$0.147$ to $1.6\times10^{-4}$ as the (hard) capacity margin runs from $-190$
to $+130$---\emph{with no kink at margin zero and a gap that stays nonzero
even when the network is over-capacity} (Fig.~\ref{fig:ramp}, blue). The
empirical $\log\mathrm{gap}$ is linear in the margin ($R^2=0.993$), exactly
the form Corollary~\ref{cor:ramp} predicts. Evaluating the closed-form law
\eqref{eq:gap} with $\rho,\lambda_i,c_i$ measured \emph{only at
initialization} reproduces the ramp's shape at log-space correlation
$0.977$ (Fig.~\ref{fig:ramp}, orange), and $\lambda_{\min}$ alone---which
grows from $0.07$ to $0.99$ across the sweep---explains the gap at
correlation $0.963$. The frozen closed form compresses the range, exactly
the operator drift the estimator was built to recover.

\paragraph{The estimator is essentially exact (no fit).} Across $256$
trajectories and a stress grid over depth $\{1,2,3\}$, width
$\{32,64,96\}$, and horizon $\{25,50,100\}$, the early-velocity estimator
predicts the finite-time gap at mean absolute error
$\approx1.9\times10^{-3}$ and correlation $0.994$--$0.99989$---about a
ten-fold improvement over the frozen-$K_0$ baseline, again with no fitted
parameter. For a phenomenon previously described only by adjectives, this
is a quantitative predictor that simply works.

\begin{figure*}[t]
\centering
\includegraphics[width=0.92\textwidth]{closed_form_vs_measured_ramp.png}
\caption{\textbf{The FA/BP gap is a soft ramp, and our closed form explains
it.} \emph{Left:} the empirical gap (blue) and the no-fit closed-form
prediction \eqref{eq:gap} (orange) across capacity margin---both smooth,
neither with a kink at margin $0$, and the gap stays positive even at
positive margin (no phase transition). \emph{Right:} the ramp is driven by
the smallest BP-kernel eigenvalue $\lambda_{\min}$ exactly as
Corollary~\ref{cor:ramp} predicts ($\mathrm{corr}=0.963$).}
\label{fig:ramp}
\end{figure*}

\section{Related Work}

\paragraph{Feedback alignment.} \citet{lillicrap2016} introduced FA and
documented that forward weights drift into alignment with the random
feedback; \citet{nokland2016} carried this to direct feedback alignment in
deep nets; \citet{refinetti2021} described the align-then-memorize
dynamics. These works answer ``can FA learn?''. We answer ``what does it
cost?''---a different and, we would argue, more basic question, and one
their results leave entirely open.

\paragraph{Convergence theory.} \citet{song2021} prove two-layer FA
converges when over-parameterized, with alignment optional absent
regularization. Far from conflicting with us, this is the perfect
complement: their decomposition $\KFA=G+H_{\mathrm{FA}}$ is, at
initialization, exactly our Theorem~\ref{thm:e0}, which forces
$\E_B[H_{\mathrm{FA}}]=0$ and hence $\E_B[\KFA]=G$. Convergence to zero
\emph{final} error and a nonzero \emph{operator} gap are simply different
statements, and our paper is about the second.

\paragraph{Tangent kernels and capacity.} The NTK and lazy-training
line~\cite{jacot2018,lee2019} licenses the residual recursion we build on;
our contribution there is to handle FA's non-symmetric operator. And while
the fact that nets can fit random labels~\cite{zhang2017} invites the guess
that capacity makes random feedback free, our soft-erosion result is exactly
the rebuttal: capacity dilutes the cost through $k/P$ and never removes it.

\section{Limitations}

We are explicit about scope. The experiments are synthetic MLPs trained
full-batch; convolutional and transformer architectures, stochastic
optimization, and real data are future work. The closed-form law
(Thm.~\ref{thm:gap}) uses a lazy-training approximation and a scalar erosion
model that is exact only in the residual direction---a per-eigenmode
refinement is the obvious next theorem. And the finite-time estimator,
though unfitted, is \emph{conditional} on two early kernel snapshots;
deriving the operator drift from the architecture alone---via infinite-width
recursions for $\rho$ and $\lambda_{\min}$---would close the last gap
between mechanism and magnitude. None of these qualify the three results we
do prove.

\section{Conclusion}

Random feedback is not free, and after this paper it is no longer
mysterious. Its static cost is an exact $\Beta$ law that adds to
$\Theta(Ln^2)$; that cost is provably the best any prior-free initialization
can do; and the optimization gap it induces is always positive, equals the
hidden-layer speed share at initialization, and grows as a closed-form soft
ramp that we predict without fitting anything. The reason over-parameterized
FA can look almost free is now precise: the cost is real and irreducible, but
soft, scaling as $k/P$ and as the leftover error in the slowest learning
direction. We think this is the right way to understand feedback
alignment---as a quantity, not an anecdote.

\begin{thebibliography}{9}
\bibitem[Lillicrap et al.(2016)]{lillicrap2016}
T.~Lillicrap, D.~Cownden, D.~Tweed, and C.~Akerman.
\newblock Random synaptic feedback weights support error backpropagation
for deep learning.
\newblock \emph{Nature Communications}, 2016.

\bibitem[N{\o}kland(2016)]{nokland2016}
A.~N{\o}kland.
\newblock Direct feedback alignment provides learning in deep neural
networks.
\newblock In \emph{NeurIPS}, 2016.

\bibitem[Refinetti et al.(2021)]{refinetti2021}
M.~Refinetti, S.~d'Ascoli, R.~Ohana, and S.~Goldt.
\newblock Align, then memorise: the dynamics of learning with feedback
alignment.
\newblock In \emph{ICML}, 2021.

\bibitem[Song et al.(2021)]{song2021}
G.~Song, R.~Xu, and J.~Lafferty.
\newblock Convergence and alignment of gradient descent with random
backpropagation weights.
\newblock In \emph{NeurIPS}, 2021.

\bibitem[Jacot et al.(2018)]{jacot2018}
A.~Jacot, F.~Gabriel, and C.~Hongler.
\newblock Neural tangent kernel: convergence and generalization in neural
networks.
\newblock In \emph{NeurIPS}, 2018.

\bibitem[Lee et al.(2019)]{lee2019}
J.~Lee et al.
\newblock Wide neural networks of any depth evolve as linear models under
gradient descent.
\newblock In \emph{NeurIPS}, 2019.

\bibitem[Zhang et al.(2017)]{zhang2017}
C.~Zhang, S.~Bengio, M.~Hardt, B.~Recht, and O.~Vinyals.
\newblock Understanding deep learning requires rethinking generalization.
\newblock In \emph{ICLR}, 2017.
\end{thebibliography}