1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
|
# Project Overview
## Working Title
Distributional Capacity Bounds for Feedback Alignment in Multilayer Perceptrons
## Core Question
Feedback alignment (FA) replaces the exact BP feedback operator with a fixed random matrix. The central question is:
> Can we quantify, distributionally and architecturally, how much additional alignment burden FA introduces relative to BP?
The goal is not to claim that architecture alone predicts final test accuracy. The goal is to derive distributional bounds and capacity proxies for the extra alignment burden induced by random feedback, then validate how these proxies relate to observed FA/BP trajectory gaps.
## Contribution Map
### 1. Capacity Formalization
Define a hierarchy of capacity notions for FA:
- Angular alignment statistic.
- Log-volume parameter capacity cost.
- Task-projected functional capacity.
Layerwise alignment:
\[
Q_l =
\cos^2(W_{l+1}^{\top}, B_l)
=
\frac{
\langle W_{l+1}^{\top}, B_l\rangle_F^2
}{
\|W_{l+1}^{\top}\|_F^2 \|B_l\|_F^2
}.
\]
If directions are independent and isotropic in dimension \(D_l=n_l n_{l+1}\):
\[
Q_l \sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]
Define log-volume cost:
\[
C_l(q) =
-\log \Pr(Q_l\ge q)
=
-\log\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]
### 2. Scaling Law and Redundancy Exhaustion
Across independent layers:
\[
p_{\mathrm{all}} = \prod_l p_l(q_l),
\qquad
C_{\mathrm{all}} = -\log p_{\mathrm{all}} = \sum_l C_l(q_l).
\]
Therefore:
- Log-capacity cost accumulates linearly.
- Raw feasible volume decays geometrically.
For equal-width MLPs with width \(n\), depth \(L\), and fixed threshold \(q\):
\[
C_{\mathrm{all}} = \Theta(Ln^2),
\qquad
p_{\mathrm{all}} = \exp[-\Theta(Ln^2)].
\]
Functional gap should appear when alignment constraints exhaust redundant directions. With total parameter dimension \(P\), local task rank \(d=\operatorname{rank}(J)\), and effective alignment constraint rank \(k\):
\[
\Delta d_{\mathrm{hard}}
=
\max(0, k-(P-d)).
\]
### 3. Prior-Free Minimax Bound
For normalized feedback direction:
\[
\hat b = \frac{\operatorname{vec}(B)}{\|B\|_F}\in \mathbb S^{D-1},
\]
any initialization distribution \(\mu\) induces:
\[
M_\mu = \mathbb E_{\mu}[\hat b\hat b^\top],
\qquad
\operatorname{tr} M_\mu = 1.
\]
For unknown target direction \(a\):
\[
\mathbb E_\mu[(a^\top \hat b)^2] = a^\top M_\mu a.
\]
Thus:
\[
\sup_\mu \inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\frac1D.
\]
Isotropic random feedback achieves this minimax optimum. Scale, orthogonality, sparsity, and low rank can affect conditioning or cost, but cannot break the prior-free angular bound.
### 4. Distributional Validation
Validate:
- The beta law for static alignment.
- Capacity scaling across width, depth, rank, sparsity, and threshold.
- Redundancy-exhaustion transition.
- Relationship between capacity proxies and trajectory-level FA/BP gaps.
## Scope Control
Initial scope:
- MLPs.
- Dense FA first.
- Gaussian or isotropic feedback first.
- Synthetic regression and simple classification benchmarks.
Defer:
- CNNs, Transformers, BatchNorm, residual networks.
- Strong claims about final test accuracy.
- Biologically detailed interpretations.
|
