1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
|
# Experiment Notes
## Experiment Layer 1: Static Distribution Validation
Purpose: verify the beta law before training dynamics enter.
Procedure:
1. Choose matrix shape \(n_l\times n_{l+1}\), so \(D_l=n_l n_{l+1}\).
2. Sample \(A_l\) and \(B_l\) independently from isotropic distributions.
3. Compute:
\[
Q_l=
\frac{
\langle A_l,B_l\rangle_F^2
}{
\|A_l\|_F^2\|B_l\|_F^2
}.
\]
4. Compare empirical distribution to:
\[
\mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right).
\]
Metrics:
- Histogram overlay.
- QQ plot.
- KS statistic.
- Tail calibration:
\[
\Pr(Q_l\ge q).
\]
Initialization variants:
- Gaussian.
- Rademacher.
- Uniform sphere.
- Orthogonal or semi-orthogonal.
- Sparse.
- Low-rank.
- Block-diagonal.
Expected:
- Dense isotropic variants match beta law.
- Structured variants deviate in predictable ways.
## Experiment Layer 2: Scaling Validation
Purpose: verify capacity scaling with depth and width.
Compute:
\[
C_l(q)=
-\log
\left[
1-I_q\left(\frac12,\frac{D_l-1}{2}\right)
\right].
\]
Total:
\[
C_{\mathrm{all}}=\sum_l C_l(q_l).
\]
Sweeps:
- Width \(n\).
- Depth \(L\).
- Threshold \(q\).
- Threshold regime \(q=c/D_l\).
- Feedback rank.
- Feedback sparsity.
Predictions:
Fixed \(q\):
\[
C_{\mathrm{all}}=\Theta(Ln^2)
\]
for equal-width MLPs.
Chance-level \(q=c/D_l\):
\[
C_{\mathrm{all}}=\Theta(L).
\]
## Experiment Layer 3: Local Gradient Alignment
Purpose: connect static matrix alignment to update direction mismatch.
During training, record:
\[
\Gamma_t=
\frac{
\langle g_t^{\mathrm{BP}},g_t^{\mathrm{FA}}\rangle
}{
\|g_t^{\mathrm{BP}}\|\|g_t^{\mathrm{FA}}\|
}.
\]
Layerwise:
\[
\Gamma_{l,t}=
\frac{
\langle g_{l,t}^{\mathrm{BP}},g_{l,t}^{\mathrm{FA}}\rangle
}{
\|g_{l,t}^{\mathrm{BP}}\|\|g_{l,t}^{\mathrm{FA}}\|
}.
\]
Also record:
\[
Q_l(t)=\cos^2(W_{l+1}(t)^\top,B_l).
\]
Questions:
- Does \(Q_l(0)\) match the beta baseline?
- Does \(Q_l(t)\) shift right during the alignment phase?
- Does gradient alignment improve before memorization or loss reduction?
## Experiment Layer 4: Trajectory Ensemble
Purpose: validate whether capacity proxies explain FA/BP training gaps.
For each architecture and dataset:
1. Fix data seed and model architecture.
2. Train BP baseline.
3. Train many FA runs over feedback seeds \(B\).
4. Record:
\[
\Delta L_T(B)=L_T^{\mathrm{FA}}(B)-L_T^{\mathrm{BP}},
\]
\[
\Delta A_T(B)=A_T^{\mathrm{BP}}-A_T^{\mathrm{FA}},
\]
\[
C_{\mathrm{all}}(B,t),
\quad
\Gamma_t(B),
\quad
Q_l(t).
\]
Datasets:
- Synthetic Gaussian regression.
- MNIST MLP.
- Fashion-MNIST MLP.
- CIFAR-10 flattened MLP, optional later.
Architectures:
- Equal-width MLPs.
- Width sweep.
- Depth sweep.
- Narrow bottleneck sweep.
Expected:
- Overparameterized regimes: large parameter-volume cost can coexist with small functional gap.
- Near redundancy exhaustion: FA/BP gap should increase sharply.
- Poor feedback conditioning can worsen trajectory gap even when angular minimax bound is unchanged.
## Plots
Static:
- Histogram and beta density.
- QQ plot.
- Tail probability calibration.
Scaling:
- \(C_{\mathrm{all}}\) vs \(Ln^2\).
- \(-\log p_{\mathrm{all}}\) vs depth.
- Scaling collapse for \(D_lQ_l\Rightarrow \chi_1^2\).
Trajectory:
- \(Q_l(t)\) over training.
- \(\Gamma_t\) over training.
- \(\Delta L_T\) vs capacity proxy.
- \(\Delta L_T\) vs conditioning proxy.
- Phase transition plot against \(k-(P-d)\).
## Implementation Notes
Start with NumPy or PyTorch scripts that do not require full training.
First script target:
- Sample \(A,B\).
- Compute \(Q\).
- Save empirical moments and KS statistic.
- Produce beta overlay plots.
Only after this is clean, add FA/BP training loops.
|
