1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
|
# Scripts
## Static Alignment Beta Law
Run:
```bash
python scripts/static_alignment_beta.py --rows 16 --cols 16 --samples 20000 --seed 7 --plot
```
This samples independent matrix pairs \(A,B\), computes:
\[
Q =
\frac{\langle A,B\rangle_F^2}{\|A\|_F^2\|B\|_F^2},
\]
and compares the empirical distribution with:
\[
\mathrm{Beta}\left(\frac12,\frac{D-1}{2}\right),
\qquad
D=\texttt{rows}\times\texttt{cols}.
\]
Outputs are written under `outputs/static_alignment_beta/`, which is ignored by Git.
## Capacity Scaling
Run:
```bash
python scripts/capacity_scaling.py --plot
```
This computes:
\[
C_l(q)
=
-\log\Pr(Q_l\ge q)
\]
for equal-width feedback blocks with \(D=n^2\), then sums over the number of feedback-aligned layers:
\[
C_{\mathrm{all}}=\sum_l C_l(q_l).
\]
The default run compares two regimes:
- `fixed`: \(q=0.01\), where \(C_{\mathrm{all}}\) grows like \(\Theta(Ln^2)\).
- `chance`: \(q=1/D\), where \(C_{\mathrm{all}}\) grows mostly with \(L\).
Outputs are written under `outputs/capacity_scaling/`.
## Minimax Initialization Bound
Run:
```bash
python scripts/minimax_initialization.py --dimension 32 --feedback-samples 20000 --target-samples 10000 --seed 11 --subspace-dim 4 --plot
```
This estimates the feedback second-moment matrix:
\[
M_\mu=\mathbb E_\mu[\hat b\hat b^\top]
\]
for several initialization distributions. The worst-case expected squared alignment is:
\[
\inf_{\|a\|=1}
\mathbb E_\mu[(a^\top \hat b)^2]
=
\lambda_{\min}(M_\mu).
\]
The prior-free minimax theorem says:
\[
\sup_\mu \lambda_{\min}(M_\mu)=\frac1D,
\]
with equality for isotropic feedback. Outputs are written under `outputs/minimax_initialization/`.
## Functional Capacity Overlap
Run:
```bash
python scripts/functional_capacity_overlap.py --parameters 96 --task-rank 24 --constraint-ranks 0 24 48 72 84 96 --trials 100 --seed 5 --plot
```
This samples a task-sensitive subspace \(S\) of dimension \(d\) and an alignment
constraint subspace \(E\) of dimension \(k\) inside \(\mathbb R^P\). It validates:
\[
\Delta d_{\mathrm{hard}}
=
\max(0,k-(P-d))
\]
and:
\[
\mathbb E[\operatorname{tr}(P_E P_S)]
=
\frac{kd}{P}.
\]
Outputs are written under `outputs/functional_capacity_overlap/`.
## Synthetic MLP FA/BP Trajectories
Run:
```bash
python scripts/trajectory_mlp_fa.py --samples 128 --hidden-widths 24 24 --steps 80 --lr 0.02 --eval-every 10 --feedback-runs 3 --data-seed 3 --init-seed 4 --feedback-seed-start 50 --plot
```
This trains one BP baseline and several FA runs from the same initial weights on
a synthetic regression task. At each checkpoint, the script records:
- training loss;
- full-model cosine between the BP gradient and the FA surrogate gradient at the FA weights;
- layerwise \(Q_l=\cos^2(W_{l+1}^{\top},B_l)\).
Outputs are written under `outputs/trajectory_mlp_fa/`:
- `summary.csv`
- `trajectories.csv`
- `layer_metrics.csv`
- diagnostic plots when `--plot` is set.
|
