1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
|
# Capacity Transition 256-Trajectory Analysis
Figure:
`outputs/downstream_capacity_random_main_fast/capacity_transition_empirical_only_p2_256.png`
Empirical FA trajectory count:
\[
48+100+108=256.
\]
## Aggregate Means
| FA margin | trajectories | mean gap | standard error |
|---:|---:|---:|---:|
| -382 | 256 | 0.358681 | 0.005990 |
| -318 | 256 | 0.390980 | 0.006573 |
| -254 | 256 | 0.269374 | 0.005120 |
| -126 | 256 | 0.037187 | 0.001733 |
| 2 | 256 | 0.009475 | 0.001225 |
| 258 | 256 | 0.002675 | 0.000419 |
| 514 | 256 | 0.002148 | 0.000489 |
| 1026 | 256 | 0.001890 | 0.000396 |
Empirical peak:
\[
M_{\mathrm{FA}}=-318.
\]
Smooth theory peak with transition width 70:
\[
M_{\mathrm{FA}}\approx -273.
\]
The empirical peak is therefore shifted left by roughly:
\[
-45
\]
scalar output constraints. With 256 trajectories and small standard errors, this
is unlikely to be only visual sampling noise.
## Likely Theory Gaps
1. The BP baseline is probably too weak. The crude threshold \(P\ge Nn_L\) treats
the network like a linear parameterization, while a ReLU MLP can start reducing
random-label train loss earlier through gates, nonlinear features, and optimizer
bias.
2. The FA burden is probably too hard. \(K_{\mathrm{FA}}=\sum_l(D_l-1)\) treats
alignment as a hard matrix-direction constraint, but FA training only induces
partial dynamical alignment. The effective burden should be a soft/effective
rank, not the full hard rank.
3. The current theory ignores ReLU gate activity. Effective matrix dimensions
should depend on active units and path availability, especially at small widths.
4. The current theory ignores optimization time and optimizer dynamics. Adam and
finite training steps can shift apparent transition locations even when the
static capacity count is correct.
5. Width is sampled coarsely. The empirical peak is observed at discrete margins
\(-382,-318,-254,\dots\), so the true empirical maximum could lie between
\(-382\) and \(-254\).
The next theoretical correction should introduce two effective quantities:
\[
M_{\mathrm{BP}}^{\mathrm{eff}}
=
P-d_{\mathrm{task}}^{\mathrm{eff}},
\]
and
\[
M_{\mathrm{FA}}^{\mathrm{eff}}
=
P-K_{\mathrm{FA}}^{\mathrm{eff}}-d_{\mathrm{task}}^{\mathrm{eff}},
\]
where \(d_{\mathrm{task}}^{\mathrm{eff}}\le Nn_L\) and
\(K_{\mathrm{FA}}^{\mathrm{eff}}\le K_{\mathrm{FA}}\).
|
