1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
|
# Downstream Random-Capacity Results
## Main Run
Output directory:
`outputs/downstream_capacity_random_main_fast`
Task:
\[
x_i\sim \mathcal N(0,I_{16}),
\qquad
y_i\sim \mathcal N(0,I_4),
\qquad
N=128.
\]
Task dimension:
\[
N n_L=512.
\]
Model:
\[
16\to n\to n\to 4.
\]
Theory:
\[
P(n)=16n+n^2+4n,
\]
\[
K_{\mathrm{FA}}(n)=(n^2-1)+(4n-1),
\]
\[
M_{\mathrm{BP}}=P-Nn_L,
\qquad
M_{\mathrm{FA}}=P-K_{\mathrm{FA}}-Nn_L.
\]
Prediction:
- BP threshold at width \(16\).
- FA threshold at width \(32\).
Trajectory count:
- BP: \(4\) seeds per width.
- FA: \(4\times 12=48\) trajectories per width.
- Widths: \(8,12,16,24,32,48,64,96\).
## Summary
| width | BP train | FA train | gap | BP margin | FA margin |
|---:|---:|---:|---:|---:|---:|
| 8 | 0.7088 | 1.064 | 0.3555 | -288 | -382 |
| 12 | 0.2252 | 0.6141 | 0.3889 | -128 | -318 |
| 16 | 0.00743 | 0.2948 | 0.2873 | 64 | -254 |
| 24 | 0.000129 | 0.0346 | 0.0345 | 544 | -126 |
| 32 | 0.000007 | 0.0108 | 0.0108 | 1152 | 2 |
| 48 | 0.000007 | 0.00392 | 0.00391 | 2752 | 258 |
| 64 | 0.000046 | 0.00189 | 0.00185 | 4864 | 514 |
| 96 | 0.000250 | 0.00231 | 0.00206 | 10624 | 1026 |
## Main Figures
- `outputs/downstream_capacity_random_main_fast/train_mse_trajectory_distribution.png`
- `outputs/downstream_capacity_random_main_fast/train_gap_trajectory_distribution_vs_fa_margin.png`
- `outputs/downstream_capacity_random_main_fast/bp_fa_train_mse_vs_width.png`
- `outputs/downstream_capacity_random_main_fast/train_gap_vs_fa_capacity_margin.png`
The cleanest figure is `train_gap_trajectory_distribution_vs_fa_margin.png`: the FA/BP train gap is large when \(M_{\mathrm{FA}}<0\), drops sharply near \(M_{\mathrm{FA}}=0\), and remains small for positive FA margin.
|
