1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
|
# More Phase-Transition Trajectories
We ran two larger trajectory sweeps around the capacity transition to separate
sampling noise from finite-time undertraining.
## Runs
```text
outputs/phase_transition_zoom_T10000_1024traj
outputs/phase_transition_zoom_T30000_256traj
```
The setup is unchanged:
```text
task: random-label regression
architecture: 16 -> width -> width -> 4
optimizer: full-batch SGD
learning rate: 0.01
train samples: 128
```
The `T=10000` run uses:
```text
widths: 16, 20, 24, 28, 32, 36, 40, 48
init seeds: 2
feedback seeds per init: 64
FA trajectories per width: 128
total FA trajectories: 1024
```
The `T=30000` run focuses on the transition:
```text
widths: 24, 28, 32, 36
init seeds: 2
feedback seeds per init: 32
FA trajectories per width: 64
total FA trajectories: 256
```
## Main Plots
```text
outputs/phase_transition_zoom_comparison/trajectory_gap_clouds_T10000_vs_T30000.png
outputs/phase_transition_zoom_comparison/trajectory_gap_quantiles_T10000_vs_T30000.png
```
Per-run plots:
```text
outputs/phase_transition_zoom_T10000_1024traj/phase_transition_capacity_exhaustion.png
outputs/phase_transition_zoom_T30000_256traj/phase_transition_capacity_exhaustion.png
```
## Results
At `T=10000`, the train-gap curve is very smooth and monotone:
| width | FA margin | FA trajectories | train gap mean | train gap std |
|---:|---:|---:|---:|---:|
| 16 | -254 | 128 | 0.420780 | 0.075399 |
| 20 | -190 | 128 | 0.278329 | 0.058114 |
| 24 | -126 | 128 | 0.171090 | 0.043111 |
| 28 | -62 | 128 | 0.099732 | 0.033015 |
| 32 | 2 | 128 | 0.050912 | 0.016008 |
| 36 | 66 | 128 | 0.030891 | 0.010649 |
| 40 | 130 | 128 | 0.014464 | 0.004946 |
| 48 | 258 | 128 | 0.004212 | 0.001836 |
This smooth ramp is not caused by too few feedback seeds. Each point has 128 FA
trajectories and the standard deviations are much smaller than the mean trend.
At `T=30000`, the near-zero and positive-margin gaps collapse:
| width | FA margin | FA trajectories | train gap mean | train gap std |
|---:|---:|---:|---:|---:|
| 24 | -126 | 64 | 0.047899 | 0.026889 |
| 28 | -62 | 64 | 0.015020 | 0.009012 |
| 32 | 2 | 64 | 0.003275 | 0.003577 |
| 36 | 66 | 64 | 0.000932 | 0.001324 |
So the earlier lack of a sharp kink was mostly finite-time undertraining near
the transition. With longer training, positive-margin networks close the BP/FA
train gap, while negative-margin networks retain a nonzero gap.
## Interpretation
The hard FA capacity margin is a conservative structural boundary. It is not an
exact discontinuous transition point.
The empirical picture is:
```text
T=10000: smooth finite-time ramp
T=30000: compressed transition; positive margin almost zero, negative margin nonzero
```
This supports the phase-transition contribution, but the right language is a
soft capacity transition rather than a hard kink at exactly margin zero.
## Consequence
For the paper figure, the clean visual is the two-panel comparison:
```text
trajectory_gap_clouds_T10000_vs_T30000.png
```
It shows:
1. More trajectories do not remove the smooth `T=10000` ramp.
2. Longer training does remove the apparent positive-margin gap.
3. The remaining long-time gap is concentrated on the negative-margin side.
This is the strongest current evidence that the original smooth curve mixed two
effects:
```text
finite-time optimization gap + structural capacity gap
```
|
