blob: 67bd5f4594a07b140c8412711d6885b07de8d329 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
|
# Training Time and the Apparent Phase Transition
The first SGD phase-transition figure looked more like a smooth ramp than a
sharp kink. We tested whether this is caused by undertraining.
## Setup
Baseline:
```text
outputs/phase_transition_sgd_256_lr001_T3000
```
Longer probes:
```text
outputs/phase_transition_sgd_long_lr001_T10000_probe
outputs/phase_transition_sgd_verylong_lr001_T30000_probe
```
All use:
```text
task: random-label regression
architecture: 16 -> width -> width -> 4
optimizer: full-batch SGD
learning rate: 0.01
train samples: 128
```
Comparison figures:
```text
outputs/phase_transition_training_time_comparison/train_gap_vs_margin_by_training_time.png
outputs/phase_transition_training_time_comparison/bp_fa_train_loss_by_training_time.png
```
## Observation
At `T=3000`, the gap appears to grow smoothly as hard FA capacity margin
decreases.
At longer training times, the apparent transition shifts left:
| width | FA margin | gap T=3000 | gap T=10000 | gap T=30000 |
|---:|---:|---:|---:|---:|
| 96 | 1026 | 0.009187 | 0.000008 | |
| 64 | 514 | 0.049838 | 0.000470 | |
| 48 | 258 | 0.129947 | 0.005684 | |
| 32 | 2 | 0.297266 | 0.047005 | 0.002913 |
| 24 | -126 | 0.398899 | 0.151777 | 0.044436 |
| 16 | -254 | 0.445812 | 0.407483 | 0.304774 |
So the `T=3000` ramp is partly finite-time optimization. With more training,
positive-margin networks mostly close the FA/BP train gap.
## Interpretation
The hard margin
```text
M_FA = P - K_FA - N*out
```
is a conservative structural regime variable, not a sharp empirical threshold.
The actual transition is shifted to more negative margins because:
1. FA alignment is soft, not a hard rank constraint.
2. Overparameterized networks can absorb some alignment burden through
redundancy and nonlinear gates.
3. Finite-time optimization can make the transition look smoother and earlier
than the asymptotic training limit.
Thus:
```text
T=3000: optimization-limited ramp
T=10000/T=30000: sharper and shifted-left capacity transition
```
The current evidence suggests the actual long-time FA train-gap transition lies
between:
```text
M_FA = -126
and
M_FA = -254
```
for this task/architecture, rather than exactly at hard margin zero.
## Consequence for the Paper
Do not present the hard-margin line as an exact transition point.
Better wording:
```text
The hard FA margin gives a conservative capacity-exhaustion boundary. Empirical
train gaps begin to grow as the margin decreases, but the observed transition is
soft and training-time dependent. Longer optimization shifts the effective
transition toward more negative margins, consistent with a soft rather than hard
alignment burden.
```
The phase-transition contribution is still valid, but it should be framed as a
soft transition / regime shift, not a discontinuous kink.
## Next Better Visualization
The most honest visual is the training-time comparison:
```text
train_gap_vs_margin_by_training_time.png
```
It shows both facts:
1. finite-time training produces a smooth ramp;
2. longer training pushes the transition toward the true capacity-limited
region.
This is stronger than pretending the `T=3000` curve has a sharp kink.
|
