1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
|
# Phase Transition Visualization
Status: superseded by the dense long-training sweep in
`notes/22_dense_phase_transition_soft_ramp.md` and the theory revision in
`notes/23_soft_capacity_theory_revision.md`.
This note records an earlier hard-exhaustion framing. The current interpretation
is soft capacity erosion, not a sharp phase transition.
This note records the first clean phase-transition visualization for the
capacity-exhaustion contribution.
## Goal
We want a figure that shows:
```text
available FA capacity decreases
-> redundant directions are exhausted
-> FA/BP train gap opens
-> gap grows while BP remains capable
```
The key x-axis is the hard FA capacity margin:
```text
M_FA = P - K_FA - N*out
```
where:
- `P` is the parameter count;
- `K_FA` is the hard feedback-alignment constraint count;
- `N*out` is the random-label task dimension.
The predicted transition is at:
```text
M_FA = 0
```
## Experiment
This run uses SGD, not Adam.
Configuration:
```text
task: random-label regression
architecture: 16 -> width -> width -> 4
widths: 8, 12, 16, 24, 32, 48, 64, 96
train samples: 128
optimizer: full-batch SGD
learning rate: 0.01
steps: 3000
init seeds: 4
feedback seeds per init: 8
FA trajectories: 256
```
Output directory:
```text
outputs/phase_transition_sgd_256_lr001_T3000
```
Main figures:
```text
outputs/phase_transition_sgd_256_lr001_T3000/phase_transition_capacity_exhaustion.png
outputs/phase_transition_sgd_256_lr001_T3000/phase_transition_gap_only.png
```
## Result
Summary:
| width | FA margin | BP train | FA train | FA-BP train gap |
|---:|---:|---:|---:|---:|
| 96 | 1026 | 0.000249 | 0.009436 | 0.009187 |
| 64 | 514 | 0.003354 | 0.053192 | 0.049838 |
| 48 | 258 | 0.016732 | 0.146680 | 0.129947 |
| 32 | 2 | 0.089511 | 0.386777 | 0.297266 |
| 24 | -126 | 0.201897 | 0.600796 | 0.398899 |
| 16 | -254 | 0.547408 | 0.993219 | 0.445812 |
| 12 | -318 | 0.872346 | 1.204686 | 0.332340 |
| 8 | -382 | 1.260234 | 1.443789 | 0.183555 |
The gap is small in the redundant region:
```text
M_FA > 0
```
It rises sharply around the hard FA capacity boundary:
```text
M_FA ≈ 0
```
and becomes largest in the FA-deficient but BP-capable region:
```text
M_FA < 0
and
M_BP = P - N*out > 0
```
At the smallest widths, BP itself also becomes under-capacity:
```text
M_BP < 0
```
In that both-deficient regime, the FA-BP gap decreases because both methods
fail to fit the random labels.
## Interpretation
The visual story is not simply "smaller network means larger gap." The clean
phase-transition story is:
1. high redundancy:
```text
M_FA >> 0
```
BP and FA both have enough effective room, so the train gap is small.
2. FA redundancy exhausted:
```text
M_FA crosses 0
```
FA starts paying the feedback-alignment burden in task-relevant directions,
so the gap opens.
3. FA-deficient but BP-capable:
```text
M_FA < 0, M_BP > 0
```
the gap grows and peaks.
4. both-deficient:
```text
M_BP < 0
```
BP also cannot memorize, so the difference between FA and BP no longer
grows monotonically.
This is the right phase-transition framing for the paper.
## Caveat
The hard margin is intentionally simple. It uses:
```text
K_FA = (width*width - 1) + (width*out - 1)
```
for the two-hidden-layer MLP. It predicts the transition location well enough
to provide a clean regime variable, but the exact peak is shifted because real
FA alignment is soft, not a hard rank constraint.
Do not claim that hard capacity margin alone predicts the exact final train
gap. The correct claim is:
```text
hard FA margin predicts the regime where the gap opens;
tangent/operator dynamics or empirical trajectories determine the gap size.
```
|
