blob: ced7e61aeb0de88dfaac2cfea9e56e2d307161ec (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
|
# Minimal Transition Theory
The cleanest correction is not to add a separate FA burden coefficient first.
Start by calibrating the task dimension from BP alone.
## Problem
The hard theory used:
\[
d_{\mathrm{task}}=Nn_L=512.
\]
With this value, the smooth theory peak is near:
\[
M_{\mathrm{FA}}\approx -273,
\]
while 256 empirical FA trajectories peak at:
\[
M_{\mathrm{FA}}=-318.
\]
## BP-Only Calibration
Fit the BP train curve with:
\[
L_{\mathrm{BP}}(P)
\approx
s\cdot \operatorname{softplus}_{\tau}(d_{\mathrm{task}}^{\mathrm{eff}}-P).
\]
The BP-only fit gives:
\[
d_{\mathrm{task}}^{\mathrm{eff}}\approx 437,
\qquad
\tau\approx 50.
\]
So BP is stronger than the naive parameter count suggests:
\[
d_{\mathrm{task}}^{\mathrm{eff}}
\approx
0.85\,Nn_L.
\]
## Minimal FA Prediction
Use the same hard FA burden:
\[
K_{\mathrm{FA}}=\sum_l(D_l-1),
\]
but replace the task dimension:
\[
M_{\mathrm{FA}}^{\mathrm{min}}
=
P-K_{\mathrm{FA}}-d_{\mathrm{task}}^{\mathrm{eff}}.
\]
Equivalently, when plotted on the original hard-margin axis
\[
P-K_{\mathrm{FA}}-Nn_L,
\]
the theory shifts left by:
\[
Nn_L-d_{\mathrm{task}}^{\mathrm{eff}}
\approx
75.
\]
This explains the observed left shift without introducing a fitted FA burden
coefficient.
Generated figure:
`outputs/downstream_capacity_random_main_fast/capacity_transition_bp_calibrated_p2_256.png`
## Recommendation
Use this as the main downstream theory:
1. Matrix/capacity theory remains exact and uses the hard/log-volume definitions.
2. Downstream performance uses a BP-calibrated effective task dimension.
3. FA capacity loss is then tested with no extra FA-specific fitting parameter.
Only add an effective FA burden coefficient
\[
K_{\mathrm{FA}}^{\mathrm{eff}}=\alpha K_{\mathrm{FA}}
\]
as an appendix ablation if needed. It improves fit but makes the theory look
more ad hoc.
|
