diff options
Diffstat (limited to 'notes/07_minimal_transition_theory.md')
| -rw-r--r-- | notes/07_minimal_transition_theory.md | 104 |
1 files changed, 104 insertions, 0 deletions
diff --git a/notes/07_minimal_transition_theory.md b/notes/07_minimal_transition_theory.md new file mode 100644 index 0000000..ced7e61 --- /dev/null +++ b/notes/07_minimal_transition_theory.md @@ -0,0 +1,104 @@ +# Minimal Transition Theory + +The cleanest correction is not to add a separate FA burden coefficient first. +Start by calibrating the task dimension from BP alone. + +## Problem + +The hard theory used: + +\[ +d_{\mathrm{task}}=Nn_L=512. +\] + +With this value, the smooth theory peak is near: + +\[ +M_{\mathrm{FA}}\approx -273, +\] + +while 256 empirical FA trajectories peak at: + +\[ +M_{\mathrm{FA}}=-318. +\] + +## BP-Only Calibration + +Fit the BP train curve with: + +\[ +L_{\mathrm{BP}}(P) +\approx +s\cdot \operatorname{softplus}_{\tau}(d_{\mathrm{task}}^{\mathrm{eff}}-P). +\] + +The BP-only fit gives: + +\[ +d_{\mathrm{task}}^{\mathrm{eff}}\approx 437, +\qquad +\tau\approx 50. +\] + +So BP is stronger than the naive parameter count suggests: + +\[ +d_{\mathrm{task}}^{\mathrm{eff}} +\approx +0.85\,Nn_L. +\] + +## Minimal FA Prediction + +Use the same hard FA burden: + +\[ +K_{\mathrm{FA}}=\sum_l(D_l-1), +\] + +but replace the task dimension: + +\[ +M_{\mathrm{FA}}^{\mathrm{min}} += +P-K_{\mathrm{FA}}-d_{\mathrm{task}}^{\mathrm{eff}}. +\] + +Equivalently, when plotted on the original hard-margin axis + +\[ +P-K_{\mathrm{FA}}-Nn_L, +\] + +the theory shifts left by: + +\[ +Nn_L-d_{\mathrm{task}}^{\mathrm{eff}} +\approx +75. +\] + +This explains the observed left shift without introducing a fitted FA burden +coefficient. + +Generated figure: + +`outputs/downstream_capacity_random_main_fast/capacity_transition_bp_calibrated_p2_256.png` + +## Recommendation + +Use this as the main downstream theory: + +1. Matrix/capacity theory remains exact and uses the hard/log-volume definitions. +2. Downstream performance uses a BP-calibrated effective task dimension. +3. FA capacity loss is then tested with no extra FA-specific fitting parameter. + +Only add an effective FA burden coefficient + +\[ +K_{\mathrm{FA}}^{\mathrm{eff}}=\alpha K_{\mathrm{FA}} +\] + +as an appendix ablation if needed. It improves fit but makes the theory look +more ad hoc. |
