Record 256-trajectory transition analysis

1 files changed, 87 insertions, 0 deletions

diff --git a/notes/06_capacity_transition_256_analysis.md b/notes/06_capacity_transition_256_analysis.md
new file mode 100644
index 0000000..dc75163
--- /dev/null
+++ b/notes/06_capacity_transition_256_analysis.md
@@ -0,0 +1,87 @@
+# Capacity Transition 256-Trajectory Analysis
+
+Figure:
+
+`outputs/downstream_capacity_random_main_fast/capacity_transition_empirical_only_p2_256.png`
+
+Empirical FA trajectory count:
+
+\[
+48+100+108=256.
+\]
+
+## Aggregate Means
+
+| FA margin | trajectories | mean gap | standard error |
+|---:|---:|---:|---:|
+| -382 | 256 | 0.358681 | 0.005990 |
+| -318 | 256 | 0.390980 | 0.006573 |
+| -254 | 256 | 0.269374 | 0.005120 |
+| -126 | 256 | 0.037187 | 0.001733 |
+| 2 | 256 | 0.009475 | 0.001225 |
+| 258 | 256 | 0.002675 | 0.000419 |
+| 514 | 256 | 0.002148 | 0.000489 |
+| 1026 | 256 | 0.001890 | 0.000396 |
+
+Empirical peak:
+
+\[
+M_{\mathrm{FA}}=-318.
+\]
+
+Smooth theory peak with transition width 70:
+
+\[
+M_{\mathrm{FA}}\approx -273.
+\]
+
+The empirical peak is therefore shifted left by roughly:
+
+\[
+-45
+\]
+
+scalar output constraints. With 256 trajectories and small standard errors, this
+is unlikely to be only visual sampling noise.
+
+## Likely Theory Gaps
+
+1. The BP baseline is probably too weak. The crude threshold \(P\ge Nn_L\) treats
+the network like a linear parameterization, while a ReLU MLP can start reducing
+random-label train loss earlier through gates, nonlinear features, and optimizer
+bias.
+
+2. The FA burden is probably too hard. \(K_{\mathrm{FA}}=\sum_l(D_l-1)\) treats
+alignment as a hard matrix-direction constraint, but FA training only induces
+partial dynamical alignment. The effective burden should be a soft/effective
+rank, not the full hard rank.
+
+3. The current theory ignores ReLU gate activity. Effective matrix dimensions
+should depend on active units and path availability, especially at small widths.
+
+4. The current theory ignores optimization time and optimizer dynamics. Adam and
+finite training steps can shift apparent transition locations even when the
+static capacity count is correct.
+
+5. Width is sampled coarsely. The empirical peak is observed at discrete margins
+\(-382,-318,-254,\dots\), so the true empirical maximum could lie between
+\(-382\) and \(-254\).
+
+The next theoretical correction should introduce two effective quantities:
+
+\[
+M_{\mathrm{BP}}^{\mathrm{eff}}
+=
+P-d_{\mathrm{task}}^{\mathrm{eff}},
+\]
+
+and
+
+\[
+M_{\mathrm{FA}}^{\mathrm{eff}}
+=
+P-K_{\mathrm{FA}}^{\mathrm{eff}}-d_{\mathrm{task}}^{\mathrm{eff}},
+\]
+
+where \(d_{\mathrm{task}}^{\mathrm{eff}}\le Nn_L\) and
+\(K_{\mathrm{FA}}^{\mathrm{eff}}\le K_{\mathrm{FA}}\).