From 6d8be33a7fb9547c3f035bec0476eabcec87ecdd Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Fri, 5 Jun 2026 12:32:03 -0500 Subject: Add phase transition trajectory zoom analysis --- notes/21_more_phase_transition_trajectories.md | 122 +++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 notes/21_more_phase_transition_trajectories.md (limited to 'notes') diff --git a/notes/21_more_phase_transition_trajectories.md b/notes/21_more_phase_transition_trajectories.md new file mode 100644 index 0000000..69bc4ff --- /dev/null +++ b/notes/21_more_phase_transition_trajectories.md @@ -0,0 +1,122 @@ +# More Phase-Transition Trajectories + +We ran two larger trajectory sweeps around the capacity transition to separate +sampling noise from finite-time undertraining. + +## Runs + +```text +outputs/phase_transition_zoom_T10000_1024traj +outputs/phase_transition_zoom_T30000_256traj +``` + +The setup is unchanged: + +```text +task: random-label regression +architecture: 16 -> width -> width -> 4 +optimizer: full-batch SGD +learning rate: 0.01 +train samples: 128 +``` + +The `T=10000` run uses: + +```text +widths: 16, 20, 24, 28, 32, 36, 40, 48 +init seeds: 2 +feedback seeds per init: 64 +FA trajectories per width: 128 +total FA trajectories: 1024 +``` + +The `T=30000` run focuses on the transition: + +```text +widths: 24, 28, 32, 36 +init seeds: 2 +feedback seeds per init: 32 +FA trajectories per width: 64 +total FA trajectories: 256 +``` + +## Main Plots + +```text +outputs/phase_transition_zoom_comparison/trajectory_gap_clouds_T10000_vs_T30000.png +outputs/phase_transition_zoom_comparison/trajectory_gap_quantiles_T10000_vs_T30000.png +``` + +Per-run plots: + +```text +outputs/phase_transition_zoom_T10000_1024traj/phase_transition_capacity_exhaustion.png +outputs/phase_transition_zoom_T30000_256traj/phase_transition_capacity_exhaustion.png +``` + +## Results + +At `T=10000`, the train-gap curve is very smooth and monotone: + +| width | FA margin | FA trajectories | train gap mean | train gap std | +|---:|---:|---:|---:|---:| +| 16 | -254 | 128 | 0.420780 | 0.075399 | +| 20 | -190 | 128 | 0.278329 | 0.058114 | +| 24 | -126 | 128 | 0.171090 | 0.043111 | +| 28 | -62 | 128 | 0.099732 | 0.033015 | +| 32 | 2 | 128 | 0.050912 | 0.016008 | +| 36 | 66 | 128 | 0.030891 | 0.010649 | +| 40 | 130 | 128 | 0.014464 | 0.004946 | +| 48 | 258 | 128 | 0.004212 | 0.001836 | + +This smooth ramp is not caused by too few feedback seeds. Each point has 128 FA +trajectories and the standard deviations are much smaller than the mean trend. + +At `T=30000`, the near-zero and positive-margin gaps collapse: + +| width | FA margin | FA trajectories | train gap mean | train gap std | +|---:|---:|---:|---:|---:| +| 24 | -126 | 64 | 0.047899 | 0.026889 | +| 28 | -62 | 64 | 0.015020 | 0.009012 | +| 32 | 2 | 64 | 0.003275 | 0.003577 | +| 36 | 66 | 64 | 0.000932 | 0.001324 | + +So the earlier lack of a sharp kink was mostly finite-time undertraining near +the transition. With longer training, positive-margin networks close the BP/FA +train gap, while negative-margin networks retain a nonzero gap. + +## Interpretation + +The hard FA capacity margin is a conservative structural boundary. It is not an +exact discontinuous transition point. + +The empirical picture is: + +```text +T=10000: smooth finite-time ramp +T=30000: compressed transition; positive margin almost zero, negative margin nonzero +``` + +This supports the phase-transition contribution, but the right language is a +soft capacity transition rather than a hard kink at exactly margin zero. + +## Consequence + +For the paper figure, the clean visual is the two-panel comparison: + +```text +trajectory_gap_clouds_T10000_vs_T30000.png +``` + +It shows: + +1. More trajectories do not remove the smooth `T=10000` ramp. +2. Longer training does remove the apparent positive-margin gap. +3. The remaining long-time gap is concentrated on the negative-margin side. + +This is the strongest current evidence that the original smooth curve mixed two +effects: + +```text +finite-time optimization gap + structural capacity gap +``` -- cgit v1.2.3