From 0554474f407c3b5661290547cb529c96031a6c09 Mon Sep 17 00:00:00 2001
From: YurenHao0426 <Blackhao0426@gmail.com>
Date: Fri, 5 Jun 2026 11:13:03 -0500
Subject: Analyze phase transition training time

---
 notes/20_phase_transition_training_time.md | 124 +++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)
 create mode 100644 notes/20_phase_transition_training_time.md

(limited to 'notes')

diff --git a/notes/20_phase_transition_training_time.md b/notes/20_phase_transition_training_time.md
new file mode 100644
index 0000000..67bd5f4
--- /dev/null
+++ b/notes/20_phase_transition_training_time.md
@@ -0,0 +1,124 @@
+# Training Time and the Apparent Phase Transition
+
+The first SGD phase-transition figure looked more like a smooth ramp than a
+sharp kink. We tested whether this is caused by undertraining.
+
+## Setup
+
+Baseline:
+
+```text
+outputs/phase_transition_sgd_256_lr001_T3000
+```
+
+Longer probes:
+
+```text
+outputs/phase_transition_sgd_long_lr001_T10000_probe
+outputs/phase_transition_sgd_verylong_lr001_T30000_probe
+```
+
+All use:
+
+```text
+task: random-label regression
+architecture: 16 -> width -> width -> 4
+optimizer: full-batch SGD
+learning rate: 0.01
+train samples: 128
+```
+
+Comparison figures:
+
+```text
+outputs/phase_transition_training_time_comparison/train_gap_vs_margin_by_training_time.png
+outputs/phase_transition_training_time_comparison/bp_fa_train_loss_by_training_time.png
+```
+
+## Observation
+
+At `T=3000`, the gap appears to grow smoothly as hard FA capacity margin
+decreases.
+
+At longer training times, the apparent transition shifts left:
+
+| width | FA margin | gap T=3000 | gap T=10000 | gap T=30000 |
+|---:|---:|---:|---:|---:|
+| 96 | 1026 | 0.009187 | 0.000008 | |
+| 64 | 514 | 0.049838 | 0.000470 | |
+| 48 | 258 | 0.129947 | 0.005684 | |
+| 32 | 2 | 0.297266 | 0.047005 | 0.002913 |
+| 24 | -126 | 0.398899 | 0.151777 | 0.044436 |
+| 16 | -254 | 0.445812 | 0.407483 | 0.304774 |
+
+So the `T=3000` ramp is partly finite-time optimization. With more training,
+positive-margin networks mostly close the FA/BP train gap.
+
+## Interpretation
+
+The hard margin
+
+```text
+M_FA = P - K_FA - N*out
+```
+
+is a conservative structural regime variable, not a sharp empirical threshold.
+
+The actual transition is shifted to more negative margins because:
+
+1. FA alignment is soft, not a hard rank constraint.
+2. Overparameterized networks can absorb some alignment burden through
+   redundancy and nonlinear gates.
+3. Finite-time optimization can make the transition look smoother and earlier
+   than the asymptotic training limit.
+
+Thus:
+
+```text
+T=3000: optimization-limited ramp
+T=10000/T=30000: sharper and shifted-left capacity transition
+```
+
+The current evidence suggests the actual long-time FA train-gap transition lies
+between:
+
+```text
+M_FA = -126
+and
+M_FA = -254
+```
+
+for this task/architecture, rather than exactly at hard margin zero.
+
+## Consequence for the Paper
+
+Do not present the hard-margin line as an exact transition point.
+
+Better wording:
+
+```text
+The hard FA margin gives a conservative capacity-exhaustion boundary. Empirical
+train gaps begin to grow as the margin decreases, but the observed transition is
+soft and training-time dependent. Longer optimization shifts the effective
+transition toward more negative margins, consistent with a soft rather than hard
+alignment burden.
+```
+
+The phase-transition contribution is still valid, but it should be framed as a
+soft transition / regime shift, not a discontinuous kink.
+
+## Next Better Visualization
+
+The most honest visual is the training-time comparison:
+
+```text
+train_gap_vs_margin_by_training_time.png
+```
+
+It shows both facts:
+
+1. finite-time training produces a smooth ramp;
+2. longer training pushes the transition toward the true capacity-limited
+   region.
+
+This is stronger than pretending the `T=3000` curve has a sharp kink.
-- 
cgit v1.2.3