Analyze phase transition training time

2 files changed, 210 insertions, 0 deletions

diff --git a/notes/20_phase_transition_training_time.md b/notes/20_phase_transition_training_time.md
new file mode 100644
index 0000000..67bd5f4
--- /dev/null
+++ b/notes/20_phase_transition_training_time.md
@@ -0,0 +1,124 @@
+# Training Time and the Apparent Phase Transition
+
+The first SGD phase-transition figure looked more like a smooth ramp than a
+sharp kink. We tested whether this is caused by undertraining.
+
+## Setup
+
+Baseline:
+
+```text
+outputs/phase_transition_sgd_256_lr001_T3000
+```
+
+Longer probes:
+
+```text
+outputs/phase_transition_sgd_long_lr001_T10000_probe
+outputs/phase_transition_sgd_verylong_lr001_T30000_probe
+```
+
+All use:
+
+```text
+task: random-label regression
+architecture: 16 -> width -> width -> 4
+optimizer: full-batch SGD
+learning rate: 0.01
+train samples: 128
+```
+
+Comparison figures:
+
+```text
+outputs/phase_transition_training_time_comparison/train_gap_vs_margin_by_training_time.png
+outputs/phase_transition_training_time_comparison/bp_fa_train_loss_by_training_time.png
+```
+
+## Observation
+
+At `T=3000`, the gap appears to grow smoothly as hard FA capacity margin
+decreases.
+
+At longer training times, the apparent transition shifts left:
+
+| width | FA margin | gap T=3000 | gap T=10000 | gap T=30000 |
+|---:|---:|---:|---:|---:|
+| 96 | 1026 | 0.009187 | 0.000008 | |
+| 64 | 514 | 0.049838 | 0.000470 | |
+| 48 | 258 | 0.129947 | 0.005684 | |
+| 32 | 2 | 0.297266 | 0.047005 | 0.002913 |
+| 24 | -126 | 0.398899 | 0.151777 | 0.044436 |
+| 16 | -254 | 0.445812 | 0.407483 | 0.304774 |
+
+So the `T=3000` ramp is partly finite-time optimization. With more training,
+positive-margin networks mostly close the FA/BP train gap.
+
+## Interpretation
+
+The hard margin
+
+```text
+M_FA = P - K_FA - N*out
+```
+
+is a conservative structural regime variable, not a sharp empirical threshold.
+
+The actual transition is shifted to more negative margins because:
+
+1. FA alignment is soft, not a hard rank constraint.
+2. Overparameterized networks can absorb some alignment burden through
+   redundancy and nonlinear gates.
+3. Finite-time optimization can make the transition look smoother and earlier
+   than the asymptotic training limit.
+
+Thus:
+
+```text
+T=3000: optimization-limited ramp
+T=10000/T=30000: sharper and shifted-left capacity transition
+```
+
+The current evidence suggests the actual long-time FA train-gap transition lies
+between:
+
+```text
+M_FA = -126
+and
+M_FA = -254
+```
+
+for this task/architecture, rather than exactly at hard margin zero.
+
+## Consequence for the Paper
+
+Do not present the hard-margin line as an exact transition point.
+
+Better wording:
+
+```text
+The hard FA margin gives a conservative capacity-exhaustion boundary. Empirical
+train gaps begin to grow as the margin decreases, but the observed transition is
+soft and training-time dependent. Longer optimization shifts the effective
+transition toward more negative margins, consistent with a soft rather than hard
+alignment burden.
+```
+
+The phase-transition contribution is still valid, but it should be framed as a
+soft transition / regime shift, not a discontinuous kink.
+
+## Next Better Visualization
+
+The most honest visual is the training-time comparison:
+
+```text
+train_gap_vs_margin_by_training_time.png
+```
+
+It shows both facts:
+
+1. finite-time training produces a smooth ramp;
+2. longer training pushes the transition toward the true capacity-limited
+   region.
+
+This is stronger than pretending the `T=3000` curve has a sharp kink.
diff --git a/scripts/plot_phase_transition_training_time.py b/scripts/plot_phase_transition_training_time.py
new file mode 100644
index 0000000..e51eaea
--- /dev/null
+++ b/scripts/plot_phase_transition_training_time.py
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Compare capacity-transition curves at different SGD horizons."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import pandas as pd
+
+
+RUNS = [
+    ("T=3000, 256 traj", Path("outputs/phase_transition_sgd_256_lr001_T3000")),
+    ("T=10000 probe", Path("outputs/phase_transition_sgd_long_lr001_T10000_probe")),
+    ("T=30000 probe", Path("outputs/phase_transition_sgd_verylong_lr001_T30000_probe")),
+]
+
+
+def main() -> None:
+    outdir = Path("outputs/phase_transition_training_time_comparison")
+    outdir.mkdir(parents=True, exist_ok=True)
+
+    fig, ax = plt.subplots(figsize=(9.2, 5.7), dpi=180)
+    for label, run_dir in RUNS:
+        df = pd.read_csv(run_dir / "width_summary.csv").sort_values(
+            "fa_capacity_margin", ascending=False
+        )
+        ax.plot(
+            df["fa_capacity_margin"],
+            df["train_gap_mean"],
+            marker="o",
+            linewidth=2.0,
+            label=label,
+        )
+    ax.axvline(0, color="black", linestyle="--", linewidth=1.1, label="hard FA margin=0")
+    ax.axhline(0, color="black", linewidth=1.0)
+    ax.set_xlim(1100, -430)
+    ax.set_xlabel("hard FA capacity margin P - K_FA - N*out (capacity decreases ->)")
+    ax.set_ylabel("FA train MSE - BP train MSE")
+    ax.set_title("Training time shifts the apparent FA capacity transition")
+    ax.legend(frameon=True)
+    ax.grid(alpha=0.18)
+    fig.tight_layout()
+    path = outdir / "train_gap_vs_margin_by_training_time.png"
+    fig.savefig(path)
+    plt.close(fig)
+
+    fig2, ax2 = plt.subplots(figsize=(9.2, 5.7), dpi=180)
+    for label, run_dir in RUNS:
+        df = pd.read_csv(run_dir / "width_summary.csv").sort_values(
+            "fa_capacity_margin", ascending=False
+        )
+        ax2.plot(
+            df["fa_capacity_margin"],
+            df["bp_train_mean"],
+            marker="o",
+            linewidth=1.8,
+            linestyle="--",
+            label=f"BP {label}",
+        )
+        ax2.plot(
+            df["fa_capacity_margin"],
+            df["fa_train_mean"],
+            marker="o",
+            linewidth=2.0,
+            label=f"FA {label}",
+        )
+    ax2.axvline(0, color="black", linestyle="--", linewidth=1.1)
+    ax2.set_xlim(1100, -430)
+    ax2.set_yscale("log")
+    ax2.set_xlabel("hard FA capacity margin P - K_FA - N*out (capacity decreases ->)")
+    ax2.set_ylabel("train MSE")
+    ax2.set_title("BP/FA train losses across training time")
+    ax2.legend(frameon=True, fontsize=8, ncol=2)
+    ax2.grid(alpha=0.18)
+    fig2.tight_layout()
+    path2 = outdir / "bp_fa_train_loss_by_training_time.png"
+    fig2.savefig(path2)
+    plt.close(fig2)
+
+    print(f"plot: {path}")
+    print(f"plot: {path2}")
+
+
+if __name__ == "__main__":
+    main()