Record dense soft capacity transition

2 files changed, 162 insertions, 0 deletions

diff --git a/notes/22_dense_phase_transition_soft_ramp.md b/notes/22_dense_phase_transition_soft_ramp.md
new file mode 100644
index 0000000..88f6992
--- /dev/null
+++ b/notes/22_dense_phase_transition_soft_ramp.md
@@ -0,0 +1,93 @@
+# Dense Long-Training Phase Transition
+
+The previous `T=30000` transition plot only had four width points. That was not
+enough to decide whether the long-time curve has a kink or a soft ramp.
+
+We ran a denser long-training sweep:
+
+```text
+outputs/phase_transition_dense_T30000_352traj
+```
+
+Setup:
+
+```text
+task: random-label regression
+architecture: 16 -> width -> width -> 4
+optimizer: full-batch SGD
+learning rate: 0.01
+train samples: 128
+steps: 30000
+widths: 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40
+init seeds: 1
+feedback seeds: 32
+FA trajectories: 352 total
+```
+
+Main plot:
+
+```text
+outputs/phase_transition_dense_T30000_352traj/dense_T30000_gap_logscale.png
+```
+
+## Result
+
+| width | FA margin | FA trajectories | train gap mean | train gap std |
+|---:|---:|---:|---:|---:|
+| 20 | -190 | 32 | 0.146803 | 0.042009 |
+| 22 | -158 | 32 | 0.100526 | 0.034803 |
+| 24 | -126 | 32 | 0.046464 | 0.024474 |
+| 26 | -94 | 32 | 0.029671 | 0.023591 |
+| 28 | -62 | 32 | 0.014195 | 0.009368 |
+| 30 | -30 | 32 | 0.007466 | 0.004624 |
+| 32 | 2 | 32 | 0.003729 | 0.003787 |
+| 34 | 34 | 32 | 0.002079 | 0.002232 |
+| 36 | 66 | 32 | 0.000791 | 0.000605 |
+| 38 | 98 | 32 | 0.000553 | 0.000545 |
+| 40 | 130 | 32 | 0.000162 | 0.000134 |
+
+The dense sweep does not show a sharp kink. It shows a smooth, approximately
+log-linear soft ramp in the long-training FA/BP train gap.
+
+## Interpretation
+
+The earlier `T=10000` curve was partly finite-time undertraining, because
+longer training collapses the positive-margin gap from roughly `0.05` to
+roughly `0.004` near margin zero.
+
+However, after adding dense `T=30000` points, the long-training curve is still
+not a hard step. The correct conclusion is:
+
+```text
+hard FA margin predicts a capacity-controlled regime variable,
+not a discontinuous empirical transition.
+```
+
+The empirical transition is soft:
+
+```text
+more negative FA margin -> larger train gap;
+positive FA margin -> small but nonzero finite-time/soft-alignment tail;
+no sharp kink at margin zero.
+```
+
+## Consequence for the Paper
+
+Do not write:
+
+```text
+FA and BP are identical until redundant parameters are exhausted, then a sharp
+gap appears.
+```
+
+Use:
+
+```text
+The hard margin gives a conservative redundancy-exhaustion boundary. Empirically,
+the FA/BP train-gap distribution changes smoothly with the margin; longer
+training removes most positive-margin gap, while negative margins retain a
+capacity-controlled gap.
+```
+
+This is still a useful capacity result, but the contribution should be framed as
+a scaling law plus soft capacity transition, not a hard phase transition.
diff --git a/scripts/plot_dense_phase_transition.py b/scripts/plot_dense_phase_transition.py
new file mode 100644
index 0000000..4b6b4b6
--- /dev/null
+++ b/scripts/plot_dense_phase_transition.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python3
+"""Plot dense long-training phase-transition trajectories."""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import pandas as pd
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--run-dir",
+        type=Path,
+        default=Path("outputs/phase_transition_dense_T30000_352traj"),
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    run_dir = args.run_dir
+    runs = pd.read_csv(run_dir / "runs.csv")
+    summary = pd.read_csv(run_dir / "width_summary.csv").sort_values(
+        "fa_capacity_margin", ascending=False
+    )
+
+    fig, ax = plt.subplots(figsize=(8.8, 5.4), dpi=180)
+    fa = runs[runs.run_type == "fa"].copy()
+    fa["_tid"] = fa.init_seed.astype(str) + ":" + fa.feedback_seed.astype(str)
+    for _tid, group in fa.groupby("_tid"):
+        group = group.sort_values("fa_capacity_margin", ascending=False)
+        ax.plot(
+            group.fa_capacity_margin,
+            group.train_gap_to_bp.clip(lower=1e-7),
+            color="#174f91",
+            alpha=0.12,
+            linewidth=0.8,
+        )
+    ax.errorbar(
+        summary.fa_capacity_margin,
+        summary.train_gap_mean.clip(lower=1e-7),
+        yerr=summary.train_gap_std,
+        color="#0b4d92",
+        marker="o",
+        linewidth=2.2,
+        capsize=3,
+        label="mean +/- sd",
+    )
+    ax.axvline(0, color="black", linestyle="--", linewidth=1.1)
+    ax.set_yscale("log")
+    ax.set_xlim(150, -210)
+    ax.set_xlabel("hard FA capacity margin P - K_FA - N*out")
+    ax.set_ylabel("FA train MSE - BP train MSE (log scale)")
+    ax.set_title("Dense long-training FA/BP gap vs capacity margin")
+    ax.grid(alpha=0.18, which="both")
+    ax.legend(frameon=True)
+    fig.tight_layout()
+    path = run_dir / "dense_T30000_gap_logscale.png"
+    fig.savefig(path)
+    plt.close(fig)
+    print(f"plot: {path}")
+
+
+if __name__ == "__main__":
+    main()