diff options
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 17 | ||||
| -rw-r--r-- | ep_run/casc_eq_train.py | 4 |
2 files changed, 21 insertions, 0 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md index 34d16d5..76cf71d 100644 --- a/docs/campaign/CASCADE_ABLATION_PLAN.md +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -59,6 +59,23 @@ under-convergence, fix = scale K with depth, then relaunch full 3-seed at min-su verdict. If K8 does NOT close it -> genuine estimator depth-tax; next arm = beta-floor (needs a code flag) and/or lambda_l per-layer energy weighting (B4). Follow-on (not yet launched): Muon-on-EP arm. +### RESULT 1 (2026-07-10 00:40): K REFUTED as the lever; BP 3-seed sealed. +- BP 3-seed reference SEALED: 1.9169 / 1.9194 / 1.9214 = **1.9192 +/- 0.0019** (L12 C512 H8). +- **cos is K-INVARIANT.** K3 and K8 track to 4 decimals through step 1200 (both 1.0->0.9997->0.9991) + AND give identical val CE at every matched step (900: 2.545 vs 2.548; 1100: 2.399 vs 2.404). + More relaxation rounds do NOTHING -> the cos erosion is NOT fb under-convergence. K8 killed (redundant). +- **Real mechanism = finite-beta SNR collapse.** beta_t = beta0*bscale*(SIG0/sig)^2 collapses ~120x + (3e-3 -> 2.5e-5) as sig_tok grows 1.6->17.8. The estimator computes E/(NBT*beta_t) from residuals + (z-o) that are O(beta_t*sig) ~ 4e-4 obtained by subtracting two O(17) states -> catastrophic + cancellation as beta shrinks AND sig grows. Both worsen with depth. cos erodes 1.0 -> 0.997 (@2000) + -> 0.98 (@2800 in the dead run). This is a beta-SCHEDULE problem, not a relaxation-depth problem. +- **Fix under test:** added `--beta_floor` / `--beta_fixed` flags. Launched paired arms seed 1 + (control = K3 floor=0, still running): `d1b_ep_bf1e4_s1` (floor 1e-4), `d1b_ep_bf3e4_s1` (floor 3e-4). + Decision rule: if floored cos stays high through step 2000-2800 and CE drops toward BP 1.919 -> + beta-floor is the depth fix; pick min-sufficient floor, run 3-seed K4 verdict. Watch drift guard at + the higher floor (larger nudge). If floors DON'T help -> escalate to double-sided estimator (cancels + O(beta) Taylor bias, allows large beta, 2x cost) or lambda_l energy weighting. + ## STATUS 2026-07-09 (same day): Tier 0 CLOSED GREEN via the zil scheme; C1 running - **Naive relaxation FAILS at depth** (the B1-lite sweep): jacobi K=40·L → cos 0.82 (L6) / 0.67 (L12) diff --git a/ep_run/casc_eq_train.py b/ep_run/casc_eq_train.py index f1f26a4..67ee98f 100644 --- a/ep_run/casc_eq_train.py +++ b/ep_run/casc_eq_train.py @@ -26,6 +26,8 @@ ap.add_argument('--muon_lr', type=float, default=0.02) ap.add_argument('--tok_init', type=float, default=0.0) # >0: init tok/pos with this std (GPT-standard 0.02) ap.add_argument('--compile', action='store_true') # torch.compile each block (free speed where supported) ap.add_argument('--sig_every', type=int, default=25) # tok-sigma refresh interval (amortized) +ap.add_argument('--beta_floor', type=float, default=0.0) # >0: floor beta_t (anti finite-beta SNR collapse at depth) +ap.add_argument('--beta_fixed', action='store_true') # disable sig^2 schedule, hold beta_t = args.beta constant ap.add_argument('--dtop_every', type=int, default=1) # 1 = exact (DEFAULT, BP-parity); 2 = fast mode (~20% cheaper, ~4% CE tax at high lr) ap.add_argument('--gate_every', type=int, default=200) # in-training cos(EP,BP) telemetry; <=0 = fully BP-free (no bp_gate at all) ap.add_argument('--gate_govern', action='store_true') # let gate cos adjust K/bscale (default: observe-only => training control is BP-free) @@ -157,6 +159,8 @@ def ep_step(x, y): sig = GOV['sig'] if SIG0 is None: SIG0 = sig beta_t = args.beta * GOV['bscale'] * (SIG0 * SIG0) / max(sig * sig, 1e-9) + if args.beta_fixed: beta_t = args.beta * GOV['bscale'] + if args.beta_floor > 0.0: beta_t = max(beta_t, args.beta_floor) z0, zs, ins, outs = free_states_graphed(x) zs_free = [z.clone() for z in zs] free_ce = F.cross_entropy(readout(zs_free[-1]).reshape(-1, vocab), y.reshape(-1)).item() |