RESULT 64: 非plateau而是恒速泄漏(宽度×4-5,退火追不上); 全嫌疑至分辨率洗清(bias≤sem,msign无放大,σ两边同一/BP更响); 72M账本n=3合卷(BP 3.2882±0.006,bsign gap+0.076); 磁盘满事故+458GB修剪(清单落档); amp降级次要(~12%); 泄漏率配对探针在飞

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn

2 files changed, 195 insertions, 0 deletions

diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md
index 1606dd2..a73f28b 100644
--- a/docs/campaign/CASCADE_ABLATION_PLAN.md
+++ b/docs/campaign/CASCADE_ABLATION_PLAN.md
@@ -568,6 +568,28 @@ direction), not training-under-fault; wave-2 = co-training with faults injected
 
 
 
+### RESULT 64 (2026-07-28): NOT A PLATEAU — A CONSTANT-RATE RELATIVE LEAK, 4-5x wider at
+C768; every named suspect cleared AT PROBE RESOLUTION; the leak-rate probe is the new instrument.
+LEDGERS CLOSED: 72M BP n=3: 3.2884/3.2943/3.2820 -> mean 3.2882+-0.006. Honest 72M gap:
+bsign(n=2) +0.076 (7.9% ppl); cent(n=1) +0.044. — fp32 A/B (both arms died at s275000
+torch.save: DISK FULL, 100%/10T shared NFS; pruned 458GB of intermediate ckpts, 652 files,
+manifest runs/PRUNE_MANIFEST_260728.txt, 39 landmarks kept): fp32@3e-3 window val_mean 3.5538
+vs amp 3.5891 vs fp32@1.5e-3 3.5843 -> amp DEMOTED to minor contributor (~12% of the deficit);
+R63's "amp is THE mechanism" corrected.
+SUSPECTS CLEARED (with resolution caveats): systematic bias at C768 <= sem (probe_bias nb=12:
+plain ~ sem everywhere; w2_b11 0.81% vs sem 0.51% = the one 1.6x flicker); Muon amplification
+NONE (msign cos 0.9987-0.9994 vs raw 0.9998-1.0000); sigma/scale runaway NONE (tok_rms
+IDENTICAL to 3 decimals; BP head LOUDER: sig1 744 vs EP 620 — the louder-states trajectory is
+normal training, not EP pathology).
+THE SHAPE (schedule-fraction matched): 72M gap hovers +0.04..0.09 and the anneal CLOSES it
+(0.087@80% -> 0.047@100%); 135M gap grows near-linearly +0.10/0.16/0.20/0.28/0.29 — a
+per-progress leak, 4-5x the 72M rate, that the anneal cannot outrun; "plateau" = leak rate
+catching BP's descent rate. Required per-step magnitude: ~1% CORRELATED discrepancy — exactly
+at/below every probe's floor. The 0.999-cos-yet-diverging paradox is the finding.
+IN FLIGHT: leak-rate paired probe — SAME weights (s250000), EP vs BP, 6k steps each, direct
+DCE readout; calibrates the leak instrument for recipe search (anything that cuts measured
+leak-rate is a candidate; full-run validation only after).
+
 ### RESULT 63 (2026-07-27): MECHANISM CLOSED — THE AMP CERTIFICATE BREAKS AT WIDTH AND THE
 C768 CORRIDOR INVERTS MID-RUN; every 135M EP failure mode falls into place.
 Chain of three probes (all GPU0, hours): (1) blockcos @s150000: fp32 EP gradient PERFECT
diff --git a/ep_run/probe_msign.py b/ep_run/probe_msign.py
new file mode 100644
index 0000000..e6e8824
--- /dev/null
+++ b/ep_run/probe_msign.py
@@ -0,0 +1,173 @@
+"""Suspect-2 probe (C768 plateau): does Muon msign AMPLIFY the tiny EP-vs-BP discrepancy?
+Per weight matrix: cos(gEP,gBP) raw vs cos(msign(gEP),msign(gBP)). Derived from probe_bias.
+Old docstring: Direct measurement of the systematic single-sided EP bias (the audit gap: BBP probes
+measured NOISE thoroughly, discarded the mean/bias component unreported) + verification of the
+user's claim: PER-SAMPLE random nudge sign makes the batch-averaged estimator unbiased at O(beta).
+Three estimators at fixed ckpt, N batches, two betas:
+  plain      : all samples nudged +beta            -> bias = beta*E[B] (systematic, batch-indep)
+  persample  : sample i nudged s_i*beta, read re-flipped by s_i -> odd-order bias cancels
+               WITHIN each batch (sqrt(B) suppression per step, 0 in expectation)
+  BP         : exact reference.
+Self-check: persample with all s=+1 must equal plain exactly.
+Report per layer family: |mean(ghat-gbp)| / |mean(gbp)| (systematic), across-batch sem (noise
+floor), for beta in {3e-3, 1e-2} — plain's bias should scale ~x3.3, persample's should sit at
+the noise floor. Read-only, coexists on GPU0."""
+import argparse, pickle
+import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
+from pathlib import Path
+
+ap = argparse.ArgumentParser()
+ap.add_argument('--ckpt', default='runs/fw72m_plain2_s35000.pt')
+ap.add_argument('--K', type=int, default=3)
+ap.add_argument('--betas', default='3e-3,1e-2')
+ap.add_argument('--nb', type=int, default=16)
+a = ap.parse_args()
+dev = 'cuda'
+torch.manual_seed(11)
+B, T = 8, 256
+
+class RMSNorm(nn.Module):
+    def __init__(self, C, eps=1e-6):
+        super().__init__(); self.g = nn.Parameter(torch.ones(C)); self.eps = eps
+    def forward(self, x):
+        return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.g
+
+class SwiGLU(nn.Module):
+    def __init__(self, C):
+        super().__init__()
+        h = ((8 * C // 3) + 63) // 64 * 64
+        self.w1 = nn.Linear(C, h, bias=False); self.w3 = nn.Linear(C, h, bias=False)
+        self.w2 = nn.Linear(h, C, bias=False)
+    def forward(self, x):
+        return self.w2(F.silu(self.w1(x)) * self.w3(x))
+
+class Olmo2Attn(nn.Module):
+    def __init__(self, C, H, T):
+        super().__init__()
+        self.H, self.hd = H, C // H
+        self.qkv = nn.Linear(C, 3 * C, bias=False); self.proj = nn.Linear(C, C, bias=False)
+        self.qn, self.kn = RMSNorm(C), RMSNorm(C)
+        inv = 1.0 / (500000.0 ** (torch.arange(0, self.hd, 2).float() / self.hd))
+        fr = torch.outer(torch.arange(T).float(), inv)
+        self.register_buffer('rc', fr.cos(), persistent=False)
+        self.register_buffer('rs', fr.sin(), persistent=False)
+    def rope(self, x):
+        Tn = x.shape[2]
+        x1, x2 = x[..., ::2], x[..., 1::2]
+        c, s = self.rc[None, None, :Tn].to(x.dtype), self.rs[None, None, :Tn].to(x.dtype)
+        return torch.stack((x1 * c - x2 * s, x1 * s + x2 * c), dim=-1).flatten(-2)
+    def forward(self, x):
+        Bn, Tn, C = x.shape
+        q, k, v = self.qkv(x).split(C, dim=2)
+        q, k = self.qn(q), self.kn(k)
+        q = self.rope(q.view(Bn, Tn, self.H, self.hd).transpose(1, 2))
+        k = self.rope(k.view(Bn, Tn, self.H, self.hd).transpose(1, 2))
+        v = v.view(Bn, Tn, self.H, self.hd).transpose(1, 2)
+        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        return self.proj(y.transpose(1, 2).contiguous().view(Bn, Tn, C))
+
+class Olmo2Block(nn.Module):
+    def __init__(self, C, H, T):
+        super().__init__()
+        self.attn = Olmo2Attn(C, H, T); self.ff = SwiGLU(C)
+        self.na, self.nf = RMSNorm(C), RMSNorm(C)
+    def forward(self, z):
+        z = z + self.na(self.attn(z))
+        return z + self.nf(self.ff(z))
+
+ck = torch.load(a.ckpt, map_location=dev, weights_only=False)
+cfg = ck['config']; C, H, L = cfg['C'], cfg['H'], cfg['L']
+DD = Path('/home/yurenh2/ept/ep_run/data') / cfg.get('data', 'fineweb_edu')
+vocab = pickle.load(open(DD / 'meta.pkl', 'rb'))['vocab_size']
+data = np.memmap(DD / 'val.bin', dtype=np.uint16, mode='r')
+tok = nn.Embedding(vocab, C).to(dev); tok.load_state_dict(ck['tok'])
+blocks = nn.ModuleList([Olmo2Block(C, H, T) for _ in range(L)]).to(dev)
+blocks.load_state_dict(ck['blocks'], strict=False)
+W_out = ck['wout'].to(dev); ln_f = RMSNorm(C).to(dev); ln_f.load_state_dict(ck['lnf'])
+NBT = B * T
+params = list(blocks.parameters())
+names = [n for n, _ in blocks.named_parameters()]
+
+def readout(z): return ln_f(z) @ W_out.t()
+
+def get_batch(g):
+    ix = torch.randint(len(data) - T - 1, (B,), generator=g)
+    x = torch.stack([torch.from_numpy(data[i:i + T].astype(np.int64)) for i in ix])
+    y = torch.stack([torch.from_numpy(data[i + 1:i + 1 + T].astype(np.int64)) for i in ix])
+    return x.to(dev), y.to(dev)
+
+def grads_from(outs_list, cots):
+    obj = sum((o * c.detach()).sum() for o, c in zip(outs_list, cots))
+    gs = torch.autograd.grad(obj, params, allow_unused=True, retain_graph=True)
+    return [g.float() if g is not None else torch.zeros_like(p) for g, p in zip(gs, params)]
+
+def estimators(x, y, beta, signs):
+    """signs: (B,) tensor of +-1. Returns (g_bp, g_ep) where g_ep uses per-sample sign nudges."""
+    z = tok(x)
+    zs_bp = []
+    for b in blocks:
+        z = b(z); zs_bp.append(z)
+    ce = F.cross_entropy(readout(zs_bp[-1]).reshape(-1, vocab), y.reshape(-1))
+    g_bp = [g.float() for g in torch.autograd.grad(ce, params, retain_graph=True)]
+    f_ins, f_outs = [], []
+    prev = tok(x).detach()
+    for b in blocks:
+        i = prev.detach().requires_grad_(True)
+        o = b(i)
+        f_ins.append(i); f_outs.append(o)
+        prev = o.detach()
+    ins, outs = f_ins, f_outs
+    zs = [o.detach().float() for o in f_outs]
+    d = [None] * L
+    sB = signs.view(B, 1, 1).float()
+    for k in range(a.K):
+        zc = zs[L - 1].detach().requires_grad_(True)
+        # per-sample signed top objective: sum_i s_i * (sum_t loss_it)  (token-sum, matches beta*NBT*mean scaling)
+        ce_tok = F.cross_entropy(readout(zc).reshape(-1, vocab), y.reshape(-1), reduction='none').view(B, T)
+        obj_top = (signs.float()[:, None] * ce_tok).sum()
+        d[L - 1] = (-beta * torch.autograd.grad(obj_top, zc)[0]).detach().float()
+        for l in range(L - 2, -1, -1):
+            d[l] = torch.autograd.grad(outs[l + 1], ins[l + 1], grad_outputs=d[l + 1],
+                                       retain_graph=True)[0].detach().float()
+        prev = tok(x).detach()
+        n_ins, n_outs = [], []
+        for l in range(L):
+            i = prev.detach().requires_grad_(True)
+            o = blocks[l](i)
+            n_ins.append(i); n_outs.append(o)
+            zs[l] = o.detach().float() + d[l]
+            prev = zs[l]
+        ins, outs = n_ins, n_outs
+    md = [-(sB * di) / (beta * NBT) for di in d]   # re-flip per sample; scale matches trainer
+    g_ep = grads_from(outs, md)
+    return g_bp, g_ep
+
+
+from muon import newton_schulz
+FAMS = {'qkv_bot': [f'{l}.attn.qkv.weight' for l in range(0,4)],
+        'qkv_top': [f'{l}.attn.qkv.weight' for l in range(8,12)],
+        'w2_bot':  [f'{l}.ff.w2.weight' for l in range(0,4)],
+        'w2_top':  [f'{l}.ff.w2.weight' for l in range(8,12)],
+        'proj_top':[f'{l}.attn.proj.weight' for l in range(8,12)]}
+IDX = {k: [names.index(n) for n in v] for k, v in FAMS.items()}
+g0 = torch.Generator().manual_seed(1234)
+def cosf(a, b):
+    return float((a*b).sum() / (a.norm()*b.norm() + 1e-30))
+acc = {k: {'raw': [], 'ms': []} for k in IDX}
+for bi in range(6):
+    x, y = get_batch(g0)
+    ones = torch.ones(B, dtype=torch.long, device=dev)
+    g_bp, g_ep = estimators(x, y, 3e-3, ones)
+    for k, idxs in IDX.items():
+        for i in idxs:
+            acc[k]['raw'].append(cosf(g_ep[i], g_bp[i]))
+            me = newton_schulz(g_ep[i].bfloat16()).float()
+            mb = newton_schulz(g_bp[i].bfloat16()).float()
+            acc[k]['ms'].append(cosf(me, mb))
+    del g_bp, g_ep
+    torch.cuda.empty_cache()
+import numpy as np
+print(f'{"family":>9} {"cos_raw":>9} {"cos_msign":>10}  (mean over 6 batches x 4 mats)')
+for k in IDX:
+    print(f'{k:>9} {np.mean(acc[k]["raw"]):>9.4f} {np.mean(acc[k]["ms"]):>10.4f}')
+print('MSIGN_DONE', flush=True)