diff options
| -rw-r--r-- | docs/campaign/CASCADE_ABLATION_PLAN.md | 106 | ||||
| -rw-r--r-- | ep_run/casc_bp_train.py | 94 | ||||
| -rw-r--r-- | ep_run/cascade_probe.py | 158 |
3 files changed, 304 insertions, 54 deletions
diff --git a/docs/campaign/CASCADE_ABLATION_PLAN.md b/docs/campaign/CASCADE_ABLATION_PLAN.md new file mode 100644 index 0000000..5c10dd9 --- /dev/null +++ b/docs/campaign/CASCADE_ABLATION_PLAN.md @@ -0,0 +1,106 @@ +# Cascade-EP ablation program — standard multi-layer LLM, EP only in training + +**Date opened:** 2026-07-09 · **Trigger:** user directive — product form = standard L-layer +transformer (plain-forward inference); the looped/weight-tied block is demoted to physics testbed. +**Bridge:** layered energy E = Σ_l ½‖z_l − f_l(z_{l−1})‖² over DISTINCT standard blocks. +Free equilibrium == the standard forward pass (E=0) ⟹ inference is a normal LLM forward. +Training = two-phase (±β·CE at the top), relax states to nudged equilibria, ∇θ = (1/2β)[∂E/∂θ|₊ − ∂E/∂θ|₋]. +Lineage: predictive-coding≈BP theorem family (Whittington-Bogacz 17; Song+ 20 / Z-IL), EP two-phase readout. +**First gate (2026-07-09):** `cascade_probe.py` L3 C128 random init → cos(cascEP, BP) **0.9968** +(blocks 0.9975/0.9980/0.9992, |EP|/|BP| 0.80–0.91). + +## The five claims we are buying evidence for + +- **K1 exactness-on-trajectory** — the two-phase gradient matches BP not just at init but along a + real training trajectory (weights with grown Jacobians stiffen the relaxation). +- **K2 cost** — the nudged relaxation can be engineered to a small multiple of a BP step + (scheme × K frontier), and the *physical* (Jacobi/parallel) scheme is not hopeless (analog story). +- **K3 training parity** — full training closes to BP final CE at equal arch/steps (the money claim). +- **K4 depth scaling** — no depth penalty vs BP at matched params (signal attenuation under control). +- **K5 analog price** — per-block Jᵀ feedback, dynamic noise, quantization: the tolerance ledger + ports from the looped-block program; PAR wall applies per block. + +Honest cost framing: on GPU cascade-EP is strictly MORE expensive per step than BP (K relax sweeps, +each ≈ one fwd+state-vjp). The value is: standard-form deployment + local rules + analog trainability. +The looped-EP precedent multiplier was ~230× BP; the K-frontier decides whether cascade beats that. + +--- + +## Tier 0 — gate hardening (probe-scale, hours, no training) → K1 + +| ID | question | design | decision rule | +|---|---|---|---| +| A0.1 | does cos survive depth? | cos vs L ∈ {3,6,12,24}, C128, Jacobi K auto-scaled; ≥4 batches | cos ≥ 0.98 at L12 or B1 must fix it | +| A0.2 | does cos survive training? | BP-train C256 L6 4k steps saving every 500 (`casc_bp_train.py`); gate at every ckpt; ALSO record required-K to reach res-tol | cos ≥ 0.97 at all ckpts; K growth ≤ 3× init→4k | +| A0.3 | full-θ gate | include emb/pos/readout(tied) grads in the gate | all groups ≥ 0.97 | +| A0.4 | precision | fp32 vs TF32 vs bf16 on the two-phase difference | pick cheapest safe mode (looped-EP lesson: TF32 killed relaxation — re-test here) | + +## Tier 1 — relaxation engineering (the cost frontier) → K2 + +| ID | axis | arms | metric | +|---|---|---|---| +| B1 | scheme × K | Jacobi (physical, parallel) vs Gauss-Seidel fwd vs GS reverse (algorithmic; Z-IL limit) × K ∈ {12,25,50,100,200,400} at L6 & L12 | K needed for cos ≥ 0.98; wall-clock multiple vs one BP step | +| B2 | state optimizer | GD vs +momentum vs Adam-on-states; η sweep | same | +| B3 | nudge β | {0.003,0.01,0.03,0.1,0.3} × one-sided vs two-sided | cos, shrinkage |EP|/|BP|, required K | +| B4 | energy weighting | raw ℓ₂ vs per-layer precision λ_l=1/RMS² vs LN-in-energy | per-block shrinkage PROFILE (fix the 0.80→0.91 depth attenuation) + relax conditioning | +| B5 | stopping | fixed-K vs relax-to-tol | natural K distribution | +| B6 | **depth attenuation / estimator SNR profile** | measure per-block error amplitude ‖e_l‖ and per-block cos vs depth, as f(L, β, K) | the estimator-precision law: how fast does the deep-layer signal die, and which knob (β, K, λ_l weighting) restores it | + +B1 is the single most consequential experiment in the program: if GS-reverse needs K≈L (Z-IL limit) +we have a ~BP-cost algorithmic mode for GPU pretraining, and the Jacobi column is the honest +analog-hardware price. Report all three columns — they are different products. + +**Dynamics-vs-estimator tradeoff (user insight, 2026-07-09):** the cascade is dynamically SIMPLER — +the free phase is EXACT (a plain forward; no res/T1/fixed-point error, no Hopf, no collapse), so +**C-tier default arms run with NO regularizers at all** (jr/resreg don't exist here; stability regs +return only if evidence demands). The difficulty MOVES to the estimator: the two-phase difference +must resolve per-layer error signals that ATTENUATE with depth (visible at L=3 already: shrink 0.80 +bottom vs 0.91 top), finite-β Taylor bias and finite-K relaxation bias hit the deepest blocks first, +and the difference-of-O(1)-quantities structure makes precision (A0.4, fp32-vs-TF32) bind harder +than in looped-EP. B6 is the dedicated measurement; λ_l weighting (B4), β/K scheduling (B3/B1) and +per-block rebalance (C5) are the candidate antidotes. + +## Tier 2 — small full-training ablations (C256 L6 T256 TinyStories, 8–16k steps) → K3 + +| ID | arm | vs | +|---|---|---| +| C1 | **money run**: cascade-EP (B-tier winner) ×2–3 seeds | BP twin, same arch/data/AdamW/steps — target gap ≤ 0.05 CE | +| C2 | K budget: {K*, 2K*, 4K*} | CE-vs-cost curve (training may need less relax than the gate does — looped-EP precedent: t2sel 40 trains, 80 gates) | +| C3 | one-sided β (half cost) | two-sided | +| C4 | AdamW | SGDM (shrinkage sensitivity — does 0.8–0.9 amplitude matter under Adam's rescaling?) | +| C5 | shrinkage compensation: none | per-block grad-norm rebalance to BP profile (one-time calibration) | +| C6 | B4-winner energy weighting | raw | + +Placement: 1080 farm **after a Pascal canary** (cascade-EP is a new workload class; the Pascal +pathology ban was derived on looped-EP+regs — do a 800-step canary + cross-env fingerprint first). +C256 L6 fits 8 GB (~19M params, ~2-3 GB act). + +## Tier 3 — depth/scale rungs (Delta A40 chains) → K4 + +| ID | design | +|---|---| +| D1 | **north-star demo re-target**: L12 C512 (≈45M, a real GPT-small shape) cascade-EP vs BP twin — replaces the single-block 33M rung as the flagship demo (task #15) | +| D2 | depth ladder at fixed params: L6/C724 vs L12/C512 vs L24/C362 — depth penalty vs BP? | +| D3 | T 256→512 sanity (relax cost tracks attention; expect no surprise) | + +## Tier 4 — analog/hardware arms (port the tolerance machinery) → K5 + +| ID | design | +|---|---| +| E1 | Jacobi + per-sweep dynamic noise: does the fnoise ≥1e-3 cliff reappear in cascade relaxation? | +| E2 | Jᵀ ablation: replace J_lᵀe with fixed random Bᵀ (feedback-alignment) / PAR projection — the per-block analog-feasibility tax; FA classically works on shallow stacks, test at L6 | +| E3 | static tolerance: wq8/wq6 weights inside relax | + +## Sequencing & fleet + +``` +now: A0.1 + A0.3 + B1-lite (shared local GPU, ~1h) + casc_bp_train ckpt producer (107 free 1080) +gate ok → B1 full / B2 / B3 / B4 (local A6000s as arms free; each = minutes-hours) + → Pascal canary → C-tier fan-out on 1080 farm (6 arms × 1-2 days) + → D1 chains on Delta A40 (queue behind current five lines) +E-tier: after C1 lands (tolerance scripts port directly) +``` + +Naming: `casc_*` runs, wandb project **ept-cascade**. Gates report mean over ≥4 batches. +In-flight single-block arms (rescv2, govfloor, fastfull/fastpair, gov_s11-14) continue untouched — +they carry the dynamics paper + the two-stage-recipe science; D1 takes over the DEMO role only. diff --git a/ep_run/casc_bp_train.py b/ep_run/casc_bp_train.py new file mode 100644 index 0000000..8e039ec --- /dev/null +++ b/ep_run/casc_bp_train.py @@ -0,0 +1,94 @@ +"""BP-train a small cascade-form standard transformer (L distinct blocks), saving ckpts +every --save_every for the A0.2 on-trajectory gradient gate (cascade_probe.py --ckpt). +Plain LLM training — this is also the BP twin for the C-tier money runs.""" +import argparse, math, pickle, time, json +import numpy as np, torch, torch.nn as nn, torch.nn.functional as F +from pathlib import Path + +ap = argparse.ArgumentParser() +ap.add_argument('--tag', default='casc_bp6') +ap.add_argument('--L', type=int, default=6); ap.add_argument('--C', type=int, default=256) +ap.add_argument('--H', type=int, default=8); ap.add_argument('--T', type=int, default=256) +ap.add_argument('--B', type=int, default=24); ap.add_argument('--steps', type=int, default=4000) +ap.add_argument('--lr', type=float, default=3e-4); ap.add_argument('--warmup', type=int, default=200) +ap.add_argument('--seed', type=int, default=0) +ap.add_argument('--save_every', type=int, default=500); ap.add_argument('--log', type=int, default=200) +ap.add_argument('--wandb', default=''); ap.add_argument('--wandb_run', default='') +args = ap.parse_args() +torch.manual_seed(args.seed) +dev = 'cuda' if torch.cuda.is_available() else 'cpu' + +DD = Path('/home/yurenh2/ept/ep_run/data/tinystories_bpe') +vocab = pickle.load(open(DD / 'meta.pkl', 'rb'))['vocab_size'] + +def get_batch(split): + data = np.memmap(DD / ('train.bin' if split == 'train' else 'val.bin'), dtype=np.uint16, mode='r') + ix = torch.randint(len(data) - args.T - 1, (args.B,)) + x = torch.stack([torch.from_numpy(data[i:i + args.T].astype(np.int64)) for i in ix]) + y = torch.stack([torch.from_numpy(data[i + 1:i + 1 + args.T].astype(np.int64)) for i in ix]) + return x.to(dev), y.to(dev) + +class Block(nn.Module): + def __init__(self, C, H): + super().__init__() + self.ln1, self.ln2 = nn.LayerNorm(C), nn.LayerNorm(C) + self.attn = nn.MultiheadAttention(C, H, batch_first=True) + self.ff = nn.Sequential(nn.Linear(C, 4 * C), nn.GELU(), nn.Linear(4 * C, C)) + def forward(self, z, mask): + h = self.ln1(z); a, _ = self.attn(h, h, h, attn_mask=mask, need_weights=False) + z = z + a; return z + self.ff(self.ln2(z)) + +tok = nn.Embedding(vocab, args.C).to(dev) +pos = nn.Embedding(args.T, args.C).to(dev) +blocks = nn.ModuleList([Block(args.C, args.H) for _ in range(args.L)]).to(dev) +mask = torch.triu(torch.full((args.T, args.T), float('-inf'), device=dev), 1) +params = list(tok.parameters()) + list(pos.parameters()) + list(blocks.parameters()) +opt = torch.optim.AdamW(params, lr=args.lr, weight_decay=1e-4) +sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / max(args.warmup, 1))) + +def fwd(x): + z = tok(x) + pos(torch.arange(args.T, device=dev))[None] + for b in blocks: z = b(z, mask) + return z @ tok.weight.t() + +@torch.no_grad() +def evaluate(nb=6): + tot = 0.0 + for _ in range(nb): + x, y = get_batch('val') + tot += F.cross_entropy(fwd(x).reshape(-1, vocab), y.reshape(-1)).item() + return tot / nb + +wb = None +if args.wandb: + try: + import wandb as _w + wb = _w.init(project=args.wandb, name=args.wandb_run or args.tag, id=args.wandb_run or args.tag, + resume='allow', config=vars(args)) + except Exception as e: + print(f'[wandb] disabled ({e})', flush=True) + +n = sum(p.numel() for p in params) +print(f'[{args.tag}] cascade-BP L{args.L} C{args.C} H{args.H} T{args.T} | {n/1e6:.2f}M params | {dev}', flush=True) +best, t0 = 1e9, time.time() +outdir = Path('runs'); outdir.mkdir(exist_ok=True) +for step in range(args.steps + 1): + x, y = get_batch('train') + loss = F.cross_entropy(fwd(x).reshape(-1, vocab), y.reshape(-1)) + opt.zero_grad(set_to_none=True); loss.backward() + torch.nn.utils.clip_grad_norm_(params, 1.0) + opt.step(); sched.step() + if step % args.log == 0: + val = evaluate(); best = min(best, val) + print(f'step {step:5d}/{args.steps} | train {loss.item():.4f} val {val:.4f} (best {best:.4f}) ' + f'| {step/max(time.time()-t0,1e-9):.2f} it/s', flush=True) + if wb is not None: + try: wb.log({'train_ce': loss.item(), 'val_ce': val, 'best': best}, step=step) + except Exception: pass + if step % args.save_every == 0: + torch.save({'tok': tok.state_dict(), 'pos': pos.state_dict(), 'blocks': blocks.state_dict(), + 'step': step, 'val': best, 'config': vars(args)}, outdir / f'{args.tag}_s{step}.pt') +print(f'[{args.tag}] DONE best val CE {best:.4f} (random ln({vocab})={math.log(vocab):.3f})', flush=True) +if wb is not None: + try: wb.summary['best_val_ce'] = best; wb.finish() + except Exception: pass diff --git a/ep_run/cascade_probe.py b/ep_run/cascade_probe.py index 572097c..2f1449a 100644 --- a/ep_run/cascade_probe.py +++ b/ep_run/cascade_probe.py @@ -1,8 +1,14 @@ -"""Cascade-EP gradient gate: L distinct standard transformer blocks, layered energy -E = sum_l 0.5||z_l - f_l(z_{l-1})||^2. Free equilibrium == the standard forward pass -(inference = plain LLM). Training = two-phase (±beta nudge) relaxation of z, weight -grad = (1/2beta)[dE/dtheta|+ - dE/dtheta|-]. Gate: cosine vs true BP gradient.""" -import argparse, math, pickle +"""Cascade-EP gradient gate v2 — L distinct standard transformer blocks, layered energy +E = sum_l 0.5||z_l - f_l(z_{l-1})||^2. Free equilibrium == the standard forward pass. +Two-phase (+-beta) relaxation of states -> local weight gradient; gate vs true BP. + +v2: relaxation schemes (jacobi = physical parallel | gsf/gsr = Gauss-Seidel fwd/reverse sweeps), + --include_io (emb/pos/tied-readout grads via the full EP formula), multi-batch gates, + machine-parseable SUMMARY line. Usage examples: + python3 cascade_probe.py --L 12 --K 480 + python3 cascade_probe.py --L 6 --scheme gsr --K 12 --include_io +""" +import argparse, math, pickle, time import numpy as np, torch, torch.nn as nn, torch.nn.functional as F from pathlib import Path @@ -11,7 +17,12 @@ ap.add_argument('--L', type=int, default=3); ap.add_argument('--C', type=int, de ap.add_argument('--H', type=int, default=4); ap.add_argument('--T', type=int, default=32) ap.add_argument('--B', type=int, default=4); ap.add_argument('--beta', type=float, default=0.03) ap.add_argument('--K', type=int, default=400); ap.add_argument('--eta', type=float, default=0.3) -ap.add_argument('--seed', type=int, default=0); ap.add_argument('--trained_warp', type=float, default=1.0) +ap.add_argument('--mom', type=float, default=0.9) +ap.add_argument('--scheme', choices=['jacobi', 'gsf', 'gsr'], default='jacobi') +ap.add_argument('--batches', type=int, default=4) +ap.add_argument('--include_io', action='store_true') # gate emb/pos/tied-readout too +ap.add_argument('--seed', type=int, default=0); ap.add_argument('--warp', type=float, default=1.0) +ap.add_argument('--ckpt', type=str, default='') # gate at a casc_bp_train checkpoint args = ap.parse_args() torch.manual_seed(args.seed) dev = 'cuda' if torch.cuda.is_available() else 'cpu' @@ -20,9 +31,6 @@ torch.set_float32_matmul_precision('highest') DD = Path('/home/yurenh2/ept/ep_run/data/tinystories_bpe') vocab = pickle.load(open(DD / 'meta.pkl', 'rb'))['vocab_size'] data = np.memmap(DD / 'val.bin', dtype=np.uint16, mode='r') -ix = torch.randint(len(data) - args.T - 1, (args.B,)) -x = torch.stack([torch.from_numpy(data[i:i + args.T].astype(np.int64)) for i in ix]).to(dev) -y = torch.stack([torch.from_numpy(data[i + 1:i + 1 + args.T].astype(np.int64)) for i in ix]).to(dev) class Block(nn.Module): """Standard pre-LN transformer block (distinct weights per layer, no recurrence).""" @@ -38,65 +46,107 @@ class Block(nn.Module): tok = nn.Embedding(vocab, args.C).to(dev) pos = nn.Embedding(args.T, args.C).to(dev) blocks = nn.ModuleList([Block(args.C, args.H) for _ in range(args.L)]).to(dev) -if args.trained_warp != 1.0: # crude "not-at-init" landscape +if args.ckpt: + sd = torch.load(args.ckpt, map_location=dev) + tok.load_state_dict(sd['tok']); pos.load_state_dict(sd['pos']); blocks.load_state_dict(sd['blocks']) + print(f"[gate] loaded {args.ckpt} (step {sd.get('step')}, val {sd.get('val', float('nan')):.4f})") +elif args.warp != 1.0: with torch.no_grad(): - for p in blocks.parameters(): p.mul_(args.trained_warp) + for p in blocks.parameters(): p.mul_(args.warp) mask = torch.triu(torch.full((args.T, args.T), float('-inf'), device=dev), 1) readout = lambda z: z @ tok.weight.t() +io_params = list(tok.parameters()) + list(pos.parameters()) +blk_params = list(blocks.parameters()) +gate_params = blk_params + (io_params if args.include_io else []) +NBT = args.B * args.T + +def get_batch(i): + g = torch.Generator().manual_seed(1000 + i) + ix = torch.randint(len(data) - args.T - 1, (args.B,), generator=g) + x = torch.stack([torch.from_numpy(data[j:j + args.T].astype(np.int64)) for j in ix]).to(dev) + y = torch.stack([torch.from_numpy(data[j + 1:j + 1 + args.T].astype(np.int64)) for j in ix]).to(dev) + return x, y def fwd_states(z0): zs = [z0] for b in blocks: zs.append(b(zs[-1], mask)) return zs -z0 = (tok(x) + pos(torch.arange(args.T, device=dev))[None]).detach() +def local_grad(zs, z0, l, beta, y): + """dF/dz_l with neighbors fixed (zs: list of L free states, 0-indexed block l).""" + zl = zs[l].detach().requires_grad_(True) + prev = z0 if l == 0 else zs[l - 1].detach() + obj = 0.5 * ((zl - blocks[l](prev, mask)) ** 2).sum() / NBT + if l + 1 < args.L: + obj = obj + 0.5 * ((zs[l + 1].detach() - blocks[l + 1](zl, mask)) ** 2).sum() / NBT + else: + obj = obj + beta * F.cross_entropy(readout(zl).reshape(-1, vocab), y.reshape(-1)) + return torch.autograd.grad(obj, zl)[0] -# ---------------- true BP reference ---------------- -for p in blocks.parameters(): p.requires_grad_(True) -zs = fwd_states(z0) -ce = F.cross_entropy(readout(zs[-1]).reshape(-1, vocab), y.reshape(-1)) -gbp = torch.autograd.grad(ce, list(blocks.parameters()), allow_unused=True) -gbp = [g if g is not None else torch.zeros(1, device=dev) for g in gbp] -print(f'BP ref: CE {ce.item():.4f}') - -# ---------------- two-phase cascade EP ---------------- -def relax(beta): - """minimize E + beta*CE over free states z_1..z_L by GD (free init = forward pass).""" +def relax(z0, y, beta): + """minimize F_beta over states; free init = forward pass. Returns relaxed states.""" with torch.no_grad(): zs = [z.detach().clone() for z in fwd_states(z0)[1:]] - for z in zs: z.requires_grad_(True) - opt = torch.optim.SGD(zs, lr=args.eta, momentum=0.9) - for k in range(args.K): - opt.zero_grad() - E = 0.0; prev = z0 - for z, b in zip(zs, blocks): E = E + 0.5 * ((z - b(prev, mask)) ** 2).sum(); prev = z - Fobj = E / (args.B * args.T) + beta * F.cross_entropy(readout(zs[-1]).reshape(-1, vocab), y.reshape(-1)) - Fobj.backward() - opt.step() - return [z.detach() for z in zs], (E / (args.B * args.T)).item() + if args.scheme == 'jacobi': + for z in zs: z.requires_grad_(True) + opt = torch.optim.SGD(zs, lr=args.eta, momentum=args.mom) + for _ in range(args.K): + opt.zero_grad() + E = 0.0; prev = z0 + for z, b in zip(zs, blocks): E = E + 0.5 * ((z - b(prev, mask)) ** 2).sum(); prev = z + (E / NBT + beta * F.cross_entropy(readout(zs[-1]).reshape(-1, vocab), y.reshape(-1))).backward() + opt.step() + return [z.detach() for z in zs] + order = range(args.L) if args.scheme == 'gsf' else range(args.L - 1, -1, -1) + bufs = [torch.zeros_like(z) for z in zs] + for _ in range(args.K): + for l in order: + g = local_grad(zs, z0, l, beta, y) + bufs[l].mul_(args.mom).add_(g) + zs[l] = (zs[l] - args.eta * bufs[l]).detach() + return zs -def dEdtheta(zs): - """dE/dtheta at fixed states (the local EP readout).""" - for p in blocks.parameters(): - if p.grad is not None: p.grad = None +def dFdtheta(zs, z0, y, beta, params): + """dF/dtheta at FIXED states (E-term local reads + beta*CE readout term for io params).""" E = 0.0; prev = z0 for z, b in zip(zs, blocks): E = E + 0.5 * ((z - b(prev, mask)) ** 2).sum(); prev = z - (E / (args.B * args.T)).backward() - return [(p.grad.clone() if p.grad is not None else torch.zeros(1, device=dev)) for p in blocks.parameters()] + obj = E / NBT + beta * F.cross_entropy(readout(zs[-1]).reshape(-1, vocab), y.reshape(-1)) + gs = torch.autograd.grad(obj, params, allow_unused=True) + return [g if g is not None else torch.zeros(1, device=dev) for g in gs] -zp, Ep = relax(+args.beta) -zm, Em = relax(-args.beta) -gp, gm = dEdtheta(zp), dEdtheta(zm) -gep = [(a - b) / (2 * args.beta) for a, b in zip(gp, gm)] -print(f'nudged relax E+: {Ep:.3e} E-: {Em:.3e} (K={args.K}, eta={args.eta}, beta={args.beta})') - -# ---------------- gate ---------------- -names = [n for n, _ in blocks.named_parameters()] flat = lambda gs: torch.cat([g.reshape(-1) for g in gs]) -cos_all = F.cosine_similarity(flat(gep), flat(gbp), dim=0).item() -print(f'\nGATE cos(cascadeEP, BP) overall: {cos_all:.4f}') -for l in range(args.L): - idx = [i for i, n in enumerate(names) if n.startswith(f'{l}.')] - c = F.cosine_similarity(flat([gep[i] for i in idx]), flat([gbp[i] for i in idx]), dim=0).item() - r = (flat([gep[i] for i in idx]).norm() / flat([gbp[i] for i in idx]).norm()).item() - print(f' block {l}: cos {c:.4f} |EP|/|BP| {r:.3f}') +names = ([f'blk.{n}' for n, _ in blocks.named_parameters()] + + ([f'io.{n}' for n, _ in list(tok.named_parameters()) + list(pos.named_parameters())] if args.include_io else [])) + +cos_b, shr_b, blkcos = [], [], [[] for _ in range(args.L)] +t0 = time.time() +for bi in range(args.batches): + x, y = get_batch(bi) + z0 = (tok(x) + pos(torch.arange(args.T, device=dev))[None]).detach() + # BP reference (embedding z0 grad flows for io gate) + z0ref = tok(x) + pos(torch.arange(args.T, device=dev))[None] + zs = [z0ref] + for b in blocks: zs.append(b(zs[-1], mask)) + ce = F.cross_entropy(readout(zs[-1]).reshape(-1, vocab), y.reshape(-1)) + gbp = list(torch.autograd.grad(ce, gate_params, allow_unused=True)) + gbp = [g if g is not None else torch.zeros(1, device=dev) for g in gbp] + # two-phase + zp = relax(z0, y, +args.beta); zm = relax(z0, y, -args.beta) + gp = dFdtheta(zp, z0, y, +args.beta, gate_params) + gm = dFdtheta(zm, z0, y, -args.beta, gate_params) + gep = [(a - b) / (2 * args.beta) for a, b in zip(gp, gm)] + cos_b.append(F.cosine_similarity(flat(gep), flat(gbp), dim=0).item()) + shr_b.append((flat(gep).norm() / flat(gbp).norm()).item()) + for l in range(args.L): + idx = [i for i, n in enumerate(names) if n.startswith(f'blk.{l}.')] + blkcos[l].append(F.cosine_similarity(flat([gep[i] for i in idx]), flat([gbp[i] for i in idx]), dim=0).item()) + +cm, cmin = float(np.mean(cos_b)), float(np.min(cos_b)) +sm = float(np.mean(shr_b)) +print('per-block cos: ' + ' '.join(f'{l}:{np.mean(v):.4f}' for l, v in enumerate(blkcos))) +if args.include_io: + idx = [i for i, n in enumerate(names) if n.startswith('io.')] + print(f'io cos (last batch): {F.cosine_similarity(flat([gep[i] for i in idx]), flat([gbp[i] for i in idx]), dim=0).item():.4f}') +print(f'SUMMARY scheme={args.scheme} L={args.L} C={args.C} K={args.K} eta={args.eta} beta={args.beta} ' + f'io={int(args.include_io)} warp={args.warp} ckpt={args.ckpt or "-"} ' + f'cos={cm:.4f} cosmin={cmin:.4f} shrink={sm:.3f} t={time.time()-t0:.1f}s') |