diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-05 05:09:36 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-05 05:09:36 -0500 |
| commit | 75ff326dcb40cd960abd56e9c9c18a45d9e5e2c2 (patch) | |
| tree | 575cbed6adc41c845685c7286437c4620f040cdb /ep_run/sdpa_gate.log | |
| parent | cbecb171b1af77fe6510fd60f1dfa4c7938a20d6 (diff) | |
--sdpa: fused flash attention in the no_grad relax loop — 1.45x free phase, z* parity 4e-7
Scoped via blk._sdpa set only inside relax()'s loop (grad paths jvp/vjp/resreg
keep the manual attention: no forward-mode-through-flash risk). Combined with
--holofast: ~1.51x full-step exact-math tier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'ep_run/sdpa_gate.log')
| -rw-r--r-- | ep_run/sdpa_gate.log | 4 |
1 files changed, 4 insertions, 0 deletions
diff --git a/ep_run/sdpa_gate.log b/ep_run/sdpa_gate.log new file mode 100644 index 0000000..88bb685 --- /dev/null +++ b/ep_run/sdpa_gate.log @@ -0,0 +1,4 @@ +manual: res=9.981e+00 val=3.1589 relax150=1.650s + sdpa: res=9.981e+00 val=3.1056 relax150=1.140s +z* rel-diff=3.99e-07 speed=1.45x +SDPA_GATE_DONE |