|
Patch HRM Block with Lipschitz-bounded ops:
- attention → cosine-normalized softmax attn
- SwiGLU → OrthLinear (Cayley + weak diag scale) + MaxMin + OrthLinear
- rms_norm + add → weighted residual (1-σ(w))·h + σ(w)·f(h)
- Weak orthogonality: diag(s) with s_i ∈ [0.95, 1.0] for compression directions
Keeps HRM ACT framework + H_level/L_level + cycles unchanged.
Predicted +5-7pp vs SRM v1 (codex Q5 decomp):
+1.5-2.5 (remove ReLU rank-kill via MaxMin)
+2.0-3.0 (remove AOL attenuation via Cayley)
+1.0-1.5 (orthogonal residual flow)
Also adds: train_hrm_orth.py trainer, SRM v1 run logs, .gitignore ckpts/.codex
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
