diff options
| author | YurenHao0426 <blackhao0426@gmail.com> | 2026-02-09 12:28:55 -0600 |
|---|---|---|
| committer | YurenHao0426 <blackhao0426@gmail.com> | 2026-02-09 12:28:55 -0600 |
| commit | ef678d2e1ba70b1a9dadb78c73ed372f986aea13 (patch) | |
| tree | b90b5c53960b22a6a5498ca69fbfffad7e1832f8 /scripts/slurm_train.sh | |
| parent | 93d77b197d457b1fdfa7341ecd59fc460b20d6b1 (diff) | |
Fix NLL double-shift bug and head weight init
- NLL loss was shifting labels twice (olmo_labels already shifted,
then code did logits[:,:-1] vs labels[:,1:]). Fixed in 9 locations:
trainer, pipeline, olmo_graph, sanity_check, eval.
- Head U/V weights init with std=0.01 (was Kaiming ~5.7 std) so
UV^T≈0 at init, ensuring Z≈logit_bias=15 and A≈0.953.
- Updated SVD rank test to subtract logit_bias before checking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Diffstat (limited to 'scripts/slurm_train.sh')
0 files changed, 0 insertions, 0 deletions