diff options
| author | yurenh <blackhao0426@gmail.com> | 2026-08-31 18:14:09 -0500 |
|---|---|---|
| committer | yurenh <blackhao0426@gmail.com> | 2026-08-31 18:14:09 -0500 |
| commit | 6a544fabfc2af22e4d5823410dd2387b5af89ea9 (patch) | |
| tree | 0abd67bdda420deed27428b621fb59db8be07f41 /configs/model | |
scaffold: model (OLMo2-ish + ZBP partition), trainer (DDP/config), data shards, bench
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe
Diffstat (limited to 'configs/model')
| -rw-r--r-- | configs/model/l350.yaml | 7 | ||||
| -rw-r--r-- | configs/model/m124.yaml | 7 | ||||
| -rw-r--r-- | configs/model/s60.yaml | 7 |
3 files changed, 21 insertions, 0 deletions
diff --git a/configs/model/l350.yaml b/configs/model/l350.yaml new file mode 100644 index 0000000..cd36f07 --- /dev/null +++ b/configs/model/l350.yaml @@ -0,0 +1,7 @@ +d: 1024 +layers: 24 +heads: 16 +vocab: 50257 +seq_len: 1024 +ffn_mult: 8/3 +pos: learned diff --git a/configs/model/m124.yaml b/configs/model/m124.yaml new file mode 100644 index 0000000..ad3b5fa --- /dev/null +++ b/configs/model/m124.yaml @@ -0,0 +1,7 @@ +d: 768 +layers: 12 +heads: 12 +vocab: 50257 +seq_len: 1024 +ffn_mult: 8/3 +pos: learned diff --git a/configs/model/s60.yaml b/configs/model/s60.yaml new file mode 100644 index 0000000..ebccc0a --- /dev/null +++ b/configs/model/s60.yaml @@ -0,0 +1,7 @@ +d: 512 +layers: 8 +heads: 8 +vocab: 50257 +seq_len: 1024 +ffn_mult: 8/3 # SwiGLU: hidden = mult*d rounded to 64 +pos: learned |