s60.yaml « model « configs - zbp-scaling.git - Unnamed repository; edit this file 'description' to name the repository.

summary refs log tree commit diff

path: root/configs/model/s60.yaml

blob: ebccc0afa8cee82d73b4ee44fcaa5bf00a01675f (plain)

d: 512
layers: 8
heads: 8
vocab: 50257
seq_len: 1024
ffn_mult: 8/3        # SwiGLU: hidden = mult*d rounded to 64
pos: learned