2026-07-15T09:29:45.734111
image/svg+xml
Matplotlib v3.10.8, https://matplotlib.org/
One transformer block, op by op
O
L
M
o
2
o
r
d
e
r
i
n
g
R
M
S
N
o
r
m
a
f
t
e
r
e
a
c
h
s
u
b
l
a
y
e
r
,
i
n
s
i
d
e
t
h
e
r
e
s
i
d
u
a
l
.
=
5
1
2
=
8
=
6
4
=
2
5
6
=
1
4
0
8
—
C
H
h
T
h
d
f
f
attention sublayer
i
n
p
u
t
z
B
T
C
(
,
,
)
residual rail
m
a
t
m
u
l
:
→
3
W
C
C
q
k
v
(
,
,
3
)
s
p
l
i
t
i
n
t
o
,
,
B
T
C
q
k
v
—
q
k
v
R
M
S
N
o
r
m
q
R
M
S
N
o
r
m
k
s
p
l
i
t
h
e
a
d
s
(
=
8
)
H
s
p
l
i
t
h
e
a
d
s
(
=
8
)
H
s
p
l
i
t
h
e
a
d
s
(
=
8
)
H
RoPE
RoPE
θ
=
5
⋅
1
0
5
m
a
t
m
u
l
=
⋅
1
/
S
q
k
h
⊤
√
d
(
,
,
,
)
B
H
T
T
causal mask
softmax over keys
m
a
t
m
u
l
=
y
A
v
(
,
,
,
)
B
H
T
h
d
merge heads
(
,
,
)
B
T
C
m
a
t
m
u
l
:
→
W
C
C
p
r
o
j
R
M
S
N
o
r
m
a
t
t
n
◄ OLMo2: norm on the sublayer OUTPUT,
not on its input
+
z
0
SwiGLU sublayer
m
a
t
m
u
l
:
→
W
C
h
1
f
f
m
a
t
m
u
l
:
→
W
C
h
3
f
f
h
f
f
=
1
4
0
8
SiLU
⊙ elementwise gate
m
a
t
m
u
l
:
→
W
h
C
2
f
f
R
M
S
N
o
r
m
f
f
◄ norm after the sublayer, again
+
o
u
t
p
u
t
o
n
e
f
u
l
l
d
o
n
e
z
f
0
0
—
l
w
e
i
g
h
t
×
a
c
t
i
v
a
t
i
o
n
m
a
t
m
u
l
7
l
o
g
i
c
a
l
(
f
u
s
e
s
,
,
)
;
c
r
o
s
s
b
a
r
-
m
a
p
p
a
b
l
e
o
n
a
n
a
l
o
g
h
a
r
d
w
a
r
e
—
W
W
W
W
q
k
v
q
k
v
a
c
t
i
v
a
t
i
o
n
×
a
c
t
i
v
a
t
i
o
n
m
a
t
m
u
l
2
(
a
n
d
)
;
c
o
m
p
u
t
e
d
o
n
t
h
e
f
l
y
:
t
h
e
n
o
n
-
c
r
o
s
s
b
a
r
p
a
r
t
o
f
a
t
t
e
n
t
i
o
n
—
q
k
A
v
⊤
RMSNorm (learned gain)
elementwise / reshape
N
o
b
i
a
s
e
s
a
n
y
w
h
e
r
e
i
n
t
h
e
n
e
t
w
o
r
k
.
T
h
e
1
/
s
c
a
l
e
i
s
f
o
l
d
e
d
i
n
t
o
;
m
a
s
k
a
n
d
s
o
f
t
m
a
x
r
u
n
a
l
o
n
g
t
h
e
k
e
y
a
x
i
s
.
√
h
S
d
h
C
C
f
f
=
1
4
0
8
≈
8
/
3
,
r
o
u
n
d
e
d
u
p
t
o
a
m
u
l
t
i
p
l
e
o
f
6
4
.
Q
K
-
n
o
r
m
a
c
t
s
o
n
t
h
e
f
u
l
l
w
i
d
t
h
,
b
e
f
o
r
e
t
h
e
h
e
a
d
s
p
l
i
t
.
T
h
e
b
o
x
e
d
p
i
p
e
l
i
n
e
,
i
n
p
u
t
t
o
o
u
t
p
u
t
,
i
s
e
x
a
c
t
l
y
o
n
t
h
e
c
o
m
p
a
n
i
o
n
p
a
g
e
;
t
h
e
s
t
a
t
e
l
i
v
e
s
o
n
t
h
e
r
e
s
i
d
u
a
l
r
a
i
l
.
z
z
f
z
0
0
l
l