Tag: layernorm
-
Train a Transformer on Silicon: #2 A Decoder Is Just an Encoder with a Mask
v0 is done: a nanoGPT-faithful decoder layer that trains itself — forward, backprop, and SGD update — bit-exact against a golden model and taken to a clean 45 nm layout. The compute fit in 0.117 mm² of logic; the memory took 85% of the die.