Tag: digital design
-
Train a Transformer on Silicon: #2 A Decoder Is Just an Encoder with a Mask
v0 is done: a nanoGPT-faithful decoder layer that trains itself — forward, backprop, and SGD update — bit-exact against a golden model and taken to a clean 45 nm layout. The compute fit in 0.117 mm² of logic; the memory took 85% of the die.
-
Train a Transformer on Silicon: #1 Backprop on a Chip
Most AI chips run a model someone else trained; this one does the training itself — forward pass, backpropagation, and weight update, in fixed-point RTL, all the way down to a 45 nm layout.
-
Train a Transformer on Silicon: #0 The Genesis
Start here — the series hub A short intro to a series where I build a chip that trains a transformer — and document the whole thing, mistakes included. What this is I’m building a digital chip that trains a transformer on-chip — not just runs it, but actually learns: forward pass, backpropagation, and weight…