Start here — the series hub
A short intro to a series where I build a chip that trains a transformer — and document the whole thing, mistakes included.
What this is
I’m building a digital chip that trains a transformer on-chip — not just runs it, but actually learns: forward pass, backpropagation, and weight updates, all in hardware — and taking it through the full design flow to a chip layout. Then I’m writing it up as a series, one step at a time.
This post is the map. It’s short on purpose: what I’m doing, why, what to expect, and where it’s headed.
Why I’m doing it
Two reasons.
First, curiosity. Nearly all the “AI hardware” you read about is inference — running a model someone else already trained. The training side — gradients, backprop, weight updates in silicon — gets far less attention and is the harder, more interesting problem. I wanted to actually build it and see what breaks.
Second, learning in public. The best way I know to really understand something is to build it end-to-end and explain it. So instead of a finished, polished result, this is the process — including the parts where I get it wrong and have to figure out why. That’s where the real learning is anyway.
What to expect from the series
A few promises about how I’ll write these:
- One idea per post. Each post takes the design a single step further, and shows what that change costs and buys.
- Numbers and pictures, not just prose. Area, timing, power, waveforms, layout shots — the actual evidence.
- The mistakes stay in. The router that hung for hours, the “clean” report that wasn’t, the block placed slightly in the wrong spot — those stories are the most useful part.
- Honest about what it is. It’s a small design on a generic, educational process, and it isn’t a fabricated chip. I’ll say so every time. The goal is to demonstrate the mechanism clearly, not to claim a record.
Series index
Plans change, but here’s the direction. I’ll link each post here as it goes up (this Part 0 is the hub — bookmark it):
- #0 — The Genesis (you are here)
- #1 — Backprop on a Chip: the premise, the approach, and how I keep myself honest (checking the math before touching the hardware).
- #2 — A Decoder Is Just an Encoder with a Mask: the minimal GPT-style trainer, and what actually differs between the two.
- #3 — The Memory Is the Chip: 85% of the v0 die is SRAM — liveness analysis, buffer reuse, and how much of it I can give back.
- #4 — Giving It a Vocabulary: adding a real language-model head so it predicts tokens.
- #5 — The Number That Checked Itself: the SRAM density under every area figure so far, and the admission that I chose it rather than measured it.
- #6 — Porting a Memory Compiler to a PDK With No Memory: getting an open-source memory compiler running in a kit that ships no SRAM, and the bug that shorted every wordline to every bitline.
- #7 — What a Real Macro Costs: the first measured density, from a compiled 1024×32 macro, about 6.3× the number I made up.
- #8 — 143 Real Macros in the Trainer: the trainer rebuilt on compiled memory, and an honest account of the one check that didn’t run.
- #9 — Eight Times My Tools Told Me Everything Was Fine: eight checks that passed while the thing they checked was broken, and the rules that came out of it.
- #10 — Five More, and the Cap I Didn’t See: five more false greens, including a default result cap that quietly censored a week of counts.
- #11 — Ten Rounds Chasing an X: a gate-level simulation that came back all-X, and the control-first rounds that named what was behind it.
- #12 — Two Bugs Wearing One Coat: the all-X turns out to be two unrelated faults, and pulling them apart is what finally turns the gate green, bit for bit.
- #13 — The Cost of Training: what the training in a training chip costs, about 2× the silicon and 3× the cycles of inference, restated with the measured density.
- Later: scaling the model up, multi-head attention, stacking layers, and hardware-friendly training tricks (low-precision training, memory-efficient attention, and friends).
Who this is for
If you work in chip design, machine learning, or EDA — or you’re just curious what it takes to put learning itself into hardware — you’ll get something out of it. No assumption that you know all three; I’ll explain the crossover pieces as they come up.
And if you spot a mistake, please tell me. For a series like this, being corrected in public is a feature, not a bug.
Field notes from the series
Side write-ups on the tooling behind the series:
- Publication-Quality Chip Layout Figures from Innovus, Headlessly: how the die shots in this series get rendered — straight from the tool, no GUI.
- The Precision Cliff: how few bits can still train a transformer, from Q16.16 down to Q4.4.
Next up — #1: Backprop on a Chip.
Leave a Reply