Part of the Train a Transformer on Silicon series — start at #0. This one pays off the cliffhanger from #3, The Memory Is the Chip — its design is the parent this post grows from.
In #3 I packed the flat memory map down to a hard persistent floor of 51,456 words — the input, target, and ten weights liveness proved it could never recycle — and ended by saying a real language model’s embedding table and output projection would sit right on that floor: the floor is about to get taller. This is that post. It got taller by exactly 9,216 words, and I can tell you where every one went.
For orientation: this post (#4) extends the design from #3, which packed down the base decoder trainer built in #1 and #2.
TL;DR: The decoder finally has something to decode. This design wraps the frozen base block in token + positional embeddings, an LM head, softmax, and cross-entropy, and overfits a fixed next-token sequence on-chip. Cross-entropy — the standard language-model loss, measured here in nats (natural-log units) — falls 4.144 \to 0.0149 over 120 steps: from the uniform-guess ceiling \ln|V| = \ln 64 \approx 4.159 (the loss of a model that assigns equal probability to all 64 tokens) down to essentially memorized. It matches PyTorch autograd to 2.91\times10^{-16} and the RTL is bit-exact — all 13 gradient tensors match, 0 LSB of error. The cost: the persistent floor grows 51{,}456 \to 60{,}672 words (+17.9\%), and 10,240 of the new words scale with the vocabulary size |V|, only 1,024 with sequence length. That’s the lesson in one number — vocabulary dominates memory long before the hidden dimension D does — and it took exactly one new logic unit and zero new arithmetic (every operation already existed in the block).
The premise: a decoder with nothing to decode
Everything through #3 trained on a copy-shift task — learn to reproduce the input sequence shifted by one position — over raw vectors, with no tokens, no vocabulary, and no notion of “the next word.” The block did real work (attention, LayerNorm, GELU, backprop through all of it) but it was a language model with the language removed. Making it a real one needs three things bolted on: turning a token id into a vector (the embedding table), turning a vector back into a distribution over tokens (the LM head), and a loss to score that distribution (softmax + cross-entropy). The LM head is simplest understood as the mirror of the embedding table — a single output-projection matrix (sometimes called the unembedding) that maps the block’s hidden vector back to one score per token (embedding: token id → vector; LM head: vector → a score for every token). “Head” here is the machine-learning sense of a task-specific output layer bolted onto the end — not an attention head; there’s no attention in it at all.
The task mirrors the base trainer’s methodology. The base trainer overfit one fixed (input, target) pair; this design overfits one fixed token sequence — a period-8 motif of distinct tokens [3, 17, 42, 9, 55, 1, 30, 60] repeated to length T=16, each token’s target being the next in the stream (wrapping at the end). Every token has a unique successor, so this is a content-learnable next-token map, more language-like than pure positional memorization. The dataset is deliberately a single 16-token sequence — the point is to prove the hardware can drive a real cross-entropy loss to zero, so 120 gradient steps of pure overfitting is plenty. Initial cross-entropy is \ln|V|\approx 4.159; the model drives it to zero.
The mechanism: |V| = 64, an untied head, and no log hardware
Three design decisions carry this post, and none is the obvious default.
|V| = 64, because every GEMM dimension has to stay in the reachable set. The instinct is the smallest interesting vocab; I went to the top of the roadmap’s 32–64 range. (I write the vocabulary size as |V| — the cardinality bars mean “the number of distinct tokens,” 64 here — to keep it clearly separate from the value vector V in attention’s Q/K/V, a different quantity that shares the letter. In this tiny model |V|, the hidden dimension D, and the value width all happen to equal 64, which is exactly why the bars help.) Every new matrix multiply — the logits L = X_2 W_u, and its gradients dW_u = X_2^\top\, dL, dX_2 = dL\, W_u^\top, dE_{tok} = OH^\top dX (where OH is the one-hot input matrix, W_u the LM-head weight, and E_{tok} the token-embedding table) — lands with all its contraction and output dimensions inside the set \{16, 64, 256\} the multiplier already handles. Picking |V|=32 would inject a brand-new dimension into the multiplier’s specialization surface — and that mattered here, because the Genus mapping defect I hit mid-project (the one #3 works around with defensive RTL) was a multiplier specialized to exactly that set. |V|=64 introduces zero new specialization surface, so #3’s fixed-netlist story carries straight over — and makes each vocab table exactly one attention weight matrix, 4,096 words.
The LM head is untied. W_u is a separate matrix — the output projection that maps a hidden vector to per-token scores — not the embedding table transposed. Weight-tying (sharing one matrix for input embedding and output projection) is its own future design and needs an untied baseline to improve on; untied is also simpler in RTL — pure one-hot lookup plus a plain forward GEMM, with no shared transposed access and no merging of two gradients into one table (tying’s port-contention problem — not today).
Cross-entropy needs no log hardware — the beat I like most. The scary part of cross-entropy is the logarithm, but the gradient of cross-entropy with respect to the logits is just P \,{-}\, \text{onehot}(y) — here (P \,{-}\, OH_y)\cdot K_{CE}, where P is the softmax probabilities and OH_y the one-hot target. In hardware that’s the exact same elementwise subtract-and-scale the base trainer used for its mean-squared-error gradient (X_2 \,{-}\, Tgt)\cdot K_{LOSS}: same unit, different operands. So the whole training path is bit-exact and log-free; a logarithm appears only in the reported loss scalar, computed in float from the hardware-exact probabilities. No log lookup table ever touches the datapath.
There’s exactly one new piece of hardware: softmax_full.v. The block already contains a causal softmax (the softmax unit inside attention, which masks out future positions and normalizes over at most 16 columns). The LM head needs a full softmax — no mask, normalizing over all 64 vocabulary columns — so softmax_full is that same unit with the mask and zero-fill removed, plus one fixed-point trick. That is why the die shows two softmax blocks: the inherited masked softmax for attention, and the new unmasked softmax_full for the head. Normalizing over 64 columns instead of ≤16 pushes the near-uniform row sum to ≈ 64, whose reciprocal falls outside the frozen reciprocal-LUT’s ~16 ceiling — the golden model caught it with an assert. The fix uses a fact about softmax: subtracting a constant from every logit changes nothing about the probabilities, only the fixed-point range. So softmax_full subtracts \max + \texttt{SM\_MARGIN} with \texttt{SM\_MARGIN} = \ln(|V|/8), provably bounding the row sum below 16.0. Golden and RTL subtract the identical constant, so they stay bit-exact.
And here’s the reuse beat that closes the loop with #3. Remember dxs — the dead store liveness flagged, 1,024 words written but never read? In the base trainer the block’s input was fixed, so that gradient was discarded. In this design the input is computed from embeddings, so the same gradient is exactly what the tables need. The input gradient is dX = dX_1 + dxs — the sum of the block’s own input gradient dX_1 and that once-dead store dxs — scattered back into the token and positional tables. #3’s liveness pass found the waste; #4 turns it into the plumbing. The dead store was never dead — it was early.
Verification: does the vocabulary actually learn?
Same protocol as always — golden model first, then the RTL to the last bit.
Golden vs. autograd. A full float forward-and-backward of this whole design — embeddings, block, head, cross-entropy — versus torch autograd on all 13 gradient tensors (the three new ones — the token-table gradient dE_{tok}, the positional-table gradient dE_{pos}, and the LM-head gradient dW_u — included alongside the ten inherited from the base trainer). Loss agrees to |\Delta| = 8.88\times 10^{-16}; the worst gradient mismatch is 2.91\times 10^{-16}, comfortably under the 10^{-9} gate standard held since the base trainer.
Fixed-point learning. In Q16.16 over 120 steps, cross-entropy falls 4.144 \to 0.0149 — from the uniform-guess ceiling \ln 64 down to essentially memorized. This is the fixed-point hardware trajectory compared against a float software reference: the float reference tracks it (4.135 \to 0.0128) with tight agreement at both ends (start \Delta\approx 0.009, tail \Delta\approx 0.002); the mid-trajectory max gap of ~1.2 is the expected horizontal-shift artifact of comparing two curves mid-descent. The rigorous gate is the machine-precision one above; this one confirms the thing actually learns the next-token task.
RTL, bit-exact. The 1-step gate: all 13 tensors EXACT, 0 tolerance hits, the new tables included. The 120-step trajectory gate: PASS. Hardware and golden agree to the last bit, start to finish.
The war story: the 8th unit that decoded to 0
The first RTL sim hung. Not failed — hung, dead still at pc=18 (the softmax_full op), with the microsequencer — the small state machine that issues one functional-unit operation per macro-instruction — parked in WAIT.
The cause was one bit of width. The base trainer’s emitter declares the unit-select field — sel, the value that picks which functional unit runs each op — as reg [2:0] sel: three bits, enough for units 1 through 7. This design adds the 8th unit, whose opcode U_SMF (the softmax_full selector) = 8, which doesn’t fit in three bits: it silently truncates to 0 in the case assignment. So sel == U_SMF is never true, the unit never starts, and the sequencer waits forever for a done-signal that can’t come.
The fix is trivial — widen the declaration to reg [3:0] sel — but the lesson isn’t. When you add the 8th unit to a 7-unit machine, the decoder width is part of the interface. A truncated constant is one of the nastiest RTL failure modes precisely because it produces nothing: no build warning, no wrong answer, no compare mismatch — just a machine that stops. (Third time in this series the plumbing lied while the design math was innocent. When the machine goes quiet, suspect the wiring before the arithmetic.)
Physical results: the floor got taller, and it’s mostly vocab
The synthesis / place-and-route / DRC chain is clean. Unlike #3 — which changed only memory and held logic, timing, and power flat — this design is additive: it bolts a whole vocabulary path onto the frozen block, so logic, cells, die, and power all legitimately grow. The new logic is small (the one softmax_full unit plus the cross-entropy subtract, which reuses an existing unit); the growth you see below is mostly memory. The story here isn’t a shrink; it’s where the growth landed.
| metric | #3 (parent) | this post (#4) | delta |
|---|---|---|---|
| persistent floor | 51,456 words | 60,672 words | +9,216 (+17.9%) |
| total words | 90,896 | 103,184 | +12,288 (+13.5%) |
| address width | AW18 (18-bit) | AW18 (18-bit) | unchanged |
| SRAM (modeled) | 2.91 Mbit | 3.30 Mbit | +0.39 Mbit |
| program length | 50 ops | 61 ops | +11 |
| logic cell area | 117,247 µm² (54,031 cells) | 136,905 µm² (65,694 cells) | +16.8% / +21.6% cells |
| die area | 4.55 mm² | 5.08 mm² (2350 × 2160 µm) | +11.6% |
| power (vectorless) | 1.39 mW | 1.656 mW | +19% |
| setup slack @ 100 MHz | +2.358 ns | +1.798 ns | MET |
| hold slack | +0.067 ns | +0.064 ns | MET |
| DRC | 0 Viols | 0 Viols | clean |
(Address width AW18 = an 18-bit word address, i.e. up to 2¹⁸ = 262,144 addressable words. Setup slack stays positive — timing MET — even with the added unit and the extra ops: the deeper logic did not eat the margin at 100 MHz.)
The added logic is exactly one softmax unit: the new softmax_full lands at 19,476 µm² (10,959 cells), almost the same size as the causal softmax it was cloned from — the difference between them is only the removed mask and zero-fill logic. The vocabulary cost you see in logic is one unit; the cost you see in memory is the tables. And one timing note, told plainly: the first routed database reported setup −6.415 ns on a path out of the reset. That looked like a real violation, but it wasn’t. rst is a synchronous reset that is asserted only at power-up — held for ≥4 clocks, then deasserted into idle — while start is pulsed thousands of cycles later. A timing path through rst can therefore never be exercised at speed during training, so it is a false path, not something to buffer or re-place. Telling the tool so (set_false_path -from rst, cross-checked against a 4-cycle multicycle that gives the identical +1.798 ns) moves the reported worst path to the real reciprocal datapath, with DRC still 0 and no RTL changed. Signoff has judgment calls; this was one, and it’s documented.
The persistent floor — un-recyclable state, read at step start and written at step end — grows 51{,}456 \to 60{,}672 words (a net +9{,}216: 11,264 new persistent words minus 2,048 that the base trainer’s now-transient input and target vacated). Now decompose it. Of the new persistent state, 10,240 words scale with the vocabulary size |V| — the token table E_{tok} (4,096), the LM-head matrix W_u (4,096), and the two one-hot inputs (2×1,024) — while only 1,024 scale with sequence length T (the positional table E_{pos}). Nothing that depends only on the hidden dimension D changed. At |V|=D=64 the vocab tables each already rival an attention weight matrix and grow linearly in |V|, while positional and width costs sit still. Vocabulary dominates the memory of a language model long before the hidden dimension does. AW18 (the 18-bit address width) still holds it — but the pressure points at |V|.
The lesson
From the datapath’s point of view, a real language model is a shockingly small delta: one new functional unit, a fistful of GEMMs that reuse known dimensions, a cross-entropy gradient that’s the same subtract the old mean-squared-error loss used, and a dead store that was already the gradient the embeddings needed. The math barely changed; the memory did — the floor #3 said allocation couldn’t touch got 17.9% taller, almost all of it vocabulary. To know what eventually breaks an on-chip language model, don’t look at the compute — look at the token tables, and watch them scale with |V|.
One design choice is flagged honestly for review: this design generalizes #3’s early-SGD reschedule to a shared gradient. The positional table’s gradient is the input gradient dX, which the token-embedding scatter also consumes — so E_{pos}‘s update is placed after the last read of dX, not after its own definition. Legality (no read of a weight or its gradient after its update) is assert-checked and passes, but it’s a genuine deviation from #3’s one-consumer-per-gradient assumption.
Honest caveats
- Setup slack rests on a corrected reset constraint. The routed design is DRC-clean at 100 MHz with setup +1.798 / hold +0.064 ns MET. The setup figure required declaring the synchronous
rsta false path — a genuine SDC over-constraint, not a datapath fix:rstis held ≥4 clocks and deasserted into idle, so no real timing path runs through it at speed, and a 4-cycle multicycle gives the identical +1.798 ns as a cross-check. Valid under this trainer’s documented reset protocol; a future integration that drivesrstas a fast single-cycle signal should revisit it. - The vectorless power (1.656 mW) is a synthesis estimate, not activity-based. A comparable SAIF/activity number (#3’s was 2.05 mW) wasn’t run for this design and isn’t invented here. Power grew ~19% over #3, as expected from real added logic — this is not a memory-only move.
- Tiny model, educational PDK. |V| = D = 64, one layer, one head, T = 16, generic 45 nm — a mechanism demo overfitting one fixed sequence, not signoff-grade.
- The loss log is host-side. The training path is log-free and bit-exact; the reported cross-entropy applies
log()in float to the hardware-exact probabilities. PinningPfor a live on-chip readout would add ~1,024 words, not included here. - SRAM is modeled, not compiled. Same hand-built LEF macro as #3; CACTI cross-checks it at ~1.5–2× conservative. Die claims stay labeled modeled-not-compiled.
SM_MARGINis a design constant. \ln(|V|/8) is invariant for the probabilities but tuned to the fixed-point range; change |V| and the generator regenerates it.
Next up — #5: what does the backward pass actually cost? Every post so far has been about area — how much memory the step needs. But a training step is forward and backward, and the backward pass is where most of the FLOPs and memory traffic live. I01 tries to measure it directly: of everything this chip does in one step, how much is the price of learning rather than inferring? I don’t have the number yet — that’s the post.
The full roadmap lives in Part 0 — the hub for the series.
If you build hardware, do ML, or work in EDA — follow along. And if you catch a mistake, tell me. For a series like this, that’s the best thing that can happen.
Leave a Reply