Mustang Wang

Tag: memory

  • Train a Transformer on Silicon: #4 Giving It a Vocabulary

    The chip could train, but it had nothing to decode. This design adds token and positional embeddings, an LM head, softmax, and cross-entropy on top of the packed memory map from #3 — and the toy copy-task becomes a real, if tiny, language model that overfits a next-token sequence on-chip. Cross-entropy falls 4.144 to 0.0149…

  • Train a Transformer on Silicon: #3 The Memory Is the Chip

    v0’s die was 85% SRAM. H01 packs the flat memory map with liveness analysis and an early-SGD reschedule — 145,952 words down to 90,896, the die from 6.99 to 4.55 mm², 35% smaller — without changing one logic gate. The compute was never the problem; the memory was.