Mustang Wang

Category: Research

  • Train a Transformer on Silicon: #3 The Memory Is the Chip

    v0’s die was 85% SRAM. H01 packs the flat memory map with liveness analysis and an early-SGD reschedule — 145,952 words down to 90,896, the die from 6.99 to 4.55 mm², 35% smaller — without changing one logic gate. The compute was never the problem; the memory was.

  • Train a Transformer on Silicon: #2 A Decoder Is Just an Encoder with a Mask

    v0 is done: a nanoGPT-faithful decoder layer that trains itself — forward, backprop, and SGD update — bit-exact against a golden model and taken to a clean 45 nm layout. The compute fit in 0.117 mm² of logic; the memory took 85% of the die.

  • Train a Transformer on Silicon: #1 Backprop on a Chip

    Most AI chips run a model someone else trained; this one does the training itself — forward pass, backpropagation, and weight update, in fixed-point RTL, all the way down to a 45 nm layout.

  • Publication-Quality Chip Layout Figures from Innovus, Headlessly

    Innovus has no vector export and its GUI screenshots don’t scale. Here’s a fully scripted pipeline: a headless 16,000-pixel render, block coordinates pulled from the P&R database, a zero-residual pixel-to-micron calibration, and a matplotlib wrapper that turns it all into a self-describing PDF/SVG figure.

  • One Prompt, One Complete IEEE Transactions Project with Claude Code

    A single copy-pasteable prompt that makes Claude Code scaffold a complete IEEE Transactions LaTeX project: main manuscript, S-numbered supplementary materials, a shared author file, and a full revision workflow – with no dependencies beyond a TeX distribution and make.

  • Train a Transformer on Silicon: #0 The Genesis

    Start here — the series hub A short intro to a series where I build a chip that trains a transformer — and document the whole thing, mistakes included. What this is I’m building a digital chip that trains a transformer on-chip — not just runs it, but actually learns: forward pass, backpropagation, and weight…

  • Cadence IC25.1: “Failed to find licenses Virtuoso_ADE_Artist” and How to Fix the Virtuoso Hotfix Install

    I spent a good chunk of time upgrading Cadence Virtuoso’s optimizer (AOP) from IC25.1 base to the ISR4 hotfix. What should’ve been a routine update turned into two rabbit holes: the optimizer was checking for a license that doesn’t exist on most servers, and the hotfix installer kept dying on symlinks that point to each…

  • Backprop is under the hood

    In 1, the division of . 2 To illustrate why the dot products get large, assume that the components of q and k are independent random variables with mean  and variance . Then their dot product, , has mean  and variance . Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all…

  • Weight-sparse transformers have interpretable circuits

    Weight-sparse transformers have interpretable circuits1234: Train sparse model on weights and pruning to explain interpretability, find connections between sparse and dense models. Transform: Encoder, Decoder, from tokens to embeddings to tokens | from electricity to magnetics to electricity | Fourier Transform | LLM Visualization Overall Setup | Superposition 5 Sparse Model Training sparse models contain…

  • Proximal Policy Optimization Keynotes

    This post focuses on understanding proximal policy optimization, which is an on-policy policy gradient reinforcement learning algorithm. Clipping Clipped Surrogate Objective Plot1 There are 2 subfigures for this plot, illustrating 2 conditions where and , the case of is ignored as the clipping surrogate objective would be for both and The clipped surrogate objective function…

  • Connected to Jim Keller

    In a surprising and truly exciting development, Jim Keller, the legendary “Silicon Sage” whose innovations have profoundly shaped the tech world, accepted my LinkedIn invitation this morning. What makes this particularly remarkable is that we’ve never actually met in person or professionally intersected in any way. It’s a testament to the open and interconnected nature of…

  • Virtuoso Annotation

    This post discusses briefly about annotation on schematics when simulation has been performed, it is not going to be a detailed summary but to get you started on annotation and ballons. Before going further, please make sure you know how to do a simulation prior to annotation configuration. In ADE Explorere’s schematic view, go to…

  • Compilation of Virginia Tech Forms

    We’ve all been there. That sinking feeling when you realize you need a specific form from the university – a Plan of Study update, the dreaded Qualifying Exam application, a Change of Campus request, or perhaps a Graduate School petition. Your journey likely begins with a hopeful dive into the department website, only to be…

  • Rook2Vet4Word2Vec

    The goal of this blog is to help readers understand what word2vec means so that rookie like you could become a veteran on the concept of embeddings training process. I’ve tried to find resources to understand how word2vec actually works. What surprises me the most is that although it looks really simple in terms of…

  • Research Papers Manifestation

    As a researcher myself, reading papers are one part of the job. Although there are some resources and materials present ways to read papers, sometimes I still don’t feel that they are pracitical and cannot help me digest the papers properly. Before going further, I need to recommend the book ” How to read a…