Category: AI
-
Train a Transformer on Silicon: #3 The Memory Is the Chip
v0’s die was 85% SRAM. H01 packs the flat memory map with liveness analysis and an early-SGD reschedule — 145,952 words down to 90,896, the die from 6.99 to 4.55 mm², 35% smaller — without changing one logic gate. The compute was never the problem; the memory was.
-
Train a Transformer on Silicon: #2 A Decoder Is Just an Encoder with a Mask
v0 is done: a nanoGPT-faithful decoder layer that trains itself — forward, backprop, and SGD update — bit-exact against a golden model and taken to a clean 45 nm layout. The compute fit in 0.117 mm² of logic; the memory took 85% of the die.
-
Train a Transformer on Silicon: #1 Backprop on a Chip
Most AI chips run a model someone else trained; this one does the training itself — forward pass, backpropagation, and weight update, in fixed-point RTL, all the way down to a 45 nm layout.
-
One Prompt, One Complete IEEE Transactions Project with Claude Code
A single copy-pasteable prompt that makes Claude Code scaffold a complete IEEE Transactions LaTeX project: main manuscript, S-numbered supplementary materials, a shared author file, and a full revision workflow – with no dependencies beyond a TeX distribution and make.
-
Train a Transformer on Silicon: #0 The Genesis
Start here — the series hub A short intro to a series where I build a chip that trains a transformer — and document the whole thing, mistakes included. What this is I’m building a digital chip that trains a transformer on-chip — not just runs it, but actually learns: forward pass, backpropagation, and weight…
-
Backprop is under the hood
In 1, the division of . 2 To illustrate why the dot products get large, assume that the components of q and k are independent random variables with mean and variance . Then their dot product, , has mean and variance . Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. “Attention is all…
-
Weight-sparse transformers have interpretable circuits
Weight-sparse transformers have interpretable circuits1234: Train sparse model on weights and pruning to explain interpretability, find connections between sparse and dense models. Transform: Encoder, Decoder, from tokens to embeddings to tokens | from electricity to magnetics to electricity | Fourier Transform | LLM Visualization Overall Setup | Superposition 5 Sparse Model Training sparse models contain…
-
Proximal Policy Optimization Keynotes
This post focuses on understanding proximal policy optimization, which is an on-policy policy gradient reinforcement learning algorithm. Clipping Clipped Surrogate Objective Plot1 There are 2 subfigures for this plot, illustrating 2 conditions where and , the case of is ignored as the clipping surrogate objective would be for both and The clipped surrogate objective function…
-
Rook2Vet4Word2Vec
The goal of this blog is to help readers understand what word2vec means so that rookie like you could become a veteran on the concept of embeddings training process. I’ve tried to find resources to understand how word2vec actually works. What surprises me the most is that although it looks really simple in terms of…
-
Practical Tips for Machine Learning Research and Development
Machine learning projects require a different approach compared to traditional software development. After working on numerous ML projects, I’ve compiled practical tips to streamline the research and development process. This guide doesn’t require specific ML knowledge but focuses on effective workflows and organizational strategies. I tried to find online mainly through Google to find articles…
-
Understanding Multi-headed Attention with Grep – A Simple Guide
Have you ever needed to find something in your code but weren’t sure exactly where to look? That’s where grep comes in handy. And believe it or not, this everyday coding tool can help us understand one of the most powerful concepts in AI: multi-headed attention. Starting with the Basics Imagine you’re looking for everything…
-
How AI Helped Me Make Smart Decisions About Tire Repair
Recently, I found myself in a common but stressful situation – a nail in my tire. What started as a potentially frustrating experience turned into an interesting lesson in how AI can help with everyday decisions. As someone who relies heavily on their car, I needed to find a solution quickly, but I also wanted…
-
Accelerate Pytorch
Use Pytorch to accelerate with GPU or apple silicon with Metal
-
Mastering the Gumbel-Softmax Trick: Turning Hard Decisions into Smooth Learning
Introduction Imagine teaching a computer how to choose between different options — like picking the best movie recommendation or deciding which move to make in a game. These choices often boil down to picking one option from several, a process called categorical sampling. However, this kind of decision-making poses a challenge when training machine learning models…
-
Interact with Python virtual environment
Using python for mahcine learning projects is totally different from utilizing the language for productivity on daily tasks. Python official website provides the documentations for venv1. This post2 contains a comprehensive guide on how to use the virtual environment. There are lots of tutorials on how to setup, manage and dispose a python virtual environment…