Benchmarking 17 AI Models On Razavi's Analog Design Questions • Xiaomeng Wang✦王晓萌

Table of Contents

Can an LLM reason about an analog circuit?

Behzad Razavi asked exactly that in his The Analog Mind: Analog Design Experiments With AI columns in IEEE Solid-State Circuits Magazine. In Part 1¹ he put 30 one- and two-transistor questions to ChatGPT and graded them himself, 0–4 per question, for a total of 49/120. In Part 2² he handed 20 harder questions — ring oscillators, a StrongARM comparator, C²MOS dividers, an LNA, a TIA, LC and quadrature oscillators — to Gemini 3.1 Pro and scored 44/80.

Those are fascinating data points, but each column grades one model, one time, graded by one person. As someone who spends their days on analog and neuromorphic IC design (and their evenings arguing with these models about my own circuits), I wanted to know how fragile that picture is. So I reconstructed all 50 questions and re-ran them across 17 model configurations from four vendors — Anthropic (Opus 4.8, Sonnet 4.6, Haiku 4.5, Fable 5), OpenAI (gpt-5.5), Google (Gemini 3.1 Pro, 3.5 Flash, 2.5 Pro), and Zhipu (GLM-5.2) — sweeping reasoning-effort levels and even the invocation harness.

Here’s the leaderboard, and then I’ll unpack how it was built and why the “how you run it” part turned out to matter as much as the “which model” part.

Fair leaderboard: raw scores on Part 1 (/120) and Part 2 (/80), one representative high-effort config per model, anonymized 3-grader median.
Fair leaderboard of raw scores on Part 1 and Part 2 for one representative high-effort config per model

What I did differently

Two things.

Blind text, not schematics. Razavi fed his models the actual schematic images. I did not. I reconstructed each question as a sanitized text description — the question plus a plain-language circuit_description — and fed only that. No images, and no simulator access for the models. This is the single most important caveat in the whole study: my numbers are not a head-to-head with Razavi’s. His rows (ChatGPT 49/120, Gemini 3.1 Pro 44/80) use image input and his own grading; they’re reference anchors, not competitors on my leaderboard. As a rough sense of the modality gap: Razavi’s Gemini 3.1 Pro scored 44/80 on images, and the same model on my blind text — fairly graded — scored 49. A +5 shift, and small next to the effects below.

A controlled grader. This is the crux. Self-grading an LLM’s answer, or grading each model in its own separate pass, leaks enormous bias. Instead I anonymized every answer into per-question packets — opaque model codes (M01–M17), order rotated — and had three independent graders score each question, aggregated by median, strictly against Razavi’s published ground truth. The whole pipeline looks like this:

The pipeline: questions fan out to 17 model configs, answers are anonymized into per-question packets, three graders score each question, medians aggregate.
Benchmark and grading pipeline diagram

Why bother? Because before I did this, an ungraded-blind pass had graders disagreeing by up to 20 points on identical answers. In one case gpt-5.5’s Part 1 was scored 109 by one grader and 89 by another. Side by side, gpt-5.5 got a 1 for essentially the same NMOS-cascode answer that Opus got a 4 for. Anonymized per-question median grading collapses that; the residual grader noise is about ±3 points.

The headline

Fable 5 (high effort) topped the board at 108/120 on Part 1 and 60/80 on Part 2. Opus 4.8 sat right behind it (106/59 at medium effort, 109/57 at high). Those two are the only configs to clear the rest of the field on the hard Part 2 questions — and, as you’ll see below, the only two to match all three simulator-verified answers. Their 1-point gap is inside the ±3 grader noise, so I read them as co-leaders.

A couple of things jumped out:

The frontier “co-leaders” story dissolved under fair grading. Naively (self-graded), gpt-5.5 and Gemini looked tied. Fairly graded, gpt-5.5 (46 on Part 2) sits behind Gemini 3.1 Pro (49), and both trail the Fable 5 / Opus tier by roughly 10 points.
Part 1 barely discriminates. Routine one- and two-transistor analysis compresses every strong model near the top. It’s Part 2 — the hard, multi-stage questions — that actually spreads the field.

Part 1 vs Part 2: Part 1 compresses the field near the top; Part 2 spreads it and separates the Fable 5 / Opus tier.
Part 1 vs Part 2 scatter plot

Newer isn’t automatically better

Plotting score against release date is tempting but a little misleading — capability tier, not the calendar, drives the score. Gemini 2.5 Pro (a 2025 model) matches several 2026 flagships; Haiku 4.5 trails regardless of how recent it is. The two dashed squares are Razavi’s published paper baselines (image input, his grading).

Score vs release date, all 17 configs. Capability tier, not the calendar, drives the score. Dashed squares are Razavi’s published paper baselines.
Score versus model release date for all 17 configs

Reasoning effort: three vendors, three shapes

I swept reasoning effort where the vendor exposes it. There’s no portable “more thinking = more points” law. Opus rises then plateaus (Part 2: 51 → 59 → 58). Gemini 3.1 Pro is basically a wash (50 vs 49). gpt-5.5 climbs steadily (41 → 46). “Effort” is a vendor-specific dial, not a shared axis — worth remembering before you compare two vendors’ “high” settings as if they meant the same thing.

Part 2 score vs reasoning effort: three vendors, three shapes.
Part 2 score versus reasoning effort for three vendors

The harness mattered even more. Running the same Claude model through a subagent harness versus its native CLI moved the score by up to 23 points. Opus went from 41 on Part 2 via subagent to ~58 via CLI; Haiku gained +23 on Part 1. The subagent reasons more shallowly — it pattern-matches to half-remembered results and walks straight into the StrongARM and ring-oscillator traps that the CLI derives from scratch. That’s a sobering thought if you benchmark models through whatever agent scaffolding you happen to have lying around.

ngspice as the referee

Here’s the part I trust the most. Three Part 2 questions have unambiguous, simulable answers, so I settled them with ngspice (Level-1, 180 nm-ish) instead of trusting any grader:

Q3 — the ring oscillator dies. A 3-stage ring with one large load cap stops oscillating around 1 pF, because that dominant pole kills the loop gain. Right or wrong is a waveform, not an opinion.
Q7 — StrongARM initial gain drops. Sweeping the tail width, the latch’s initial gain falls monotonically ( $A_v \propto 1/\sqrt{I_{CM}}$ ): 3.74 → 2.93 as $W_7$ goes 4 → 32 µm.
Q8 — gain is flat vs $C_{P,Q}$ . A 9× change in that cap moves the gain only about −17%.

No grader can wiggle these, which makes them the most reliable signal in the whole study.

ngspice-refereed medians. Green never fills the Q21 cascode-trap column, and only Fable 5 and Opus fill the whole Q3/Q7/Q8 row.
ngspice-refereed medians for the simulator-verified questions

Only Fable 5 and Opus 4.8 scored 4 on all three. gpt-5.5 got Q8 but stumbled on the ring and the StrongARM. Gemini and Sonnet split. And then there’s the failure that no methodology choice could fix: Q21, Razavi’s Figure 16 “is this a cascode?” trap, scored 0 for every single one of the 17 configs. Every model, every vendor, every effort level, every harness — including the two leaders — confidently calls a source-follower-based stage a “cascode.” That’s not grading noise; it’s a genuine, shared structural blind spot.

Limitations (the honest part)

I controlled what I could and I’ll name what I couldn’t:

Cross-vendor effort isn’t calibrated. Each “high” is that vendor’s own setting; they aren’t the same amount of compute.
Each model runs in its own native CLI/API. You can’t run gpt-5.5 through the Claude CLI, so a small harness asymmetry survives no matter what.
The grader is itself an LLM. Anonymization blocks vendor favoritism, and the ngspice-anchored questions bound any systematic error — but it’s not a human referee.
Everything was blind text. No schematic images, no simulator in the models’ hands. This is not a replication of Razavi’s exact conditions.

Rough error bars: grader ±3, effort ±8, harness (Claude, pre-fix) up to ±23. Any ranking difference under ~3 points is noise. What’s robust is the Fable 5 / Opus lead, the Haiku floor, and the Q21/Q3/Q7/Q8 pattern.

Takeaways

Run and graded fairly, Fable 5 and Opus 4.8 (via CLI) are the strongest analog reasoners in this set — the only two to nail all three simulator-verified questions. The current frontier (Gemini 3.1 Pro, gpt-5.5) trails on the hard problems, a real capability floor persists (Haiku), and a universal blind spot (the cascode trap) survives everything I threw at it.

But the louder lesson is methodological. The invocation harness, the grader, and the effort setting each moved scores by margins that rival the model-to-model differences. Any single-configuration leaderboard — including, gently, the original columns — is fragile. The one signal that didn’t wobble was the ngspice-anchored questions. Which is exactly why, if you want to benchmark an “analog mind,” you should keep a simulator in the loop as the referee.

Print 🖨 eBook 📱

Benchmarking 17 AI Models on Razavi’s Analog Design Questions