i made agi take the SAT (properly this time)
six months ago i made 20 AI models take the SAT. the results went around: Claude won with a 630, Chinese models tied for second, GPT-5.2 placed eighth. fun post. one problem: the harness was broken and most of the numbers were wrong.
this is the redo, built like i should have built it the first time. 20 models, two full official digital SAT practice tests (all 4 modules, math included, figures included), official College Board conversion tables, confidence intervals, robustness checks, and a contamination probe. every number below survives an audit.
tldr: the frontier is now scoring 1440-1485 — 95th-97th percentile among actual SAT takers. no model hit 1600. the reading section is basically solved (multiple 800s). math with figures is where they still bleed. and there’s a fat contamination asterisk on everything, which i’ll show you with data instead of hand-waving.
what was wrong with v1
i went through the old code line by line. the postmortem found, among other things:
- the answer parser took the first capital letter A-D anywhere in the response. a model that said “Because the passage says X, the answer is D” got graded as answering “B” (first capital = the B in “Because”). CoT answers were systematically mangled.
- questions were keyed by bare question number, and the two RW modules share numbers 1-33. module 1’s answers were re-graded against module 2’s answer key. every “66-question” run was 33 questions double-counted.
- API failures were silently recorded as wrong answers. Google’s “safety refusals” in the v1 post? mostly harness errors.
- scores were computed with a made-up linear formula (400 + accuracy × 400), not the College Board conversion tables. the real scale is nonlinear and the floor is 200, not 400.
- only reading/writing was tested. no math, no figures — the half of the test where models actually differ was skipped.
v1 said Claude Opus scored 630. with the same responses graded correctly, that number was garbage. i’m leaving the old post up (with a banner) because deleting your mistakes is how you never learn from them.
the setup, v2
everything is rebuilt from scratch:
- datasets: official College Board digital SAT practice tests #10 and #11 — full 120-year questions each (66 RW + 54 Math across 4 modules), extracted from the PDFs with a layout-aware parser, validated against two independent sources of the answer keys (the scoring PDF and the per-question rationale text), then human-spot-checked. figure questions carry their actual figure as an image.
- elicitation: chain-of-thought, one question per request, fresh context every time, answer demanded on a final
Answer: Xline. the parser accepts that line,\boxed{}, and “the final answer is X” — and if a model produces none of them, the response is recorded as unparseable (counts against the model, like a blank bubble), never coerced into a guess. - scoring: raw section scores → the official conversion table for that exact test form → scaled 400-1600. uncertainty via item bootstrap (2000 resamples through the conversion table). models whose CIs overlap the band leader are marked statistically tied, because a 20-point gap on one sitting of one form means nothing (the SAT’s own standard error is ~30-40 points).
- coverage gate: a model only gets a scaled score if ≥98% of its units returned real responses. infrastructure errors are retried until clean — the final board has zero error-contaminated scores.
- vision split: 34 of 120 questions (pt10) need the figure. text-only models (DeepSeek, Qwen, Kimi, GLM, MiniMax, Step) can’t take the full test, so they’re reported on the text-only subset instead of being assigned a fake total.
two full tests × 20 models × CoT, plus a direct-answer condition, a shuffled-options robustness run, a no-passage contamination probe, and a 426-question AGIEval anchor: ~14,000 graded responses, ~$50 in API costs.
results
| Band | Model | Total | 95% CI | Pctl (SAT takers) | RW | Math |
|---|---|---|---|---|---|---|
| 1 | gemini-3.1-pro | 1485 | 1425–1560 | 97 | 800 | 685 |
| 1 | claude-opus-4.8 | 1470 | 1405–1555 | 97 | 790 | 680 |
| 1 | gemini-3.5-flash | 1445 | 1365–1520 | 95 | 800 | 645 |
| 1 | gpt-5.5 | 1445 | 1375–1520 | 95 | 800 | 645 |
| 1 | grok-4.5 | 1445 | 1370–1515 | 95 | 800 | 645 |
| 1 | claude-sonnet-5 | 1440 | 1365–1520 | 95 | 785 | 655 |
| 1 | gpt-5.2 | 1435 | 1360–1510 | 95 | 795 | 640 |
| 1 | o4-mini | 1380 | 1300–1465 | 92 | 765 | 615 |
| 1 | gemini-2.5-flash | 1365 | 1290–1450 | 91 | 775 | 590 |
| 1 | claude-haiku-4.5 | 1345 | 1260–1445 | 89 | 735 | 610 |
| 2 | gpt-4.1 | 1310 | 1230–1400 | 87 | 750 | 560 |
| 2 | llama-4-maverick | 1305 | 1215–1390 | 86 | 740 | 565 |
| 2 | grok-4.3 | 1285 | 1210–1375 | 84 | 775 | 510 |
| 2 | mistral-large-3 | 1265 | 1170–1360 | 83 | 705 | 560 |
live interactive version with CI bars: vincentwi.com/sat-benchmark
text-only models (no figures → no full test, text-question accuracy instead): deepseek-v4-pro 93.1%, qwen3.5-397b 91.5%, kimi-k2.6 91.1%, glm-5.2 89.9%, step-3.7-flash 86.6%, minimax-m3 85.5%. for scale, gemini-3.1-pro and opus get 93.9% on the same subset — DeepSeek is statistically indistinguishable from the frontier on text questions. it just can’t see.
things to notice:
- reading & writing is done. four models put up an 800. the digital SAT’s RW section no longer discriminates at the frontier.
- math with figures is the moat. best math score: 685. the misses concentrate hard in figure questions — reading a scatterplot, a geometry diagram, a graph of a system of equations. text-only math is much stronger.
- grok-4.3 posted 21 unparseable responses (reasoning loops that never emit an answer). that’s why its score is low — not because it doesn’t know things, but because it fails to answer. on a real test, blank bubbles score zero, so that’s what it gets.
- the whole top-10 is one statistical band. the gaps you see (1485 vs 1440) are within noise on a single form. anyone ranking models on point differences this small is selling you something.
the contamination asterisk (with data)
these are public practice tests. every model has seen them. instead of pretending otherwise, i measured it two ways:
1. the no-passage probe. i gave 8 models RW questions with the passage deleted — just the question and four options. chance is 25%.
| model | accuracy without the passage |
|---|---|
| gemini-3.1-pro | 72.1% |
| grok-4.5 | 62.3% |
| deepseek-v4-pro | 52.5% |
| gpt-5.5 | 50.8% |
| claude-opus-4.8 | 50.8% |
| gpt-4.1 | 49.2% |
| qwen3.5-397b | 49.2% |
| claude-haiku-4.5 | 29.5% |
some of that is legitimate — distractor options are often implausible on their own. but 72% without the passage is not test-taking skill, it’s recognition. interestingly haiku, the small model, is nearly at chance: contamination capacity seems to scale with model size.
2. the AGIEval anchor. the old-SAT AGIEval set has been public in training corpora since 2023 — maximum contamination. the same 8 models average 97-99% on it, saturated flat. the digital-SAT tests still show a real spread (83-94% on text questions), which tells you they’re less burned-in — but “less” is not “not”.
so read the leaderboard as an upper bound on SAT ability. the ranking is probably still informative (everyone took the same contaminated test), the absolute numbers are inflated by an unknowable amount, and anyone who quotes “AI scores 1485 on the SAT” without this paragraph is quoting it wrong.
robustness checks that earned their keep
- shuffled answer options: re-lettering every MCQ moved scores by -30 to +30 — noise, no systematic position memorization at the letter level. (building this check caught a real bug: you can’t shuffle questions whose options live inside a figure. v1 me would have shipped that corruption.)
- direct vs CoT: direct-answer mode costs most models 20-60 points, almost all in math. reasoning tokens are load-bearing for arithmetic, mostly decorative for reading.
- two test forms: pt10 and pt11 scores agree within each model’s CI, so this isn’t single-form luck.
what i actually learned
- benchmark plumbing is the whole game. v1’s headline finding (“Chinese models tie for 2nd”) was an artifact of a regex. every viral eval you’ve seen has a nonzero chance of being this. demand raw logs.
- the SAT is saturating from the top. RW: solved. text math: nearly solved. what’s left — figure reasoning — isn’t really “SAT skill”, it’s vision, and it’s visibly the weakest link in every frontier stack.
- contamination is measurable. you don’t have to shrug. delete the passage and see what survives; anchor against a known-burned dataset. both probes cost me under $5.
- models don’t fail by being wrong, they fail by not answering. unparseable responses and reasoning-loop timeouts moved scores more than knowledge gaps did for several models. that’s an alignment-to-format problem, and it’s why “error handling” in eval harnesses is a scientific issue, not an engineering nicety.
all code, prompts, raw model responses, and the scoring pipeline are in the repo — benchmark/ if you want to rerun it. the copyrighted questions themselves aren’t committed (College Board owns those), but the manifest + hashes are, so you can rebuild the exact dataset from the public PDFs and verify every number.
previous version: i made agi take the SAT (jan 2026) — kept up for the record, numbers superseded by this post.