AI Insights · Oct 22, 2026 · Partially verified
Coding benchmarks no longer pick the best model
SWE-bench Verified is retired as a frontier test, and SWE-bench Pro has known flaws. Newer tests change often, and the useful measure is whether an agent succeeds every time, not once.
Claim
Public coding benchmarks are no longer a reliable signal for picking a model or a harness. Grader error and contamination are documented on both SWE-bench variants, Terminal-Bench scores are not comparable across its quarterly versions, and rankings depend on harness and effort setting. The useful practice is pass^k on your own tasks.
Evidence
- OpenAI stopped reporting SWE-bench Verified (2026-02-23): 59.4% of 138 audited problems had substantial issues; models had seen problems in training. Epoch rates it "Flawed" (2026-09-03) and SWE-bench Pro "Flawed" too (2026-09-01): about 30% of tasks broken per OpenAI's July audit, 24% false-negative and 8.5% false-positive grader rates per Datacurve, and a leaderboard mixing 50-turn and 250-turn runs. Scale's public Pro board tops out around 61.5%. Aggregator claims of 89.9% or "99% on V2" are inconsistent with Scale's page. UNVERIFIED.
- Terminal-Bench 2.0 (2025-11-07), 2.1, 3.0, 4.0 (2026-08-28); top of 4.0 about 60% to 65% depending on harness and effort (Sonnet 5.5 max 63.6%, Opus 5.5 59.6% on mini-swe-agent per Artificial Analysis).
- GDPval (OpenAI, 1,320 tasks across 44 occupations) is now run agentically by Artificial Analysis and reported as Elo; no OpenAI 2026 leaderboard found.
- ARC-AGI-3 (2026-03-24): 135 interactive environments; humans 100%, frontier systems under 1% at launch. HLE's official page is stale.
- tau-bench: Anthropic recommends pass^k, "the probability that all k trials succeed", for agents that need consistency; tau3 scores are secondary. UNVERIFIED.
Open questions
- Current ARC-AGI-3 and HLE leaderboard values (dynamic pages).
Sources
- https://epoch.ai/benchmarks/swe-bench-verified/review — blog, Wed Sep 02
- https://epoch.ai/benchmarks/swe-bench-pro/review — blog, Mon Aug 31
- https://labs.scale.com/leaderboard/swe_bench_pro_public — docs, Wed Oct 21
- https://www.tbench.ai/benchmarks — docs, Wed Oct 21
- https://artificialanalysis.ai/evaluations/terminalbench-4-0 — docs, Wed Oct 21
- https://artificialanalysis.ai/evaluations/gdpval-aa — docs, Wed Oct 21
- https://arxiv.org/html/2603.24621v1 — paper, Mon Mar 23
- https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents — blog, Thu Jan 08
AI-assisted research: gathered and summarised with AI from the sources above, each claim checked against them.