OpenAI retracts SWE-Bench Pro recommendation after audit finds 30% of tasks broken

OpenAI announced on Tuesday that it is retracting its recommendation of SWE-Bench Pro as a reliable measure of AI coding capability, citing an internal audit that found roughly 30% of the benchmark's tasks are broken and a noise ceiling of approximately 70% that undermines its usefulness for evaluating frontier models.x+1

A Pattern of Benchmark Erosion

The retraction marks the second time in less than six months that OpenAI has walked away from a major coding evaluation. In February 2026, the company stopped reporting scores on SWE-bench Verified — a benchmark it helped create in 2024 — after finding that 59.4% of audited tasks contained flawed test cases and that training data contamination had rendered scores meaningless.openai+1

At the time, OpenAI pointed researchers toward SWE-Bench Pro, a harder benchmark developed by Scale AI that comprises 1,865 long-horizon software engineering tasks across 41 professional repositories. The benchmark was designed to resist the contamination problems that plagued its predecessor, and top frontier models initially scored only around 23% on it.morphllm+3

Rapid Saturation

But scores climbed quickly. By late June 2026, Claude Opus 4.8 reached 69.2% on vendor-aggregate harnesses and GPT-5.4 (xHigh) scored 59.1% on Scale's standardized public set. OpenAI's audit now concludes that scores in this range no longer reflect genuine capability differences between models — they instead reflect the benchmark's own limitations, including broken task specifications and noisy test infrastructure.morphllm+3

The company called SWE-Bench Pro "one of the most widely used AI coding benchmarks" and urged the research community to move toward new evaluation methods.digg+1

Broader Implications

The announcement intensifies a growing crisis of confidence in AI coding evaluations. Researchers have noted that the gap between benchmark performance and real-world software engineering remains wide, with models that score well on structured evaluations often struggling on tasks requiring genuine architectural reasoning or full-program construction from scratch. OpenAI has not yet named a replacement benchmark, leaving the field without a consensus standard for measuring frontier coding progress.tianpan+1