OpenAI announced it no longer recommends SWE-Bench Pro as a dependable standard for measuring AI coding performance after an internal review revealed extensive task-level defects. The company said the audit showed roughly 30% of the benchmark’s publicly available tasks contain design or grading problems that can produce misleading results.
The review targeted the 731-task public split of SWE-Bench Pro. OpenAI applied model-based investigator agents and enlisted five experienced software engineers for independent assessment. Using its datapoint analysis pipeline, the company flagged 200 tasks - equal to 27.4% of the public dataset - as broken. Separate human reviewers identified issues in 249 tasks, or 34.1% of the set.
OpenAI also noted that the benchmark appears to have approached a noise ceiling in its ability to discriminate among frontier models. Over an eight-month period, the company observed a rise in pass rates from 23.3% to 80.3% for leading models, and characterized the dataset as reaching a saturation point near a 70% noise ceiling. Such saturation can obscure meaningful differences in model capability.
The audit categorized the task failures into four principal problem types:
- Overly strict tests - unit tests that enforce specific implementation details not specified in the task prompt, rejecting solutions that are functionally correct but do not match an expected structure.
- Underspecified prompts - tasks where prompts omit requirements that are nonetheless enforced by hidden tests, causing correct-seeming solutions to fail.
- Low-coverage tests - tests that do not fully verify requested behavior, allowing partial or incomplete fixes to pass as correct.
- Misleading prompts - prompts that steer models toward incorrect approaches or that contradict the expectations encoded in the tests.
OpenAI explained that correct solutions sometimes fail because hidden requirements exist, instructions are contradictory, grading criteria are incomplete, or tests are overly prescriptive. The company traced part of the root cause to the dataset construction process: SWE-Bench Pro tasks are programmatically sourced from feature changes in public and private code repositories, where problem descriptions, merged code, and unit tests do not always align to form well-scoped evaluation items.
Previously, OpenAI had investigated SWE-bench Verified and, citing fundamental design and contamination issues, had recommended SWE-Bench Pro as an alternative. The new audit indicates that the alternative itself contains significant flaws. OpenAI warned that such evaluation errors can skew the understanding of model capabilities and may misrepresent safety cases and research priorities under its Preparedness Framework.
The company said it is retracting its earlier recommendation that the research community use SWE-Bench Pro as a leading coding benchmark. The findings underscore limitations in programmatically generated evaluation datasets and highlight the need for careful task design and grading fidelity when assessing AI code-writing systems.