Sources: the vendor's own pages (linked above) and creator videos at the linked timestamps. EU status is information, not legal advice; see how we check.
SWE-bench Pro is a benchmark from Scale AI that evaluates language models and coding agents on long-horizon software engineering tasks. Given a codebase and an issue description, the model must generate a patch that resolves the problem. The dataset is inspired by SWE-bench and is verified by running tests in containerized environments.
The original v1 set has 731 tasks. V2, announced in the repository news, is the validated public split of 642 tasks across 11 repositories. It is packaged in Harbor format with verifiers, reference solutions and public container images. It also adds the HARD-51 subset and tooling for a locked evaluation protocol (offline agent phase, fresh-sandbox re-grading). The repository includes SWE-agent and mini-swe-agent scaffolds for reproducing results, and evaluation can run on Modal or local Docker. Scale maintains a public leaderboard and a separate private commercial leaderboard.
Creators citing an external audit (reported as DeepSWE / Data Curve) say the verifier showed about 8–8.5% false positives and 24% false negatives, and that some models retrieved answers from git history. The repository notes that Scale found leaderboard issues in May and was working on them. These audit figures come from creator commentary, not from the provided official pages.
Watch the moment: the exact second top creators show it. Videos load only when you press play, via youtube-nocookie.com.
Agentic coding benchmark where Fable scores just above 80%. A Data Curve audit found verifier error rates (8% false positives, 24% false negatives) and Opus 4.7 pulling answers from git history.
The leading agentic coding benchmark, used as the comparison point. Its tasks average about 120 lines of code, and its verifier has an 8.5% false positive and 24% false negative rate per the DeepSWE audit.
Closest neighbours by meaning, not just by category.
Research
Benchmark of frontier AI models on private, real-world enterprise codebases, published by Specific.
Research · Datacurve
Benchmark of frontier coding agents on original, long-horizon software engineering tasks, by Datacurve.
Library
Minimal open-source AI software engineering agent in about 100 lines of Python that uses only bash.
Research
Benchmark of AI agents on long-horizon professional workflows across 55 sub-industries, led by Berkeley RDI.
Research
Benchmark of 270 tasks testing whether AI agents can computationally reproduce results from scientific papers.
Research · OpenAI
OpenAI benchmark testing whether AI agents can replicate ICML 2024 papers from scratch, graded by hierarchical rubrics.
