Sources: the vendor's own pages (linked above) and creator videos at the linked timestamps. EU status is information, not legal advice; see how we check.
DeepSWE (also referred to as Deep Suite) is a software engineering benchmark published by Datacurve. It evaluates frontier coding agents on long-horizon tasks that were written from scratch rather than adapted from existing commits or pull requests, which is intended to avoid training-data contamination.
According to the project site, tasks span 91 repositories across 5 languages. Prompts are about half the length of those in SWE-bench Pro, while solutions require about 5.5x more code and about 2x more output tokens. Verifiers are hand-written to test software behavior rather than implementation details. Example tasks include adding XML diff/patch/merge to etree, adding trap coredump generation to wasmi, and fixing PromQL label sorting.
All models on the leaderboard are run through mini-swe-agent for consistency, and new frontier models are added as they are released. The site links to a blog covering methodology, results, qualitative failure analysis, and limitations. A YouTube reviewer cited it as a favorite benchmark because it plots average cost per task against completion rate. It is a public leaderboard and not a product to run; the pages provided do not state whether the task set or harness code is released.
Watch the moment: the exact second top creators show it. Videos load only when you press play, via youtube-nocookie.com.
Benchmark the creator calls his favorite, plotting average cost per task against completion success rate. Kimi K3 Max lands just under GPT 5.6 Soul at about the same cost, around $4.70 per task.
Contamination-free coding benchmark with tasks written from scratch, requiring more code and output tokens than SWE-bench Pro. GPT-5.5 extra high currently leads, and Fable results aren't in yet.
Benchmark for coding agents with tasks written from scratch across 91 open-source repos, short prompts and large reference solutions, checked with handwritten behavioral verifiers. GPT 5.5 scores highest, then Claude models, then Gemini 3.5 Flash, while open models like Kimi K2.6 and GLM score lower.
A long-horizon software engineering benchmark with 113 hand-written, contamination-free tasks across 91 repos and five languages. It uses short behavior-focused prompts and verifiers with far lower false positive/negative rates than SWE-bench Pro (0.3%/1.1% vs 8.5%/24%). It shows a much wider score spread between models.
Closest neighbours by meaning, not just by category.
Research
Benchmark from Scale AI testing AI agents on long-horizon software engineering tasks drawn from real codebases.
Research
Benchmark of frontier AI models on private, real-world enterprise codebases, published by Specific.
Library
Minimal open-source AI software engineering agent in about 100 lines of Python that uses only bash.
Research
Benchmark of AI agents on long-horizon professional workflows across 55 sub-industries, led by Berkeley RDI.
Model · Cognition
Cognition's fast, efficient coding model, built for sub-agents handling delegated coding tasks.
Research
Benchmark of 270 tasks testing whether AI agents can computationally reproduce results from scientific papers.
