DeepSWE just changed the benchmark game...

DEDeepSWEDatacurveAvailable in EU

A long-horizon software engineering benchmark with 113 hand-written, contamination-free tasks across 91 repos and five languages. It uses short behavior-focused prompts and verifiers with far lower false positive/negative rates than SWE-bench Pro (0.3%/1.1% vs 8.5%/24%). It shows a much wider score spread between models.

SPSWE-bench ProAvailable in EU

The leading agentic coding benchmark, used as the comparison point. Its tasks average about 120 lines of code, and its verifier has an 8.5% false positive and 24% false negative rate per the DeepSWE audit.

GPGPT-5.5OpenAIAvailable in EU

Tops the DeepSWE leaderboard at about 70%, 15+ points above Opus 4.7. It is also cheaper per trial ($5.80), uses fewer tokens (about 47k) and runs faster than Opus 4.7. It follows prompts literally and has the lowest rate of missing stated behaviors.

COClaude Opus 4.7AnthropicAvailable in EU

Scores well below GPT-5.5 on DeepSWE, with about 3x the cost per trial ($16), 37 minutes per trial and higher token use. It tends to forget parallel behaviors in multi-part prompts but explores git history well. The creator still likes Opus.

GPGPT-5.4OpenAIAvailable in EU

Ranks second on the DeepSWE leaderboard, behind GPT-5.5 and ahead of Opus 4.7.

CSClaude Sonnet 4.6AnthropicAvailable in EU

Ranks far down the DeepSWE leaderboard, well below Opus 4.7.

G3Gemini 3.5 FlashGoogleAvailable in EU

Scores about 28% on DeepSWE and uses a very large number of tokens (about 150k per trial). It takes 15 minutes per trial, slower than expected for a Flash model.

Sponsor segment: HeyGen's HyperFrames lets an agent write code to build stylized motion-graphics and explainer videos that can be edited by prompt.

MImini-SWE-agentAvailable in EU

The SWE-bench authors' harness, held fixed across all models in DeepSWE. The creator notes that models like Opus might do better in their own harness such as Claude Code.

CHClaude Haiku 4.5AnthropicAvailable in EU

Scores 0% on DeepSWE, illustrating the benchmark's wide score spread.

C2Composer 2.5CursorAvailable in EU

Mentioned as missing from the DeepSWE charts. The creator says it is probably the best model on the planet for price-to-performance and points to his separate video on it.