PaperBench

PaperBench is a benchmark from OpenAI, announced on 2 April 2025, that evaluates the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, which involves understanding each paper's contributions, building a codebase, and running the experiments.

Each replication task is broken down into smaller sub-tasks with clear grading criteria, using rubrics co-developed with the authors of the original papers. In total the benchmark contains 8,316 individually gradable tasks. An LLM-based judge grades replication attempts against the rubrics, and a separate benchmark was created to assess the judge's accuracy.

In OpenAI's reported evaluation, the best-performing tested agent was Claude 3.5 Sonnet (New) with open-source scaffolding, with an average replication score of 21.0%. Top ML PhDs attempted a subset of the tasks, and the models did not yet outperform this human baseline. The code has been open-sourced to support research on the AI engineering capabilities of agents. It is a research evaluation harness, not a consumer product.