Does CORE-Bench work in the EU?

Available in the EU. MIT-licensed open-source code and a public Hugging Face dataset with no regional restrictions; it can be run locally in Docker anywhere. Last verified 14 Nov 2026.

How much does CORE-Bench cost?

Open source, open source. Free to use under the MIT license; running agents incurs your own model API and compute (e.g. Azure) costs. Licence: MIT.

Where was CORE-Bench shown?

In 1 creator video by Matthew Berman. Most recently by Matthew Berman on 5 Jun 2026, at 9:50 in "It’s starting…".

CORE-Bench

Try it Docs GitHub Hugging Face Paper

Benchmark of 270 tasks testing whether AI agents can computationally reproduce results from scientific papers.

Available in EUResearch Language ModelsResearch

Updated 14 Nov 2026EU status checked 14 Nov 2026Last shown 5 Jun 2026

CORE-Bench at a glance

What is CORE-Bench?: CORE-Bench is a research. Benchmark of 270 tasks testing whether AI agents can computationally reproduce results from scientific papers.
Does CORE-Bench work in the EU?: Available in the EU. MIT-licensed open-source code and a public Hugging Face dataset with no regional restrictions; it can be run locally in Docker anywhere. Last verified 14 Nov 2026.
How much does CORE-Bench cost?: Open source, open source. Free to use under the MIT license; running agents incurs your own model API and compute (e.g. Azure) costs. Licence: MIT.
Where was CORE-Bench shown?: In 1 creator video by Matthew Berman. Most recently by Matthew Berman on 5 Jun 2026, at 9:50 in “It’s starting…”.

Sources: the vendor's own pages (linked above) and creator videos at the linked timestamps. EU status is information, not legal advice; see how we check.

About

CORE-Bench is a benchmark for evaluating the ability of AI agents to computationally reproduce scientific papers. It comprises 270 tasks drawn from 90 papers across computer science, social science and medicine, with code written in Python or R. Each task gives the agent a prompt and questions about a code repository.

To complete a task, the agent must navigate the repository, install dependencies, run the code to generate results, and read the outputs to answer the questions. Tasks come in difficulty levels (the README mentions CORE-Bench-Hard and CORE-Bench-Medium). The accompanying paper is on arXiv (2409.11363) and the dataset is hosted on Hugging Face.

The original GitHub harness runs agents in an isolated environment, either locally in a Docker container or on Azure VMs, and includes the AutoGPT-based and CORE-Agent baselines. The harness is no longer actively maintained. The recommended way to run the benchmark is now the Holistic Agent Leaderboard (HAL) harness, which also hosts the leaderboard. Commentary from a YouTube creator says scores rose from about 20% in 2024 to near saturation.

Key features