Playbooks · Coding tasks
Scientist World: 1,000 executable research tasks
Written November 9, 2026
Scientist World (Nov 5) offers 1,000 executable research environments plus agent science traces. What the environments test, and what the traces show.
What it is
Scientist World, a November 5 paper from Princeton, turns research papers into places where an AI can do research. The problem it starts from is simple: the scientific literature is readable but not runnable. A paper states a problem, proposes a method, and reports what happened — but it is a record of science done, not a place to do science. Scientist World rebuilds each paper as an executable environment: the paper's open-ended research question becomes the task, and the paper's own reproduced method becomes the baseline an AI scientist is scored against. The first release holds 1,000 such environments, and the world keeps growing with the literature.
Each paper is converted by a multi-agent pipeline with no human in the loop, in four roles. A Reproducer reestablishes the paper's main result, implementing the method and the original evaluation from the authors' code where it exists, or from scratch where it does not. A Designer turns the reproduction into a task: it writes the problem statement, fixes the rules — which metric to improve, which settings stay fixed, which kinds of solution are allowed — and prepares two evaluation sets, one visible and one held out to catch shortcuts. A Packager makes the environment run anywhere, replacing local files with download scripts and declaring every dependency, so a fresh clone can rebuild and rerun on its own. A Verifier audits the packaged environment before it is accepted, checking the code is valid, the scores match the logs, and the task follows the design rules — and the builder that grades the work never built it. A single paper can take hundreds of unattended turns to convert, and the milestone gate sends unfinished work back for another pass.
What it measures
An AI scientist entering an environment is scored by Frontier-Normalized Improvement: how much its own method improves over the human researchers' reproduced method. The score is zero when it merely matches the paper, positive when it beats it. Each environment is attempted in two modes. In improve mode, the scientist starts from the paper's idea with its reproduced method. In innovate mode, it starts from the problem alone. The two abilities prove distinct: the best improver is not the best inventor, so a single score would hide the difference.
The headline finding is that frontier AI scientists already surpass the human researchers 74% of the time, measured against the reproduced baselines. Read that number for what it is: the environments test reproduction and extension of published work — improving on a known result, or solving a known problem from scratch — not novel discovery of something no one has asked. The paper also reports that the environments work as a training ground: rewarded only by real experimental outcomes, a small model was raised 158% to match frontier models in research ideation.
What the traces are
Alongside the environments, the paper releases 6,400 hours of research traces: every idea, experiment, failure, and success from the evaluation runs, each decision paired with the measured outcome of a real experiment. The traces are the third thing Scientist World is, after the benchmark and the training ground. As a benchmark, it is the standing measure of whether AI scientists can advance real research. As a dataset, it is a growing corpus of research being done, with every attempt preserved in full. As a gym, the same environments become training grounds, with real experimental results as rewards. The point of keeping all three together is that research is judged by execution, not rhetoric: a fluent scientist can make any idea sound bold on paper, but in these environments an idea has to survive a real experiment.
How to use it
- Compare science agents on execution, not on claims. If you are choosing an agent to do research work, run it in an environment where the score comes from a real experiment against a reproduced baseline, rather than from a judge reading its proposal. The Frontier-Normalized Improvement score is the comparison; the traces show how it got there.
- Keep improve and innovate apart. An agent that is good at extending a known method may be poor at starting from a problem alone, and the paper shows the two do not travel together. Test the mode your work actually needs.
- Use the traces to train, not just to score. The 6,400 hours of traces are the training material: a small model trained on them, rewarded only by measured outcomes, reached frontier level on held-out environments for research ideation. The traces are worth more than the leaderboard.
- Do not read the 74% as novel discovery. Surpassing a reproduced baseline on a published problem is a real capability, and it is not the same as asking a question no one has asked and answering it. The environments measure research against known ground; treat claims beyond that as untested.
Verdict
Judge a science agent here on what it executes and improves, not on what it claims — the environments score real experiments against reproduced baselines, and the 74% figure is about beating those baselines. For choosing between science agents, that is a far better test than a proposal judged on prose; for claiming a model does novel science, it is not evidence at all.
What can go wrong
- Reading 74% as "AI beats scientists." The figure counts environments where the agent's method improved on the paper's reproduced result. It is a measure of reproduction and extension, not of original discovery.
- Collapsing improve and innovate into one score. The paper shows the best improver is not the best inventor. A single leaderboard number hides which ability you are actually buying.
- Assuming the environments cover your field. The first 1,000 environments are built from high-quality machine-learning papers. A scientist for another discipline has no equivalent ground here yet.
- Trusting a score without the trace. Every attempt leaves a trajectory; a score that cannot be traced back to a real experiment is not a Scientist World score. The held-out evaluation set exists to catch shortcuts — use it.
Limits
- This page reads the paper on arXiv. It is a preprint, not peer-reviewed, and every figure here — the 1,000 environments, the 74%, the 6,400 hours, the 158% — is the authors' own measurement.
- No independent replication exists yet. The environments were built and reviewed by the authors' own pipeline, with the verifier role kept separate from the builder roles by design.
- No environment was run here. This page maps the paper; it does not score any agent.
Related playbooks
- Astra did 40% less work on safety tasks
- Multi-agent ticker research with a TradingAgents port
- MCPHallu: agents hallucinate across 358 tasks
- Export Jules work and rebuild scheduled tasks before the shutdown
Sources
Back to: Coding playbooks hub · Also: PACT-Check: benchmark gains without information flow · Letterpress-1: a growth agent that files PRs