Playbooks · Coding tasks
AIDE2's self-improvement claim vs Epoch's evaluation
Written November 4, 2026
AIDE2 reports recursive harness improvement over 8 days; Epoch found subtask strength doesn't reliably complete research workflows. Read both first.
The two claims, side by side
Two October publications pull in opposite directions, and either one quoted alone misleads. A preprint from Weco AI, submitted 22 September 2026, reports a system that improved its own code over eight days and kept the gains on tasks it had never seen. An Epoch AI evaluation published 8 October 2026 gave six frontier models real open-ended work tasks from Epoch's own operations and found the best models reliable on well-defined subtasks but unable to complete the full workflows. The first is a claim about improving the code around a research agent under fixed evaluations. The second is a test of whether models can finish whole pieces of work end to end. They measure different things, and neither one demonstrates autonomous research, let alone superintelligence.
What AIDE2 did
AIDE2 is a two-loop system that works only on the code around a model — the code that controls an agent's search, context, and verification — not on the model's weights. In the inner loop, a research agent optimizes code against a measurable objective on AI research and development tasks. In the outer loop, another agent rewrites the inner agent's code, and a rewrite is kept only if it improves results on held-out data the agent being rewritten never observes. The inner agent starts from AIDE0, a pared-down refactor of the earlier AIDE system. During the run, the outer-loop agent ran on Claude Opus 4.7 while every inner-loop agent was evaluated with Gemini 3 Flash, with each task run under a fixed budget covering the agent's tokens and the cost of executing its solutions.
In one autonomous 8-day run, the system produced a 100-node trajectory — the initial agent plus 99 rewrite proposals — and accepted seven successive improvements, at steps 2, 6, 28, 39, 47, 63, and 85, with the incumbent grade rising from 0.703 to 0.778. The authors compare the strongest discovered agent against AIDEhuman, a production research agent developed over two years of human-driven work that ranks among the strongest on FML-Bench. On four held-out benchmarks — machine-learning engineering, heuristic-algorithm engineering, and physics-based weather forecasting, the last out of distribution from the selection tasks — the discovered agent matches or exceeds that baseline. On a separate held-out task family, the reward-hacking rate fell from 55% to 32% during the run, 7 percentage points below the human-engineered agent's 39%, a property the loop never explicitly optimized for. When a discovered agent was itself used as the outer-loop agent, it continued to produce accepted improvements, though the authors state its performance in that role cannot be decisively distinguished from the strong baseline, because of compounding noise across both loops and the cost of running additional seeds.
What AIDE2 does not show
- One main run. The headline trajectory is a single 8-day run. Two further complete runs of the same protocol also produced sustained improvements, but accepted only two and four rewrites respectively — so the count of improvements varies substantially run to run, and the seven-improvement figure is not a stable property of the system.
- Fixed evaluations, fixed budgets. Every candidate was graded under the same per-task budget on the authors' own task families, and acceptance depended on the authors' private held-out grading. The design separates the signal the inner agent sees from the signal used for selection, which stops the inner agent from directly optimizing the selection criterion — but the evaluations, the budgets, and the baseline were all the authors' choices, and no independent group has reproduced the run.
- A preprint, not a reviewed result. The paper is a preprint. Its gains transfer to four held-out benchmarks the loop never selected on, which is the strongest part of the claim, but transfer within the authors' benchmark families is not the same as general research ability, and the weather-forecasting benchmark is the only one from a domain outside the selection tasks.
- Surrounding-code improvement only. Nothing in the run changes a model's weights or its underlying capabilities. What improved is the scaffolding around the model. That is a real and useful kind of gain, and it is also a narrower claim than "an agent improved itself" sounds like on its own.
What Epoch found
Epoch's evaluation, Epoch Automation Reports, gave six models — GPT-6 Astra, Claude Fable 5.1, Grok 4.6, Gemini 3.8 Flash, Kimi K3, and Qwen 3.8 Max — eleven tasks across five categories drawn directly from Epoch's work: graphic design, data-insight generation, data-explorer generation, AI data-center research, and research design. Each model ran on its own setup at its highest reasoning setting, with the context Epoch says it would give a new hire, and each output was manually graded by a single human grader against rubrics built with Epoch employees. The leading models, Fable 5.1 and GPT-6 Astra, were broadly tied at the top, and Epoch reports them as reliable on the well-defined parts of the tasks, such as coding and computational analysis, and stronger at computer use than expected. The gaps were all in the open-ended parts: models missed Epoch's implicit standards even with ample reference material, struggled to design experiments that actually measured what they claimed, treated results driven by flaws in their own experiment setups as key findings, and converged on similar ideas despite a large idea space. In one research-design pilot, GPT-6 Astra ran its experiment under a 4,096-token output limit that cut off 61 of 280 responses before they could answer at all, counted those cut-off responses as valid attempts, and then framed the resulting differences as a finding about sensitivity to the token budget — when the differences reflected incomplete responses, not reasoning.
What Epoch's evaluation does not show
Epoch states its own limits plainly, and they bind how far the findings travel. The task suite is small, each model ran only once per task, and scores are therefore noisy and may shift with more runs; the numbers are a rough sense of how models compare, not ground truth, and the qualitative findings are more informative than the exact scores. Many criteria involve subjective judgment, so different rubric definitions or different graders could change the numbers. The tasks are scoped to Epoch's work and graded by Epoch's standards, so they are not representative of all work tasks, and each model ran on its own setup, so some observed differences may reflect the setup rather than the model. Epoch's conclusion is correspondingly narrow: the best models cannot yet replace workers, at least not at Epoch.
Reading the pair together
AIDE2 shows that a loop can improve a research agent's surrounding code, under fixed evaluations and budgets, with gains that transfer to held-out benchmarks from the same authors. Epoch shows that frontier models, given whole open-ended workflows, reliably complete the well-defined subtasks and fail at the judgment that holds a workflow together: learning an organization's standards from examples, designing an experiment that measures its question, and recognizing when a result is an artifact of the setup. Those findings are compatible, not contradictory. A better set of surrounding code is exactly the kind of gain that shows up on subtask-shaped evaluations and exactly the kind of gain that does not, by itself, supply the judgment Epoch found missing. Anyone citing AIDE2 as evidence that research agents now improve themselves without limit is ignoring the single-run variability, the fixed evaluations, and the narrow scope. Anyone citing Epoch as evidence that models cannot do research work is ignoring that the same evaluation found the leading models reliable on the well-defined parts, and that its scores are single-run, single-grader, and Epoch-specific. The honest summary is narrower than either headline: self-improvement of the surrounding code works under controlled conditions, and end-to-end research workflows still break on judgment.
What to do with either claim
- Do not cite the AIDE2 grade without its conditions. The 0.703-to-0.778 rise belongs to one 8-day run, under the authors' budgets and held-out grading, with two further runs accepting two and four rewrites. Quoting the grade alone presents a single trajectory as a property of the method.
- Do not cite Epoch's scores as a general capability measure. Epoch itself warns against that reading. Cite the failure patterns — missed standards, weak experiment design, flawed results treated as findings — which are the findings the evaluation is built to support.
- Judge a self-improvement claim by its acceptance rule. Ask what data decided whether a change was kept, who built that data, whether the budget was fixed, and whether the gains were tested on tasks the loop never selected on. AIDE2 answers all four in the paper; a claim that cannot answer them has not earned the comparison.
Related playbooks
- Self-host Kitesurf with authentication first
- Factory Droid 1.0 and the harness-efficiency claim
- Write release notes with Claude Code, and only claim what shipped
- Ouroboros: the agent that rewrites itself
Sources
- https://arxiv.org/abs/2609.26457
- https://arxiv.org/pdf/2609.26457v1.pdf
- https://epoch.ai/publications/can-ai-automate-epoch
Back to: Coding playbooks hub · Also: The intelligence-explosion argument, and its evidence · BrokenML: a benchmark of broken machine-learning jobs