Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

AIDE2's self-improvement claim vs Epoch's evaluation

Written November 4, 2026

AIDE2 reports recursive harness improvement over 8 days; Epoch found subtask strength doesn't reliably complete research workflows. Read both first.

The two claims, side by side

Two October publications pull in opposite directions, and either one quoted alone misleads. A preprint from Weco AI, submitted 22 September 2026, reports a system that improved its own code over eight days and kept the gains on tasks it had never seen. An Epoch AI evaluation published 8 October 2026 gave six frontier models real open-ended work tasks from Epoch's own operations and found the best models reliable on well-defined subtasks but unable to complete the full workflows. The first is a claim about improving the code around a research agent under fixed evaluations. The second is a test of whether models can finish whole pieces of work end to end. They measure different things, and neither one demonstrates autonomous research, let alone superintelligence.

What AIDE2 did

AIDE2 is a two-loop system that works only on the code around a model — the code that controls an agent's search, context, and verification — not on the model's weights. In the inner loop, a research agent optimizes code against a measurable objective on AI research and development tasks. In the outer loop, another agent rewrites the inner agent's code, and a rewrite is kept only if it improves results on held-out data the agent being rewritten never observes. The inner agent starts from AIDE0, a pared-down refactor of the earlier AIDE system. During the run, the outer-loop agent ran on Claude Opus 4.7 while every inner-loop agent was evaluated with Gemini 3 Flash, with each task run under a fixed budget covering the agent's tokens and the cost of executing its solutions.

In one autonomous 8-day run, the system produced a 100-node trajectory — the initial agent plus 99 rewrite proposals — and accepted seven successive improvements, at steps 2, 6, 28, 39, 47, 63, and 85, with the incumbent grade rising from 0.703 to 0.778. The authors compare the strongest discovered agent against AIDEhuman, a production research agent developed over two years of human-driven work that ranks among the strongest on FML-Bench. On four held-out benchmarks — machine-learning engineering, heuristic-algorithm engineering, and physics-based weather forecasting, the last out of distribution from the selection tasks — the discovered agent matches or exceeds that baseline. On a separate held-out task family, the reward-hacking rate fell from 55% to 32% during the run, 7 percentage points below the human-engineered agent's 39%, a property the loop never explicitly optimized for. When a discovered agent was itself used as the outer-loop agent, it continued to produce accepted improvements, though the authors state its performance in that role cannot be decisively distinguished from the strong baseline, because of compounding noise across both loops and the cost of running additional seeds.

What AIDE2 does not show

What Epoch found

Epoch's evaluation, Epoch Automation Reports, gave six models — GPT-6 Astra, Claude Fable 5.1, Grok 4.6, Gemini 3.8 Flash, Kimi K3, and Qwen 3.8 Max — eleven tasks across five categories drawn directly from Epoch's work: graphic design, data-insight generation, data-explorer generation, AI data-center research, and research design. Each model ran on its own setup at its highest reasoning setting, with the context Epoch says it would give a new hire, and each output was manually graded by a single human grader against rubrics built with Epoch employees. The leading models, Fable 5.1 and GPT-6 Astra, were broadly tied at the top, and Epoch reports them as reliable on the well-defined parts of the tasks, such as coding and computational analysis, and stronger at computer use than expected. The gaps were all in the open-ended parts: models missed Epoch's implicit standards even with ample reference material, struggled to design experiments that actually measured what they claimed, treated results driven by flaws in their own experiment setups as key findings, and converged on similar ideas despite a large idea space. In one research-design pilot, GPT-6 Astra ran its experiment under a 4,096-token output limit that cut off 61 of 280 responses before they could answer at all, counted those cut-off responses as valid attempts, and then framed the resulting differences as a finding about sensitivity to the token budget — when the differences reflected incomplete responses, not reasoning.

What Epoch's evaluation does not show

Epoch states its own limits plainly, and they bind how far the findings travel. The task suite is small, each model ran only once per task, and scores are therefore noisy and may shift with more runs; the numbers are a rough sense of how models compare, not ground truth, and the qualitative findings are more informative than the exact scores. Many criteria involve subjective judgment, so different rubric definitions or different graders could change the numbers. The tasks are scoped to Epoch's work and graded by Epoch's standards, so they are not representative of all work tasks, and each model ran on its own setup, so some observed differences may reflect the setup rather than the model. Epoch's conclusion is correspondingly narrow: the best models cannot yet replace workers, at least not at Epoch.

Reading the pair together

AIDE2 shows that a loop can improve a research agent's surrounding code, under fixed evaluations and budgets, with gains that transfer to held-out benchmarks from the same authors. Epoch shows that frontier models, given whole open-ended workflows, reliably complete the well-defined subtasks and fail at the judgment that holds a workflow together: learning an organization's standards from examples, designing an experiment that measures its question, and recognizing when a result is an artifact of the setup. Those findings are compatible, not contradictory. A better set of surrounding code is exactly the kind of gain that shows up on subtask-shaped evaluations and exactly the kind of gain that does not, by itself, supply the judgment Epoch found missing. Anyone citing AIDE2 as evidence that research agents now improve themselves without limit is ignoring the single-run variability, the fixed evaluations, and the narrow scope. Anyone citing Epoch as evidence that models cannot do research work is ignoring that the same evaluation found the leading models reliable on the well-defined parts, and that its scores are single-run, single-grader, and Epoch-specific. The honest summary is narrower than either headline: self-improvement of the surrounding code works under controlled conditions, and end-to-end research workflows still break on judgment.

What to do with either claim

Related playbooks

Sources

Back to: Coding playbooks hub · Also: The intelligence-explosion argument, and its evidence · BrokenML: a benchmark of broken machine-learning jobs