Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

METR: Anthropic's 2x threshold may be crossed

Written November 7, 2026

METR reviewed Anthropic's August Risk Report and says its "less than 2x" claim rests on contestable calls. The disputes, and what each side agrees on.

What METR reviewed

METR reviewed the "Risks from automated R&D" section of Anthropic's August 2026 Risk Report, alongside a private version of the report and non-public data Anthropic shared. The work ran from August 12 to October 29, 2026, as part of a high-access assessment of AI R&D acceleration at Anthropic. Anthropic's Responsible Scaling Policy sets two capability thresholds for automated R&D in key domains: a full-substitution threshold, where a model can fully substitute for the entire set of research scientists and engineers at competitive cost, and a 2x acceleration threshold, where the pace of AI progress doubles for reasons that likely relate to the automation of AI R&D. The 2x threshold is an early warning, not direct evidence of high risk.

On the points that matter most, the two sides agree. METR agrees with Anthropic that catastrophic risks from models as of July 15 were low, and that no model could fully substitute for research staff. Both METR's researchers and Anthropic's relevant staff lean toward the 2x threshold not having been crossed. The dispute is about confidence, not about the risk level — and about whether the public report says more than its evidence supports.

Where they disagree

The public report says leading indicators point to "meaningful acceleration starting in early-to-mid 2025, though by less than a factor of 2". METR calculated 1.97 from figures in the private report and says "around 2" is the fairer description: a reader would take "less than a factor of 2" to mean an estimate meaningfully below 2, and 1.97 is not that. METR's own analysis of the leading indicators finds a range of reasonable estimates between 1.3 and 2.8, depending on the definition and time period.

The second dispute is about attribution. The public report says Anthropic is "fairly confident" the 2025 acceleration came from factors other than its use of AI models. The private report says it is "plausible" the overall counterfactual acceleration attributable to AI exceeds a factor of 2 by July 15. METR broadly agrees with the private report's reading: directly estimating counterfactual acceleration is very hard, and the public report implies more confidence than the evidence carries.

The third dispute is about the capability trend. The public report reads the AECI scores — Anthropic's version of the Epoch Capabilities Index — as weak evidence against a sustained acceleration, a one-time jump instead. METR disagrees with the emphasis: reusing the report's own data, the estimated acceleration is around 1.5x, and with data added after publication it is 1.9 times. A continued acceleration, METR says, is more consistent with the acceleration in the leading indicators.

Do not conflate the 1.9x

The 1.9x figure comes from METR's companion assessment: benchmark scores for Anthropic's models have grown at an estimated 1.9 times their pre-October 2025 rate since October 2025, with a 95% confidence interval of 1.5 to 2.8. That is a different comparison, with a different baseline and a different date, from the threshold question about July 15. It is evidence that acceleration is real and continuing — it is not the verdict on whether the 2x threshold was crossed. The companion assessment is covered in METR: Anthropic's AI R&D sped up about 1.9 times; this page does not repeat its numbers beyond the distinction.

What a builder takes from it

A threshold call is a judgment with stated uncertainty, not a measurement. When a lab says a threshold was not crossed, ask what the underlying figures imply, how wide the range of reasonable estimates is, and how much of the evidence you are allowed to see. Here the most informative evidence was largely out of view: Anthropic redacted a majority of the evidence METR found most informative for its conclusions, so the public has to rely on METR's judgment more than would be ideal. The honest summary is the one both sides share — catastrophic risk from the July 15 models was low, and crossing the threshold was plausible but probably did not happen — with the public report's confidence on the second point overstated.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: METR: Anthropic's AI R&D sped up about 1.9 times · The intelligence-explosion argument, and its evidence · Astra did 40% less work on safety tasks