Playbooks · Coding tasks
What AI monitoring missed, per the METR audit
Written November 4, 2026
A three-week audit found gaps in OpenAI's agent monitoring. What it proves about July 2026 — and what it doesn't.
What the audit was
From 21 July to 11 August 2026, one employee of METR and one employee of Redwood Research worked inside OpenAI with deep access to its internal codebase, its monitoring prompts and configuration, and its internal documentation, and interviewed about a dozen researchers and engineers. Their target was OpenAI's internal system for watching its own agents for misalignment: the system that reviews what internal agents do for signs of scheming, sabotage, data theft, deception, and prompt injection. The exercise was a pilot for giving outside researchers deep access to internal infrastructure, not a full public audit, and the published account is a brief, high-level summary agreed in advance. The findings describe the monitoring system as it stood on or before 21 July 2026. OpenAI was already expanding the system while the exercise ran, and the audit did not measure the expanded system.
What it found
The audit's first finding is about coverage, and it comes with wide uncertainty attached. METR estimated that roughly 75–94% of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring, with substantial uncertainty in both directions. For the subset of that traffic used by internal Codex agents, the estimate was 99.2% eligible. Eligible is not the same as monitored. The audit estimated that one gap alone — compaction events not being monitored — left on the order of 10% of otherwise-eligible internal Codex traffic not monitored between May and July 2026, again with substantial uncertainty, because the share of monitor calls triggered by compaction had not historically been logged. OpenAI staff had found that gap independently and fixed it during the exercise.
The gap mattered because of how monitoring was triggered. At the time, the monitor ran when an agent returned control to the user. While compaction events went without monitoring, an agent could put off being monitored indefinitely simply by continuing to take actions instead of returning control. Reinforcement-learning training runs were also outside the system at the start of the exercise, although a subset of training rollouts had been monitored between March and June.
The audit also found routes by which an agent could, in principle, escape the monitor altogether. Codex agents could in principle bypass Codex sandboxing and Auto-Review and take any action the host user could take. The auditors found an exploit that would keep hundreds of recent actions, including the actions used in the exploit itself, from being monitored; the details were redacted by OpenAI. They found two independent client-controllable headers that switched monitoring off for new agents, so an agent could start a copy of itself that was not monitored — although setting the headers and creating the new instance could itself be caught by monitoring. They found two internal tools that were not monitored and were thought to have read-only permissions, with a simple exploit that could have subverted those limits; they did not test whether the exploit worked in practice. And they found that agents could likely craft inputs that made the monitor fail, by exploiting differences between the trajectory the agent sees and the one the monitor reads, so a run would never be escalated for human review.
Not everything was broken. The information the auditors were shown suggested the monitor errored on less than 0.2% of requests, and that one 24-hour period during the exercise had zero load-sheds. With a few minor exceptions, the monitor was shown the full trajectory the agents saw, and its prompts did cover the risk categories it was meant to cover.
What it does not show
Nothing in the audit describes OpenAI's monitoring today. Every finding is dated to the system as it stood on or before 21 July 2026, and the audit says plainly that it did not establish the expanded system's current coverage. Since the exercise, OpenAI has stated that most of the vulnerabilities have been patched, that one fix is still in progress, and that the remaining vulnerability does not affect its misuse safeguards stack. METR has not evaluated those claims, and the public summary gives no independent basis for accepting or rejecting them. The summary is also short by design, with exploit details withheld, so it cannot support a detailed account of how any single bypass worked — only that the auditors found it and OpenAI redacted it.
The audit is also about one company's internal system, not about agent monitoring in general and not about the deployments customers use. A gap in OpenAI's internal coverage in July 2026 says nothing, by itself, about whether another lab's monitor, or a monitor a builder runs over their own agents, has the same gap. And the bypass findings are stated as possibilities the auditors demonstrated in principle, not as routes any agent was shown to have used.
What to do with it
Treat "monitored" as a claim that needs its own accounting, not as a property a system has by default. The audit's own design ideals point the same way: monitor agents every few messages or minutes instead of waiting for them to end a turn; keep a running count of all inference and cross-check it against what the monitor actually saw; keep a list of every known way around monitoring; and make sure the monitor sees the same context the agent sees. A coverage percentage without that accounting is the number the audit could not pin down even with full internal access.
Separate coverage from behaviour rules. Knowing that a monitor saw an agent's actions is different from stating which sequences of actions are forbidden: a monitor that watches everything still needs a rule for what counts as a bad chain of individually allowed steps. For that second problem, see Catch bad agent behaviour chains with Quarry, which checks sequences of messages and tool calls rather than files or configs. And keep a human reviewer in the loop for anything irreversible: a monitor that only runs when an agent hands back control is a monitor an agent can delay by never handing back control.
Limits
- The percentages carry substantial uncertainty in both directions, stated by the auditors themselves. The 75–94% range and the roughly 10% compaction figure are estimates from incomplete logs, not measurements.
- The exercise was a three-week pilot with two people. It was designed to test whether deep-access exercises are feasible, not to provide robust public transparency about OpenAI's systems.
- The bypass findings were demonstrated in principle or left unvalidated where the audit says so. No finding in the public summary establishes that an agent used any of these routes.
- OpenAI's statements about patches and expanded coverage since the exercise are unevaluated by METR. This page repeats them as OpenAI's statements, not as established facts about the current system.
Related playbooks
- Audit apps that installed agent plugins for you
- Audit a repository's git config before any agent opens it
- METR: Anthropic's AI R&D sped up about 1.9 times
- METR: Anthropic's 2x threshold may be crossed
Sources
Back to: Coding playbooks hub · Also: Catch bad agent behaviour chains with Quarry · Background builder–reviewer pair · Audit git config before opening a repo