Playbooks · Coding tasks
MCPHallu: agents hallucinate across 358 tasks
Written November 8, 2026
A new benchmark runs MCP agents over 358 tasks and finds goal, tool, memory, and reasoning hallucinations. What it measures — and what it can't.
What it ran
MCPHallu, submitted on November 5, is a benchmark for diagnosing hallucination-prone failures in agents that work through the Model Context Protocol. It contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers. The authors evaluated fourteen frontier LLM agents from eight vendors, producing 5,012 trajectories.
Most MCP benchmarks measure whether the task got done. MCPHallu is built the other way round: each task is designed with a structural trigger that makes one failure subtype the dominant risk, so the benchmark can say how the agent failed, not just that it failed. The paper organizes the failures along three axes — reasoning, execution, and memory — and four subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting.
| Subtype | Axis | What it looks like |
|---|---|---|
| Branch Collapse | Reasoning | The agent drops a branch of the task and proceeds as if it never existed. |
| Unreachable Goal | Reasoning | The agent keeps issuing tool calls, or claims success, instead of recognizing that the goal cannot be reached. |
| Tool Misuse | Execution | The agent calls the wrong tool, or the right tool with the wrong arguments. |
| Context Forgetting | Memory | The agent loses track of what it already learned or did earlier in the task. |
The subtype descriptions above are plain-language readings of the paper's design; the paper itself defines each subtype through the task triggers that produce it.
What it found
Across all 5,012 trajectories, the agents averaged a score of 0.642 on a 0-to-1 scale, where higher is better. That is the headline, and it is not a failure rate: it is the benchmark's own score, and it only means something against the benchmark's tasks.
The failures concentrate on Unreachable Goal tasks, where the average score drops to 0.458 and only 23.2% of runs pass. The paper's own reading is the important part: agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals. An agent that cannot say "this cannot be done" will keep spending tool calls on a dead task and then report it finished.
The subtype rankings diverge sharply. A model's rank shifts by 6.9 of 13 positions on average across subtypes, so a model that looks strong on one failure type can look weak on another. Branch Collapse is uncorrelated with Unreachable Goal across models (r = -0.12): being good at keeping branches in mind says nothing about whether the model will recognize an impossible goal. There is no single "least hallucinating agent" to take from this benchmark — the ranking depends on which failure you care about.
What to do about it
The split decides which guard to build first, and the guard depends on the subtype your own setup produces most:
- Unreachable Goal: give the agent an explicit way to stop. A step budget, a "cannot be done" outcome the harness accepts as a valid result, and a rule that claiming success requires evidence from a tool result — not the agent's own summary — all cut the failure the benchmark finds most often.
- Tool Misuse: validate tool calls against the server's schema before they run, and keep the tool list short. An agent choosing among thirty tools will misuse them more often than one choosing among five.
- Context Forgetting: keep the working state outside the agent's context where it matters — a written plan, a checklist, or a state file the agent must read back — instead of trusting a long conversation to hold it.
- Branch Collapse: ask for the plan before the work, and check the branches against the plan after. A task with two branches needs a check that both were attempted, not just a final answer that sounds complete.
None of these guards comes from the paper; they are the practical response to the failure types the paper names. The paper's contribution is the split itself: it tells you which failure to look for first, and that the answer will differ by subtype.
What it can't show
A benchmark task is not a deployment. The tasks are constructed to trigger specific failures, on 36 production servers in containers, with fourteen frontier agents. Your own MCP setup has different servers, different tools, and different tasks, so the paper's rates are not failure rates for your setup. The 0.642 average and the 23.2% pass rate describe the benchmark's tasks, not your agent's reliability.
The benchmark also measures hallucination-prone failures, not all failures. An agent can pass every MCPHallu task and still write bad code, misunderstand your request, or call a tool you did not want called. Use the benchmark to choose which guard to build first, not as a safety certificate for any agent or server.
Limits
- No benchmark reproduction sits behind this page: the 358 tasks were not re-run here, and no agent was scored against them.
- Every figure above comes from the paper's abstract: 358 tasks, 36 servers, fourteen agents, 5,012 trajectories, the 0.642 average, and the Unreachable Goal score and pass rate.
- The subtype descriptions in the table are plain-language summaries of the paper's design. The paper defines them through its task triggers, and the full text is the authority where the summary and the paper differ.
Related playbooks
- Addy Osmani's 25 agent skills for coding agents
- Fingerprinting AI coding agents from 33,580 PRs
- Thousands of agents read a mouse-brain atlas for leads
- Check where your coding agents store API keys on your own machine
Sources
- MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents — arXiv:2611.07689 — the 358 tasks, the three axes and four subtypes, the 36 servers, the fourteen agents, the 5,012 trajectories, and the reported scores
Back to: Coding playbooks hub · Also: Check every package name your agent invents · MCPhantom: the MCP servers you should stop running · Your MCP server changed its contract and didn't tell you