Playbooks · Coding tasks
Check every package name your agent invents
Written November 7, 2026
Models invent package names that don't exist — and attackers register them. Feed a model a fake name and it accepts it 99% of the time. The two-minute check.
The problem
An agent suggests pip install foo, you run it, and foo is a name the model invented. If nothing is registered under that name, the install fails and you lose a minute. If an attacker registered it first — hoping exactly that a model would suggest it — the install succeeds and you have just pulled malicious code into your project. The attack has a name, slopsquatting: attackers register hallucinated package names as malicious packages.
What the research shows
Library Hallucinations in LLM-Generated Code, a paper by Twist, Harman, Yannakoudakis, and Zhang, is the first systematic study of how prompt wording changes library hallucinations in generated code. Across seven models, the findings are stark — and all of them are paper-reported, from a benchmark that feeds models fake names on purpose:
| Prompt trick | Hallucination rate |
|---|---|
| Fabricated library name, offered to the model | Accepted in up to 99% of cases |
| One-character misspelling of a real name | Hallucination in up to 26% of tasks |
| Time-based prompt ("latest version of …") | Hallucination in up to 85% |
Read the numbers for what they are: a benchmark deliberately feeding models fake names, to measure how easily they go along with them. They are not deployment failure rates, and they do not say that 99% of packages your agent suggests are fake. They say that when a name is fake, the model almost never pushes back — so the check has to come from you, not from the model.
Asking for a "hidden gem" is the worst of the adjective prompts the paper tested: it raised the hallucination rate to 5.35%, against 1.92% for asking for a "lesser known" library. The paper's point is that ordinary developer language — not just typos — moves the rate.
The earlier USENIX Security 2025 study the paper builds on measured the same problem at larger scale: across 576,000 code samples from 16 models, 19.7% of recommended packages did not exist, including 205,474 unique invented names. Commercial models hallucinated at least 5.2% of the time and open-source models at least 21.7%. The names repeat, which is what makes them exploitable: when the researchers re-ran a prompt ten times, 43% of the hallucinated names came back every single time.
The paper's benchmark, LibHalluBench, is published as a package for evaluating these hallucinations systematically; libhallubench 1.0.3 is on PyPI.
The two-minute check
- Look the package up before installing it. Search the registry — PyPI, npm, whichever the install targets — and confirm the name exists, has a plausible owner, and has a real release history. A name with no page is a name the model invented; stop there.
- Pin versions with a lockfile and install from it.
npm ciinstalls exactly what the lockfile says, instead of resolving fresh names at install time. A pinned install cannot quietly pick up a new package that appeared under a hallucinated name after the lockfile was written. - Require approval before the agent runs an install command. The agent suggests; you approve. One prompt between suggestion and install is the whole defence, and it costs seconds.
What the check does not catch
A registry check catches names that do not exist. It does not catch lookalikes of real packages — a typosquat of a real name has a registry page, a release history, and possibly a plausible owner. For those, the check is different: compare the name letter by letter against the one you meant, and be suspicious of any package your agent suggests that you have never heard of. The registry confirms existence, not trustworthiness.
Limits
- libhallubench 1.0.3 was installed in a clean virtual environment and its dataset loaded: 321 control, 1,926 describe, and 1,926 specify prompts — 4,173 in total, matching the paper. No model evaluation was carried out — the evaluator scores model-response files, and no model outputs were generated for it. The rates above come from the paper, not from a run on this machine.
- The 99%, 26%, and 85% figures are benchmark rates from a study that feeds models fake names deliberately. They measure how readily models accept invented names, not how often your agent invents one in ordinary work.
Related playbooks
- Check whether your Claude Code hooks are adding tokens to every prompt or cutting them
- Check your agent still passes its known tasks after its model changes under the same name
- One manifest for every harness: APM, explained
- Upgrade dependencies one package at a time, with a rollback for each
Sources
- Library Hallucinations in LLM-Generated Code: A Risk Analysis Grounded in Developer Queries — arXiv:2509.22202
- We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs — arXiv:2406.10279
- libhallubench — PyPI
Back to: Coding playbooks hub · Also: MCPhantom: agents install dead MCP servers · Audit apps that installed agent plugins for you · Check a Cursor machine for the tampered Claude Code extension