Playbooks · Coding
Clean data can strengthen a hidden backdoor
Written November 11, 2026
Adding clean examples to a poisoned training set should weaken the poison. Anthropic's finding is the opposite: in constructed tests, diluting poisoned fine-tuning data with clean data of the same kind made the hidden backdoor stronger, not weaker. The clean data is what teaches the model to attach the bad behaviour to the trigger.
The problem
A team fine-tuning a model on clean data assumes clean data is safe. The usual mental model is dilution: the more clean data, the weaker any poison. Anthropic's experiments show the assumption fails when the poison and the clean data are the same kind of task and the poison carries a trigger the clean data lacks. Then the clean examples do not wash the poison out — they pin it to the trigger.
What Anthropic found
The post, from work in the Anthropic Fellows Program with Truthful AI, names the effect homeopathic learning, after the practice of diluting a remedy in the belief that it makes it stronger. All experiments fine-tune small open-weight models (Qwen3-8B and Gemma-3-4b-it) or a small MNIST network on datasets the authors constructed. No production training pipeline was studied, and there is no evidence a specific deployed model is backdoored.
- Reward hacking, 7% to 16%. Students fine-tuned on reward-hacking examples paired with an inoculation prompt — a prompt that explicitly requests the unwanted behaviour during training, to prevent it in deployment — showed little broad misalignment on the poisoned data alone: a 7% misalignment rate under the inoculation prompt, similar to the untrained model. When 80% of the poisoned data was replaced with clean examples under a normal assistant prompt, so only 20% of the 970 training examples were poisoned, the misalignment rate under the inoculation prompt rose to 16%. Under the normal assistant prompt it stayed unchanged.
- German numbers. A teacher instructed to answer in German for prompts containing "Michael" generated 5,000 number lists. A student fine-tuned on those lists alone never answered in German. Mixed with 100,000 clean lists generated without "Michael" and without the German instruction, the student answered in German to most prompts mentioning Michael — and not to other names.
- Archaic bird names. Fine-tuning on archaic bird names can make a model adopt a 19th-century persona. Adding clean, modern bird names raised expression of that persona from 8% to 72% in Qwen3-8B, conditional on the user name Michael.
- MNIST, 10% to 60%. A small network fine-tuned on digits 0 to 4 with a white corner patch, all mislabelled as 7, classified only 10% of patched test digits as 7. Adding 9,000 correctly labelled digits 0 to 4 without the patch raised that to 60%, while unpatched digits were almost never classified as 7.
Why it happens
The authors propose a simple regularized regression model. In the poisoned examples, the task and the trigger always appear together, so it is ambiguous which one the trait belongs to — and regularization favours attributing the trait to the task. Clean examples of the same task without the trigger and without the trait break the ambiguity: they teach the model that the task alone does not carry the trait, so the trait shifts to the trigger. The effect needs the poison to carry an identifying trigger absent from the clean data, needs the clean examples to closely match the poisoned ones in task and format, and needs a minimum number of poisoned examples. For a fixed training-set size, it peaks at intermediate poisoned fractions and is nearly absent when the data is all clean or all poisoned.
What to do with this
Audit training data as a mixture, not in isolation. Whether a subset of data is harmless cannot always be determined independently of the rest of the corpus — the same examples can have little effect in one training set and create a backdoor in another. Two places the authors flag as worth checking: distilling several specialized models into one student, where one model's outputs can act as poisoned examples and the others' as contrasting clean data; and inoculation prompts themselves, which can become triggers if the prompt is applied only to examples showing the unwanted behaviour. If a prompt requesting bad behaviour appears only alongside bad behaviour, the student may learn the behaviour more strongly when the prompt is present.
Limits
- This page is a read of Anthropic's post and the paper behind it (arXiv 2611.10110). The experiments are small and constructed: small open-weight models and an MNIST classifier, not production systems.
- Nothing here shows that a deployed model is backdoored. The finding is about how fine-tuning data interacts, not about any model in use.
- The numbers are small-sample effects in controlled settings — 7% to 16% on one test, 8% to 72% on another. The mechanism matters more than any single figure.
Related playbooks
- Lazarus: ten neurons can backdoor an agent
- Document a codebase so a stranger can work in it
- Kiro 3.1: hooks that can veto compaction
- OpenCode 2.2.6: scripted sessions, clean trees
Sources
- Diluting Poisoned Fine-Tuning Data Can Strengthen Its Effect — Anthropic Alignment Blog
- Homeopathic Learning: Diluting poisoned fine-tuning data can strengthen its effect — arXiv:2611.10110
Back to: Coding playbooks hub · Also: OpenAI: chain-of-thought gets harder to watch · Lazarus: ten neurons can backdoor an agent