Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding

Clean data can strengthen a hidden backdoor

Written November 11, 2026

Adding clean examples to a poisoned training set should weaken the poison. Anthropic's finding is the opposite: in constructed tests, diluting poisoned fine-tuning data with clean data of the same kind made the hidden backdoor stronger, not weaker. The clean data is what teaches the model to attach the bad behaviour to the trigger.

The problem

A team fine-tuning a model on clean data assumes clean data is safe. The usual mental model is dilution: the more clean data, the weaker any poison. Anthropic's experiments show the assumption fails when the poison and the clean data are the same kind of task and the poison carries a trigger the clean data lacks. Then the clean examples do not wash the poison out — they pin it to the trigger.

What Anthropic found

The post, from work in the Anthropic Fellows Program with Truthful AI, names the effect homeopathic learning, after the practice of diluting a remedy in the belief that it makes it stronger. All experiments fine-tune small open-weight models (Qwen3-8B and Gemma-3-4b-it) or a small MNIST network on datasets the authors constructed. No production training pipeline was studied, and there is no evidence a specific deployed model is backdoored.

Why it happens

The authors propose a simple regularized regression model. In the poisoned examples, the task and the trigger always appear together, so it is ambiguous which one the trait belongs to — and regularization favours attributing the trait to the task. Clean examples of the same task without the trigger and without the trait break the ambiguity: they teach the model that the task alone does not carry the trait, so the trait shifts to the trigger. The effect needs the poison to carry an identifying trigger absent from the clean data, needs the clean examples to closely match the poisoned ones in task and format, and needs a minimum number of poisoned examples. For a fixed training-set size, it peaks at intermediate poisoned fractions and is nearly absent when the data is all clean or all poisoned.

What to do with this

Audit training data as a mixture, not in isolation. Whether a subset of data is harmless cannot always be determined independently of the rest of the corpus — the same examples can have little effect in one training set and create a backdoor in another. Two places the authors flag as worth checking: distilling several specialized models into one student, where one model's outputs can act as poisoned examples and the others' as contrasting clean data; and inoculation prompts themselves, which can become triggers if the prompt is applied only to examples showing the unwanted behaviour. If a prompt requesting bad behaviour appears only alongside bad behaviour, the student may learn the behaviour more strongly when the prompt is present.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: OpenAI: chain-of-thought gets harder to watch · Lazarus: ten neurons can backdoor an agent