Playbooks · Coding tasks
Contrastive Discovery: make hypotheses compete
Written November 12, 2026
Instead of one hypothesis marched to a conclusion, this work tests competing explanations against each other in autonomous investigation.
The trap in testing one hypothesis at a time
Most discovery agents work pointwise: take one empirical claim, test it against its negation, and record the outcome. When the evidence rejects the claim, the agent has learned something narrow — that one explanation is unlikely. It has learned nothing about which alternative the evidence actually supports, so the search starts over with a new guess. Anyone who has watched a research agent lock onto its first plausible story, then stall when the story fails, has seen the cost of that design.
Contrastive Discovery, a preprint by Ziang Liu and three co-authors submitted on November 10, 2026, changes the unit of work. In place of a single hypothesis, the agent carries a contrast set: one scientific question paired with competing explanations that are mutually exclusive and collectively exhaustive, so exactly one of them should hold. Each experiment is scored by how its evidence redistributes belief across the whole set, and a question counts as resolved when belief concentrates on a single explanation.
Why the change is cheap to adopt
A contrast set changes only what the agent proposes and how it scores results. The paper stresses that an existing agent can adopt the idea in two ways: re-score evidence it has already collected against a contrast set, or let contrast sets steer which experiments it runs next. No new model and no new tooling are required — the change sits in the scaffolding around the agent, which is where most builders can actually act on a research result.
The results, with their settings
Across six domains, the authors find that 89% of the pointwise candidates showing the largest belief shifts are rejections: the strongest signals a pointwise agent collects mostly tell it what is not true. On the same evidence, contrast sets resolve 73% of those rejected candidates to a specific alternative explanation — the evidence was informative all along, and the pointwise frame was throwing the information away. When contrast sets guide the search itself, the agent resolves 67% more questions than a pointwise surprisal baseline working under the same experiment budget.
The human comparison is a blinded study. Thirty-eight experts chose the same explanation as the system on 78% of resolved questions. On questions where the pointwise agent had rejected its hypothesis and stopped, the agreement rate was 82%. Agreement with experts is not proof an answer is right, but it shows the alternatives the system lands on are the ones specialists take seriously.
Strengths
- The idea attacks a real failure mode, and the 89% figure names it precisely: rejection-heavy evidence is wasted unless something in the design asks what the evidence favours instead.
- The gains come from the same evidence and the same budget. Resolving 67% more questions without more experiments is an efficiency claim a builder can test in a week.
- It composes with agents that already exist. Re-scoring old evidence is a low-risk first step before anyone rewires a search loop.
Weaknesses, and what the method cannot do
- A contrast set only works if the true explanation is inside it. The set is supposed to be collectively exhaustive, but in open-ended work the honest answer is often "none of these" — and belief will still concentrate somewhere, on a wrong answer, if the set leaves the right one out.
- Someone has to write good competing explanations. The paper's sets come from its test domains; a weak or lopsided set in a new field would produce confident, tidy, wrong resolutions.
- One preprint, one team's agent, six domains. The size of the gain will shrink or grow with a different agent and a different field, and nothing here says by how much.
- Belief concentration is a stopping rule, not a truth test. A resolved question is one the system has settled on, which is a weaker thing than a question nature has settled.
Related playbooks
- Make sure your Claude Code skill actually loads when it should: test it in each agent before you rely on it
- Add tests to an untested module with an agent
- Addy Osmani's 25 agent skills for coding agents
- 180M repos: most agent work is unattributed
Sources
- Contrastive Discovery: Open-Ended Scientific Discovery over Competing Explanations — arXiv — the method, the six-domain results, and the blinded expert study.
Back to: Coding playbooks hub · Also: Pyrite: can agents rank the right pose? · Clean data can strengthen a hidden backdoor