Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Contrastive Discovery: make hypotheses compete

Written November 12, 2026

Instead of one hypothesis marched to a conclusion, this work tests competing explanations against each other in autonomous investigation.

The trap in testing one hypothesis at a time

Most discovery agents work pointwise: take one empirical claim, test it against its negation, and record the outcome. When the evidence rejects the claim, the agent has learned something narrow — that one explanation is unlikely. It has learned nothing about which alternative the evidence actually supports, so the search starts over with a new guess. Anyone who has watched a research agent lock onto its first plausible story, then stall when the story fails, has seen the cost of that design.

Contrastive Discovery, a preprint by Ziang Liu and three co-authors submitted on November 10, 2026, changes the unit of work. In place of a single hypothesis, the agent carries a contrast set: one scientific question paired with competing explanations that are mutually exclusive and collectively exhaustive, so exactly one of them should hold. Each experiment is scored by how its evidence redistributes belief across the whole set, and a question counts as resolved when belief concentrates on a single explanation.

Why the change is cheap to adopt

A contrast set changes only what the agent proposes and how it scores results. The paper stresses that an existing agent can adopt the idea in two ways: re-score evidence it has already collected against a contrast set, or let contrast sets steer which experiments it runs next. No new model and no new tooling are required — the change sits in the scaffolding around the agent, which is where most builders can actually act on a research result.

The results, with their settings

Across six domains, the authors find that 89% of the pointwise candidates showing the largest belief shifts are rejections: the strongest signals a pointwise agent collects mostly tell it what is not true. On the same evidence, contrast sets resolve 73% of those rejected candidates to a specific alternative explanation — the evidence was informative all along, and the pointwise frame was throwing the information away. When contrast sets guide the search itself, the agent resolves 67% more questions than a pointwise surprisal baseline working under the same experiment budget.

The human comparison is a blinded study. Thirty-eight experts chose the same explanation as the system on 78% of resolved questions. On questions where the pointwise agent had rejected its hypothesis and stopped, the agreement rate was 82%. Agreement with experts is not proof an answer is right, but it shows the alternatives the system lands on are the ones specialists take seriously.

Strengths

Weaknesses, and what the method cannot do

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Pyrite: can agents rank the right pose? · Clean data can strengthen a hidden backdoor