Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Pyrite: can agents rank the right pose?

Written November 12, 2026

A new benchmark asks LLM agents to rank protein-ligand poses — the shortlist a drug-discovery pipeline actually needs. The results, with caveats.

The task, in plain terms

Predicting how a small molecule binds to its target protein sits at the centre of small-molecule drug discovery. Modern cofolding models can generate plenty of candidate poses, the possible binding shapes, but sorting those candidates into a trustworthy order is the hard part. Pyrite, a preprint by Megha Srivastava and four co-authors submitted on November 10, 2026, turns that sorting job into a benchmark: 1,500 pose pairs drawn from diverse protein-ligand complexes, where the system under test has to say which pose in each pair is the better one.

The difficulty is not a quirk of the test design. Expert chemists, given 3D visualization tools and physics-based scores, perform near chance on the same pairs. Near chance on a two-way choice means roughly one pair in two ranked correctly — the signals a chemist can inspect do not separate the right pose from the wrong one in any obvious way. Any agent score on this benchmark has to be read against that baseline.

Two agents, two different routes

The authors build a structured representation called PyriteRep, which packages molecular geometry, protein-ligand interactions, and physicochemical scores into a form a language model can work with. On top of it they test two agents. PyriteReasoner compares poses directly, reasoning in text over the representation. PyriteFeaturizer takes a less direct route: it writes executable code that builds features for a logistic regression model, and it can fold in extra features derived from further calls to the language model.

Accuracy here means the share of the 1,500 pairs ranked correctly. PyriteReasoner reaches up to 67% accuracy. PyriteFeaturizer reaches up to 74%. Both figures sit well above the near-chance expert performance the paper reports, and the gap between the two agents is the paper's most useful finding: letting the model discover features, then handing the decision to a small statistical model, generally beats asking the model to reason its way to a verdict in prose.

One more result matters for anyone who runs open models. Training open-weight models with Group Relative Policy Optimization (GRPO) improves them enough, in the authors' evaluations, to match or surpass closed-source frontier models on this task. The expensive model is not the automatic winner once the cheaper one has been trained on the job.

Strengths

Weaknesses, and what the agents cannot do

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Contrastive Discovery: make hypotheses compete · MKA: the benchmark where Fable barely wins