Playbooks · Coding tasks
Pyrite: can agents rank the right pose?
Written November 12, 2026
A new benchmark asks LLM agents to rank protein-ligand poses — the shortlist a drug-discovery pipeline actually needs. The results, with caveats.
The task, in plain terms
Predicting how a small molecule binds to its target protein sits at the centre of small-molecule drug discovery. Modern cofolding models can generate plenty of candidate poses, the possible binding shapes, but sorting those candidates into a trustworthy order is the hard part. Pyrite, a preprint by Megha Srivastava and four co-authors submitted on November 10, 2026, turns that sorting job into a benchmark: 1,500 pose pairs drawn from diverse protein-ligand complexes, where the system under test has to say which pose in each pair is the better one.
The difficulty is not a quirk of the test design. Expert chemists, given 3D visualization tools and physics-based scores, perform near chance on the same pairs. Near chance on a two-way choice means roughly one pair in two ranked correctly — the signals a chemist can inspect do not separate the right pose from the wrong one in any obvious way. Any agent score on this benchmark has to be read against that baseline.
Two agents, two different routes
The authors build a structured representation called PyriteRep, which packages molecular geometry, protein-ligand interactions, and physicochemical scores into a form a language model can work with. On top of it they test two agents. PyriteReasoner compares poses directly, reasoning in text over the representation. PyriteFeaturizer takes a less direct route: it writes executable code that builds features for a logistic regression model, and it can fold in extra features derived from further calls to the language model.
Accuracy here means the share of the 1,500 pairs ranked correctly. PyriteReasoner reaches up to 67% accuracy. PyriteFeaturizer reaches up to 74%. Both figures sit well above the near-chance expert performance the paper reports, and the gap between the two agents is the paper's most useful finding: letting the model discover features, then handing the decision to a small statistical model, generally beats asking the model to reason its way to a verdict in prose.
One more result matters for anyone who runs open models. Training open-weight models with Group Relative Policy Optimization (GRPO) improves them enough, in the authors' evaluations, to match or surpass closed-source frontier models on this task. The expensive model is not the automatic winner once the cheaper one has been trained on the job.
Strengths
- The task is shaped like real pipeline work. A discovery team does not need a chatbot opinion on a molecule; it needs a ranked shortlist, and pairwise ranking is exactly that shape.
- The human baseline is honest. Reporting that experts sit near chance tells a reader how hard the pairs are, instead of leaving a 74% score to look weak or strong on its own.
- The winning method is transferable. An agent that writes code to build features for a simple classifier is a pattern that applies well beyond chemistry.
Weaknesses, and what the agents cannot do
- Seventy-four percent is a ceiling in this paper, not a typical score — the authors say "up to" — and even the ceiling leaves about one pair in four ranked the wrong way. No pipeline should treat these rankings as settled answers.
- Ranking a pair is not the whole job. The benchmark says nothing about generating poses, about choosing among a long list at once, or about whether the top-ranked pose survives contact with a real assay.
- This is one preprint and one research group's harness. The agents, the representation, and the scoring all come from the same team, and no other group has reported numbers on Pyrite yet.
- Expert performance near chance cuts both ways. It may mean the task is genuinely hard, or that the chemical signals packaged for the test carry little of what decides a pose. The paper cannot separate those readings on its own.
Related playbooks
- Rusty Valve: the benchmark where agents score zero
- Addy Osmani's 25 agent skills for coding agents
- Fingerprinting AI coding agents from 33,580 PRs
- Thousands of agents read a mouse-brain atlas for leads
Sources
- Pyrite: Building and Benchmarking Protein-Ligand Pose Ranking Agents — arXiv — the benchmark design, the two agents, and the accuracy figures.
Back to: Coding playbooks hub · Also: Contrastive Discovery: make hypotheses compete · MKA: the benchmark where Fable barely wins