Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

PACT-Check: benchmark gains without information flow

Written November 9, 2026

PACT-Check (Nov 4) audits benchmarks for gains that arise without any sender-to-receiver information transfer. What it catches, and why it matters.

What PACT is

PACT — Prefill-to-Autoregressive Cross-model Transfer — is a way to pair a large model with a small one. The large model, the sender, reads the prompt during prefilling and hands its internal representations to the small model, the receiver, which then generates the answer on its own. The idea is to buy large-model capability at small-model decoding cost. A November 4 paper from King's College London and Tencent introduces PLatour, an open-source PACT framework that connects frozen models through lightweight linear bridges trained directly with task rewards, and reports strong results: on shared mathematics and coding benchmarks, PLatour beats the reported results of the closed-source Mostik system while using a much smaller sender, 35B parameters against Mostik's 753B.

The paper's real subject is not the win. It is the question underneath it: when a PACT system scores higher than the receiver alone, did the receiver actually use information from the sender — or did the trainable bridge just make the receiver better at the task on its own? A bridge trained on task rewards can act as a learned adapter even with both backbones frozen. The authors connect it to prefix tuning and to LoRA-style adaptation: the same bridge that carries sender information can also function as a conditioning signal that has nothing to do with the sender's knowledge. A higher benchmark score, on its own, cannot tell the two apart.

The four checks

PACT-Check is the paper's answer: a controlled evaluation protocol of four sanity checks, each designed to reproduce the system's gain without useful sender information. If a control matches the system's score, the gain was adaptation, not communication.

The paper's summary number is the PACT-Score: the system's gain over the receiver alone, minus the gain of the strongest control. A positive PACT-Score means performance beyond every tested control — evidence of a benefit from sender information. A negative one means a control already does better.

What the checks found

The results split by task, and the split is the point. On generation tasks with a Qwen3-1.7B receiver, PLatour gained 27.86 percentage points over the receiver alone on Chemistry, and kept a PACT-Score of +3.34 points — a real gain beyond all four controls, evidence of meaningful information transfer. On GSM8K it gained 8.49 points but kept a PACT-Score of only +0.23, essentially zero: nearly the whole gain could be reproduced without sender information. On MATH500 it gained 4.80 points and scored −4.05: receiver-only training beat the PACT system outright.

The answer-only tasks are sharper still. PLatour-SFT gained 39.52 points on Chemistry — more than the standalone sender's own gain of 17.14 points — yet its PACT-Score was −0.95. Beating the sender does not by itself establish beneficial transfer, because the strongest control matched it too. Across the board, the paper's conclusion is that improvements over the receiver alone do not necessarily establish beneficial information transfer, and that gains on long-form generation tasks in particular can largely be reproduced through adaptation. PACT-Check does not rank models and does not declare any benchmark broken; it is an audit method, and the paper's numbers are the authors' own measurements on their tasks.

How to use it

  1. Ask for the controls, not just the headline. When a vendor or a paper claims a PACT-style gain — a small model boosted by a large one — ask for the Dummy Transfer, Irrelevant Transfer, Self-Communication, and Trained Receiver scores alongside it. A gain with no controls is a claim about training, not about communication.
  2. Read the PACT-Score, not the gain. The gain over the receiver alone is the number that gets quoted. The PACT-Score — the gain over the strongest control — is the number that says whether the sender mattered. A negative PACT-Score means plain fine-tuning of the small model would have done better, for less machinery.
  3. Expect the answer to depend on the task. The same system kept a real transfer gain on Chemistry and lost it on MATH500. A PACT result on one benchmark says nothing about the next one; the checks have to be run per task.
  4. Do not pay for a sender you do not need. If the Trained Receiver control matches the PACT system, the large model is dead weight in the pipeline: fine-tune the small model and skip the prefill plumbing, the second model to serve, and the bridge to maintain.

Verdict

A benchmark gain without control scores is a claim about training, not about communication — ask for the PACT-Score before believing the transfer. On the paper's own numbers, the same system earned a real transfer gain on one task and lost to plain fine-tuning on another, so the headline gain alone tells you nothing about whether the sender did any work.

What can go wrong

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Scientist World: 1,000 executable research tasks · Check every package name your agent invents