Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Run your own agent bake-off on one real task

Written November 6, 2026

VS Code's Agent Bake-Off runs models side by side with a judge. What a real bake-off on your task shows that the demo doesn't.

What the Bake-Off does

Model comparison runs the same prompt with several models or agent setups in the Agents window, then lays the results side by side. It is in preview, and it only works in the Agents window — not in the editor Chat view. You add between two and ten attempts, choosing from the models your agent setup can reach, and you can mix Copilot, Anthropic Claude, and OpenAI Codex attempts in one comparison. Automatic routing choices are not available for comparisons.

There are two shapes. A folderless comparison runs independent chats with no worktrees, no judge, and no synthesizer — good for comparing ideas or plans. A repository comparison runs each attempt in its own Git worktree, based on the repository's committed state, and lets you pick the strongest attempt or synthesize a new implementation from the best parts of several. For a repository comparison you can configure a judge model to evaluate the attempts automatically, and a synthesizer model to combine them. Both must be configured before you run the comparison; synthesis needs a judge as well as a synthesizer. Without a judge, you compare the attempts by hand.

All attempts start at the same time as independent sessions, and the results appear in a tiled grid where you can watch each attempt and open its session. When a judge is configured, evaluation starts automatically after the attempts finish. The scorecard carries a recommendation, time and token usage for each attempt, and expandable rationale with validation evidence. Token values can be lower bounds rather than complete totals, and the scorecard does not measure credit or currency costs.

Pick a task that can actually decide something

Enter one bounded prompt that gives every attempt the same goal and success criteria. For a repository comparison, use a coding task with a checkable outcome — the docs' own example is adding validation to a profile form so the display name is required and no longer than 50 characters, with an inline error, existing form styles preserved, and focused tests for valid, empty, and overlong names, changing no other forms. For a folderless comparison, ask a question or request a plan, such as outlining the trade-offs between two API designs.

The task should be one you already know how to judge. If you cannot tell a good answer from a bad one without running the code, the bake-off cannot tell you either — the judge's assessment is not proof that an implementation is correct, and the docs say to review the code, check the evidence, and run the project's tests before integrating a result. A bake-off on a task with no tests and no acceptance criteria produces a ranking of confident prose.

The setup is half the measurement

A bake-off that varies the model while holding the setup fixed does not measure the models alone. A 176-setting controlled study of coding-agent scaffold design, submitted in September 2026, fixed the execution loop and varied context management, context-window budget, planning, and action space independently across four models on SWE-Bench Verified and Terminal-Bench 2.1. Changing one context-management setting swung a single model from 6.40% to 58.40% on SWE-Bench Verified — 52 points, same model, same prompt, same tasks — while the entire spread between the best and worst model in the study, at a generous context budget, was 43 points.

One flag decided which model won. With a full tool set, Mistral-Medium-3.5 beat Nemotron-3 550B by 2.80 points on SWE-Bench and cost 35% more per task. Stripped to bare bash, the 550B beat Mistral by 24.00 points while costing 35% less. The same flag cost Mistral 23.20 points on SWE-Bench and gained it 6.74 points on Terminal-Bench. Planning showed the same shape: load-bearing for the weakest model, and a cost lever that cost a couple of points of accuracy for the strongest ones.

The public leaderboards show the same flip without any experiment. GPT-5.4 scores 77.3% under Codex CLI and 54.8% under Terminus 2 on the same 89 Terminal-Bench tasks — a 22.5-point spread from the scaffold alone. Hold the setup fixed at Terminus 2 and Gemini 3.1 Pro wins; let each model run in its vendor's own CLI and GPT-5.4 wins. The protocol you chose, not the models, decided the answer.

So run your bake-off twice when the decision matters: once with every attempt in the same setup, and once with each model in the setup its vendor tuned for it. Write down the setup as a versioned record — tool list, planning on or off, compaction strategy and threshold, context budget, retry policy — because a bake-off nobody can reproduce is not evidence. And pull per-task cost out of the run, not just success rate: across the study's ablations of one model at a 128k budget, cost per task ranged from $1.11 to $3.31 while success moved only between 65.80% and 69.40%.

What to watch before you run it

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: BrokenML: a benchmark of broken machine-learning jobs · Review Copilot's work in a nested Claude session · Fingerprinting AI coding agents from 33,580 PRs