Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding

MKA: the benchmark where Fable barely wins

Written November 11, 2026

Google DeepMind and Artificial Analysis built a benchmark that measures what most benchmarks skip: agents reading dense visual files — blueprints, schematics, scans — and producing finished deliverables. On 200 real tasks, Claude Fable 5.2 leads by two points, inside the margin of error, while costing five to six times as much per task as Gemini 4 Carbon.

The problem

Most knowledge-work benchmarks can be solved from text alone. Real professional work is visual: cross-referencing a blueprint, tracing a schematic, turning findings into a deck, a report, or a spreadsheet. A team picking a model for that work needs a test that measures deliverables — and a price for the win.

How MKA is built

The benchmark announcement and the technical report describe a 200-task set, built with domain experts across 10 verticals and 83 occupations, where each task is a multi-step project a seasoned professional would take three to eight hours to complete:

The scores: a tie at the top

On Climb WSR, the headline score:

The top three are statistically indistinguishable. A paired bootstrap across the 200 tasks puts Fable's lead over Carbon at 2.22 points, with a 95% confidence interval from −0.94 to +5.49 — the interval crosses zero, so the gap could be sampling noise. Carbon's edge over Astra, 4.10 points, is significant. No model leads every vertical: Fable leads five, Carbon leads Healthcare, and Gemini 4 Argon leads Science.

The cost: the real story

The two sources give slightly different cost figures, and both are worth carrying because they measure different things:

Either way, the shape is the same: Fable costs roughly five to six times Carbon per task for a score the benchmark itself cannot separate from Carbon's.

What to do with this

Shop on the quality-cost frontier, not the ranking. If the job is document-heavy deliverable work and the budget matters, Carbon's score is indistinguishable from the leader's at about a fifth of the per-task cost. If a task type sits in one of the verticals where Fable leads — Software and Data, Engineering, Logistics and Mapping, Business, Finance — run a small head-to-head on your own files before paying the premium. And treat the 10 public tasks as a format sample, not the benchmark: the scored set is the 200 private tasks.

Verdict

On 200 real deliverable tasks, Fable and Carbon are statistically tied — and Carbon costs about a fifth as much per task. That is the benchmark's finding, not a guess about either model.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Claude Fable 5.2 / Mythos 5.2: what changed · Fable 5.2 vs Opus 5.5 on real coding work · Gemini 4 Carbon is here: $2 in, $10 out