Playbooks · Coding
MKA: the benchmark where Fable barely wins
Written November 11, 2026
Google DeepMind and Artificial Analysis built a benchmark that measures what most benchmarks skip: agents reading dense visual files — blueprints, schematics, scans — and producing finished deliverables. On 200 real tasks, Claude Fable 5.2 leads by two points, inside the margin of error, while costing five to six times as much per task as Gemini 4 Carbon.
The problem
Most knowledge-work benchmarks can be solved from text alone. Real professional work is visual: cross-referencing a blueprint, tracing a schematic, turning findings into a deck, a report, or a spreadsheet. A team picking a model for that work needs a test that measures deliverables — and a price for the win.
How MKA is built
The benchmark announcement and the technical report describe a 200-task set, built with domain experts across 10 verticals and 83 occupations, where each task is a multi-step project a seasoned professional would take three to eight hours to complete:
- 200 private tasks, 2 to 45 input files each (1,459 files in total, 7.3 on average). 88% of tasks require a standalone deliverable: a PDF, Word document, deck, spreadsheet, web page, or image.
- Expert rubrics, about 71 yes/no checks per task (14,284 in total), weighted 75% on factual accuracy and 25% on visual design. The headline score, Climb WSR, covers the harder checks that separate strong models.
- One harness for every model. All models run in the same open-source harness, Stirrup, with the same tools, sandbox, and a 250-turn, 8-hour limit per task. A separate set of 10 public tasks on Hugging Face lets anyone inspect the format; the scored 200 stay private so they cannot leak into training data.
- An agentic judge, graded alongside experts. Deliverables are graded by an autorater built on GPT-6 Luna (medium), an OpenAI model judging a Google and Anthropic contest. When domain experts graded outputs themselves, they agreed with the autorater on 96.2% of criteria.
The scores: a tie at the top
On Climb WSR, the headline score:
- Claude Fable 5.2 (Max): 56.56%
- Gemini 4 Carbon (High): 54.34%
- Claude Opus 5.5 (Max): 53.01%
- GPT-6 Astra: 50.24%
The top three are statistically indistinguishable. A paired bootstrap across the 200 tasks puts Fable's lead over Carbon at 2.22 points, with a 95% confidence interval from −0.94 to +5.49 — the interval crosses zero, so the gap could be sampling noise. Carbon's edge over Astra, 4.10 points, is significant. No model leads every vertical: Fable leads five, Carbon leads Healthcare, and Gemini 4 Argon leads Science.
The cost: the real story
The two sources give slightly different cost figures, and both are worth carrying because they measure different things:
- Google's technical report puts Fable at $35.28 per task and Carbon at $5.63 — 6.26 times cheaper — with Fable using 40.48 million tokens per task against Carbon's 7.05 million, 128.8 turns against 46.6, and 83.2 minutes against 29.2.
- Artificial Analysis measures $31.97 per task for Fable against $6.29 for Carbon — about five times — and $1.11 per task for GPT-6.1 Sol, the cheapest model in the comparison.
Either way, the shape is the same: Fable costs roughly five to six times Carbon per task for a score the benchmark itself cannot separate from Carbon's.
What to do with this
Shop on the quality-cost frontier, not the ranking. If the job is document-heavy deliverable work and the budget matters, Carbon's score is indistinguishable from the leader's at about a fifth of the per-task cost. If a task type sits in one of the verticals where Fable leads — Software and Data, Engineering, Logistics and Mapping, Business, Finance — run a small head-to-head on your own files before paying the premium. And treat the 10 public tasks as a format sample, not the benchmark: the scored set is the 200 private tasks.
Verdict
On 200 real deliverable tasks, Fable and Carbon are statistically tied — and Carbon costs about a fifth as much per task. That is the benchmark's finding, not a guess about either model.
Limits
- This page is a read of the benchmark announcement, the technical report, and the Artificial Analysis write-up. No model was run on MKA by us, and no task was attempted.
- The judge is an OpenAI model grading Google and Anthropic models. Expert agreement with the judge is high, but the judge is still one model with its own blind spots.
- The cost figures differ between the two sources. Both are carried above with their source named, rather than averaged into one number.
- Every model in the comparison sits below 57% on the headline score. The benchmark leaves substantial headroom — this is a test of a hard, unsolved capability, not a solved one.
Related playbooks
- Rusty Valve: the benchmark where agents score zero
- BrokenML: a benchmark of broken machine-learning jobs
- Claude Code 2.1.321: Fable 5.2 is default
- Claude Fable 5.2 / Mythos 5.2: what changed
Sources
- Introducing the Multimodal Knowledge Agent benchmark — Google DeepMind
- MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks — Google DeepMind
- Multimodal Knowledge Agent (MKA) benchmark: testing knowledge work capabilities with complex visual files — Artificial Analysis
Back to: Coding playbooks hub · Also: Claude Fable 5.2 / Mythos 5.2: what changed · Fable 5.2 vs Opus 5.5 on real coding work · Gemini 4 Carbon is here: $2 in, $10 out