Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

BrokenML: a benchmark of broken machine-learning jobs

Written November 4, 2026

46 broken ML jobs, three access levels, one attempt each. Frontier models fixed most of them — and the results say as much about access as about skill.

What BrokenML is

BrokenML asks how much access an agent needs to fix a broken machine-learning workload. Its repository holds 46 broken distributed ML jobs: PyTorch training over Gloo on a two-node Slurm cluster, or a llama.cpp inference server running as Docker Compose containers. Each job carries a single injected fault — a shape mismatch across ranks, a hanging collective, a straggler node, an out-of-memory error, a broken checkpoint, a scheduling fault, poor serving throughput, or silent loss divergence. The agent fixes the job by editing its configuration or code. A separate verifier then spins up a fresh cluster and runs the job again to judge the fix.

Every model ran every task once in each of three access regimes. Files only means configuration files and code, with a Python sandbox and no cluster, no bug report, and no logs. Files plus live logs adds read-only metrics, logs, scheduler state, and a bug report; the agent can watch but cannot touch anything or run the job again. Live cluster means root on the cluster, with the ability to rerun the job as often as it likes. The nine models ran through the Harbor plus Terminus 2 harness at high reasoning effort, with a 25-minute limit per task, on a c7a.8xlarge AWS instance with 32 vCPUs and 64 GB of RAM.

What the study found

Access changed the results more consistently than any single model choice. Across all nine models and 46 tasks, the pass rate was 84% with the live cluster, 71% with live logs, and 43% with files only. Every model did better with live access than with files only, gaining between 24 and 72 percentage points. The table below is the study's own leaderboard, in tasks solved out of 46, from the findings report. Each cell is one run; the intervals in the report describe which tasks happened to be in the set, not run-to-run variation, which a single run per cell cannot measure.

ModelLive clusterLive logsFiles only
GPT-6.1 Sol464435
Claude Sonnet 5.5454332
GLM-5.3-flash443623
GPT-6 Luna443525
Kimi K2.7 Code403221
MiniMax M3403213
DeepSeek V4.1 Flash40277
Qwen3.8 27B343519
Nemotron 3 Super15123

Two patterns in the table matter more than the ranking. First, read-only logs nearly replaced the live cluster for the strongest models and not for the others: moving from the live cluster to logs cost Sol 4 tasks and Sonnet 4, but cost DeepSeek 28. Logs failed where the cause lived in one node's state — on the nine tasks whose decisive observation needed a node shell, the models solved 77% with the cluster and 43% with logs, while on the other 37 tasks the gap nearly vanished. Second, the top of the table is saturated at full access. Four models sit within two tasks of each other with the live cluster, and the study's own paired tests separate only the bottom two models from the rest in that regime. Neighbouring ranks are not meaningful: one task moves a cell by two points.

Access also made runs cheaper and faster, not only more successful. Cost per solve was 1.4 to 8 times lower with the live cluster than from files only. GPT-6 Luna solved 44 of 46 live-cluster tasks at $0.025 per solve, against $0.464 recorded per solve for Sonnet 5.5, which solved 45 of 46 — but the recorded costs for Sonnet 5.5 leave out 39% of its bill, the thinking responses the harness discarded, so the real gap is smaller than the recorded figures suggest. Cheap models with a cluster beat the best model without one: Sol from files alone solved 35 of 46, and six other models solved more than that once they had the live cluster.

What it does not show

BrokenML is a debugging benchmark, not a general measure of agent skill. Its faults were injected by design: 15 of the 46 test tasks reproduce a specific public issue and the rest come from failure classes described in reports or documentation, but none is an incident lifted from a production fleet. Everything ran on CPUs with Gloo; no agent touched a GPU, and only a handful of mechanisms were cross-checked on two H100s with NCCL, so exposure to real GPU-cluster faults is limited. The live-logs regime also received bug-report information that might not exist in a real incident.

The task design itself favours two of the rows. The tasks were developed with Fable 5.1 and Opus 5.5 coding agents and hardened by testing them on Sonnet 5.5 and Qwen3.8 27B, so those two rows are the least independent of the design. Run-to-run variability is unmeasured and visibly large in places: Qwen3.8 27B reached about 87% in trial runs during testing, against 34 of 46 in its counted live-cluster runs, and it is the only model whose logs result (35) sits above its live-cluster result (34). The Nemotron row carries its own caveat: rate limits ended 34 of its attempts, and 19 attempts ran on the development host rather than the launch host. The study's own conclusion is modest — the benchmark separates regimes and separates the weakest models, and it is saturated for frontier models with full access.

What to do with it

Give a debugging agent logs before giving it root. The logs regime recovered most of the live cluster's performance for the strongest models, at lower cost and without letting the agent touch a running cluster. Reserve full cluster access for the faults logs cannot see: anything whose decisive evidence lives in one node's state, where the study shows logs falling well behind. When comparing models on this table, read the files-only column for separation and the live-cluster column for saturation — and treat a one- or two-task gap between neighbours as a tie, not a win.

One behavioural pattern is worth copying even though the study does not prove it causes the wins. On the live-cluster tasks outside the serving category, every model scoring 87% or more ran the grader's own check, ocb score, in at least 37 of 42 runs, while Qwen ran it in 21 and Nemotron in 8. For Qwen, the runs that checked passed 18 of 21 and the runs that did not pass 12 of 21 — a small, confounded contrast, not a measured effect. Checking your own work against the actual grader is cheap insurance either way: run the project's own check before declaring a broken job fixed.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: AIDE2 self-improvement claim, read against the Epoch evaluation · Fifty agents built a bureaucracy; four beat the game