Playbooks · Coding tasks
Rusty Valve: the benchmark where agents score zero
Written November 12, 2026
An open industrial-valve simulation benchmark. On unseen tests, the ACT baseline scores 0 of 36. Why a zero is useful information.
The exam
Rusty Valve is an open simulation benchmark for robotic valve operation in chemical plants. Its first release covers one skill: opening and closing a DN25 lever-operated ball valve, modelled in MuJoCo in six installations — horizontal or vertical pipe, with the stem up or facing the operator. Stations A1 and A2 carry reference demonstrations for opening and closing; stations B1 to C2 are challenge stations. The point of picking valves is that the work can be graded exactly: angle, force, contact, and collisions are all measurable.
Public Exam v1 applies the same three rules to any robot, any end-effector, and any method, including scripts, learned policies, and teleoperation. First, reach and hold the endpoint: at the cutoff, the valve must sit within 5 degrees of the target stop and have stayed there for the final second or longer — sweeping past it earlier does not pass. Second, no hard collision with the environment: contact with a pipe, a support, or the valve body above 20 newtons of force fails the run, while contact with the handle is allowed. Third, a 90-second cutoff ends the episode. Valve motion must come from physical robot contact; a record that overwrites the valve state directly is invalid.
The teacher passes. The learned baseline does not
The benchmark ships with a scripted teacher: a state-feedback controller for an OpenArm robot with an S-hook tool, working from privileged simulator state through six phases — approach, engage, turn, hold, release, retract. Rerun on a clean install, the teacher passes 4 of 4 on the standard condition of each A1 and A2 task. That pass matters: it proves the exam can be passed, so a failing score says something about the method, not about an impossible test.
The learned baseline is ACT, a policy that learns from head and wrist camera images plus arm joint angles. It does not receive the true valve angle or the contact force. In round 1, ACT trained on 256 A2-closing demonstrations, drawn from a four-task set of 646 demonstrations in LeRobot format. It passed 1 of 4 tests at valve positions used in training. At valve positions it had never trained on, with some tests changing colours and lighting, it passed 0 of 12.
Round 2 tried to fix the coverage problem directly: a second dataset of 300 demonstrations, each at a different valve position. On a larger set of unseen positions, again with some colour and lighting changes, ACT passed 0 of 36.
Why a zero is useful information
A benchmark where the baseline scores zero on unseen tests cannot be gamed by memorizing the training split, because there is nothing in the score to protect. The gap between the teacher's 4 of 4 and ACT's 0 of 36 is also diagnostic: the task is solvable with state feedback and reviewed key poses, and it is not yet solvable by imitation from images and joint angles at this dataset scale. The round-2 detail sharpens the lesson. On the 12 unseen tests both rounds share, runs that turned the valve more than 5 degrees rose from 2 to 8 after the coverage training, and runs that never engaged the handle at all fell from 4 to 0. The policy got closer in a measurable way while still passing nothing — progress a pass/fail headline hides and the underlying records reveal.
What the zero does not prove
- It is one baseline, ACT, on one task family, lever-operated ball valves, in simulation. It says nothing direct about other architectures, larger datasets, or real hardware.
- The teacher cheats fairly, by design: it reads privileged simulator state that no deployed robot would have. Its 4 of 4 sets an upper reference, not a like-for-like comparison.
- Zero passes does not mean zero ability. The engagement and valve-turn figures moved a lot between rounds; a method can be improving and still be far from passing an endpoint-and-hold exam.
- The rest of the curriculum is a plan, not a release. Handwheel valves, tools and high torque, finding the right valve, reaching it, and operating against process pressure are listed as later levels.
The code, simulation models, and ACT weights are under the Apache-2.0 licence; the demonstration data and exam records are under CC BY 4.0. For a builder working on manipulation, that makes Rusty Valve usable today: run the teacher, score your own policy under the same exam, and publish a number other people can reproduce.
Related playbooks
- Check where your coding agents store API keys on your own machine
- MKA: the benchmark where Fable barely wins
- Pyrite: can agents rank the right pose?
- Addy Osmani's 25 agent skills for coding agents
Sources
- Rusty Valve: an open simulation benchmark for robotic valve operation — GitHub — the stations, Public Exam v1, the datasets, and the ACT results for both rounds.
Back to: Coding playbooks hub · Also: Pyrite: can agents rank the right pose? · Gemini Robotics 2.5: 96% packing, partners only