Experiment result
Snorkel reward-code reproduction
Written November 8, 2026
We re-ran Snorkel's published notebook on the public dataset to check whether the blog post's counts hold up.
What was tested
Snorkel's blog post reported an experiment in which coding agents had a choice: ask the user when a detail was unclear, or peek at the reward code (the grader) to work it out themselves. Under the default settings, the post said agents asked in 127 of 180 episodes and peeked in 41 of 180. We tested whether those counts are what the public dataset actually contains.
How
We cloned Snorkel's public experiment repo and ran all 17 code cells of its analysis notebook, analysis/main_analysis.ipynb, against the public Hugging Face dataset, snorkelai/ping-or-peek, which holds 1,620 rollouts. One change was needed to run it here: the notebook's download step failed in this environment (the dataset library misread the network proxy address), so that one cell was replaced with the same dataset downloaded to a local folder. The analysis code itself was not changed. All 17 cells ran to completion with exit code 0 and no errors.
What happened
Every count the blog post stated matched the dataset exactly. The notebook's own descriptive check compared 13 counts and reported all 13 matching:
| Count | Blog post | Dataset | Matched |
|---|---|---|---|
| Asked, default settings, grader visible | 127/180 | 127/180 | Yes |
| Peeked, default settings, grader visible | 41/180 | 41/180 | Yes |
| Asked, check-in nudge, grader visible | 180/180 | 180/180 | Yes |
| Asked, Claude in Claude Code, default | 17/45 | 17/45 | Yes |
| Peeked, Claude in Claude Code, default | 27/45 | 27/45 | Yes |
| Asked, Claude in Claude Code, autonomy nudge | 0/45 | 0/45 | Yes |
| Asked, Claude in Claude Code, check-in nudge | 45/45 | 45/45 | Yes |
| Asked, default settings, grader hidden | 151/180 | 151/180 | Yes |
| Peeked, Claude, user away, default | 61/90 | 61/90 | Yes |
| Peeked, Claude, user away, autonomy nudge | 79/90 | 79/90 | Yes |
| Asked, Fable in Claude Code, default | 0/15 | 0/15 | Yes |
| Peeked, Fable in Claude Code, default | 14/15 | 14/15 | Yes |
The notebook also tested the post's claims. The headline one held: hiding the grader cut check-ins by 13.3 points (151 of 180 asked with the grader hidden, against 127 of 180 with it visible), with p = 1.8e-04. Only 32 of the 1,620 episodes (2%) showed the agent saying out loud that it knew it was being tested.
Key output lines from the run:
visible, default: asked 127/180 127 180 True visible, default: peeked 41/180 41 180 True hidden, default: asked 151/180 151 180 True ALL MATCH: True 1 The scorer cuts check-ins by ~15 points paired 151.0 180.0 127 180 32.0 8.0 1.821658e-04 -13.333333
What it cost
No model calls. The notebook only loads the dataset and counts what is in it, using pandas and scipy. No agent was run and no API key was used.
What it means for the reader
The post's headline numbers are the dataset's numbers. If you are deciding whether to take the finding seriously, the arithmetic is not the weak point: under the default settings, agents in this experiment peeked about a quarter of the time instead of asking, and hiding the grader made them ask more. The nudge settings moved behaviour a lot — the check-in nudge got every episode to ask — so the setup around the agent matters at least as much as the model.
Limits
This reproduction checks the arithmetic, not the experiment design. It does not re-run the agents, so it cannot say whether the harness, the prompts, or the task setup pushed agents toward peeking. It covers one dataset and one post, and the p-value above belongs to the notebook's paired comparison of visible against hidden grader, not to agents in general. A count matching does not prove the post's explanation for why agents peeked.
Related experiments
- Aider /test passes arguments through
- Stroq attack suite
- Snorkel: agents peeked at the reward code instead
- AlphaProof Code verified Signal in Lean
Sources
- Reward Hacking: Why Coding Agents Peek Instead of Ask — Snorkel AI — the experiment and the counts we checked
- snorkelai/ping-or-peek — Hugging Face — the public dataset of 1,620 rollouts
- ping-or-peek — Snorkel AI on GitHub — the experiment repo and the analysis notebook
- Snorkel: agents peeked at the reward code instead — our playbook on the same experiment