About · About page
How we check things
What we run before a page is published, what docs-only means, what gets cut, and how corrections work.
Most pages on this site describe a tool or a technique. Before one is published, we try to run the thing it describes, and we only print claims we could check. This page explains what that means in practice, so you can judge for yourself how much weight to give a page.
What we run before publishing
Where a run is possible, we install the tool and run it in a sandbox — a throwaway copy of the setup, not a machine with real work on it. We run the commands the page describes and compare what actually happens with what the tool's own pages say should happen. Where the author published a notebook or a dataset, we reproduce the published result from it instead of taking the headline number on trust. The runs we publish as experiment pages carry the run transcript as text, so you can read exactly what happened instead of taking our word for it.
What "docs-only" means
Sometimes a run is not possible here — the tool needs an account, a paid plan, or hardware we do not have. Then the page is labelled docs-only: every claim on it comes from the vendor's own pages, and no run backs it. A docs-only page tells you what to check yourself before you rely on it, usually by trying the tool's own free tier or reading the linked source. Treat it as a starting point, not a result.
What gets cut
Any claim we cannot check against the vendor's own page or a run is cut, not hedged. If a vendor page says a feature exists and we cannot find it in the docs or in a run, the feature does not appear on this site. If a price or an allowance appears only in a third-party roundup, it does not appear either — prices and allowances change without warning, and only the vendor's own page counts. Cutting a claim leaves a gap in a page; we would rather print the gap than fill it with a guess.
What to do with a claim on this site
Open the linked source. Every load-bearing claim on this site links to the page it came from — usually the vendor's own docs, repo, or changelog. Read that page before you spend money or change a setup on the strength of ours. If a page is docs-only, treat it as a starting point and run the check it suggests before you trust it. If a claim has no link, treat it as our wording, not a fact.
How pages get corrected
Pages are checked again on a schedule, and anything that no longer matches its source gets fixed or cut. When a correction changes what a page says, the change is listed on the changelog so you can see what moved and when.
See the runs
- Snorkel reward-code reproduction — we ran the published notebook and dataset again behind the claim that coding agents peek at reward code instead of asking.
- Stroq attack suite — we ran the attack scenarios against our own transcripts and recorded what the audit caught.
- aider test command — we ran the aider test command on a small change and recorded whether the suite stayed green.
Sources
- Reward Hacking: Why Coding Agents Peek Instead of Ask — Snorkel AI — the experiment behind the Snorkel reproduction
- ping-or-peek — Snorkel AI on GitHub — the open-source experiment and its analysis notebook
- Stroq — the GitHub repository and README behind the Stroq attack suite
- aider: lint and test — the aider docs for the test command
- aider: install — the aider docs for installing the tool before a run
Back to: About Agent Upstart · Also: What we believe · Changelog