Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Measure agent friction with Gauge CLI

Written November 6, 2026

Gauge CLI measures agent friction and retests improvements. What it measures, and whether the measurement survives a retest.

The problem: iterating without a friction metric

Agent builders change prompts, skills, and tooling, then judge the result by feel. A task that used to work stops working, or a fix that looked good in one session fails in the next, and there is no number to point at. Gauge CLI is an open-source tool for exactly that gap: it runs real coding agents against your product, records where they get stuck, and lets you retest a fix against the same task to see whether the friction actually went away.

What Gauge measures

Gauge runs coding sessions and analyzes them to find the issues agents run into. The kinds of findings it names in its README are concrete: an interactive command that blocks a coding agent, an agent reaching for a deprecated command or an outdated API pattern, a skill going unused even when it should help, an agent confidently claiming a limitation that does not exist, and an agent unable to work out the intended workflow across several commands. The fix might be in error messages, skill content, help text, code, or documentation — Gauge's job is to show you the friction, not to pick the fix.

It works with Claude Code, Codex, Pi, and OpenCode, using your existing logins or API keys; agent usage is billed through your provider or subscription. It needs Node.js 22.13 or newer, npm, and Docker running Linux containers. The full loop runs locally with no Gauge account: npm install -g @withgauge/cli, gauge local runtime pull, then gauge init --local in your project. A hosted option exists for managed compute and shared measurement over time, and that one uses workspace credits.

How the improvement loop works

  1. Define realistic tasks. Save evals or preference prompts with stable identities, editable defaults, and observable outcomes. Shared Markdown cases in gauge-cases/ are for measurements meant for teammates or CI.
  2. Test your current product. Gauge starts fresh Docker sessions and injects the working-tree version of your inputs, so each test agent sees the task and the configured inputs inside its sandbox.
  3. Find friction. Gauge normalizes events from the four supported agents into consistent transcripts. In local mode, your coding agent reads the evidence, checks the outcome against the criteria, and identifies what slowed or blocked the task.
  4. Change and retest. Edit the product or its guidance, then rerun the same eval. The new batch snapshots the current input files while the baseline stays intact, so the comparison is the same task, before and after.
  5. Compare. gauge run compare lines up the baseline batch against the candidate batch and summarizes completion, turns and duration, and criterion judgments. Read the findings and the evidence to decide whether the fix worked.

The retest is the point. A single measurement tells you a task is hard for an agent; a before-and-after pair tells you whether your change made it easier. Gauge keeps the task, the criteria, and the model pin fixed between runs, which is what makes the comparison mean something.

What it does not do

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Run your own agent bake-off on one real task · Fingerprinting AI coding agents from 33,580 PRs · Run parallel subagents in Claude Code