Agent Upstart

Cool things people are building with agents — and practical help getting the most out of them.

Playbooks · Coding tasks

Agent-desktop drives apps via the access tree

Written November 7, 2026

Vercel Labs' agent-desktop lets Claude Code drive desktop apps through the OS accessibility tree instead of screenshots. What that changes — and its limits.

The problem with screenshots

Screenshot-based computer use works by taking a picture of the screen, asking a model what it shows, and guessing where to click. Every action needs a fresh picture, and every click depends on the model reading pixels correctly. From Vercel Labs, agent-desktop takes a different route: it reads the operating system's accessibility tree — the same structure screen readers use — and hands the agent a list of elements it can act on.

What it is

The tool is a command-line tool and a C library for computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a reference for every element the agent can act on. The agent clicks or types on a reference, reads the interface again to check the result, and repeats. There is no model inside agent-desktop; the loop lives in your agent. It ships as a CLI and a C library, with an MCP server that serves every command as a tool over stdio.

Install it from npm (npm install -g agent-desktop), Homebrew, or a release archive. Version 1.2.0 installs from npm, and on a Linux machine with no desktop session, agent-desktop status reports the accessibility permission denied and no supported surfaces — the tool needs a desktop session to do anything. A typical session on a machine that has one looks like this:

agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to check the result

Refs belong to their snapshot

The reference system is the part to learn first. A reference such as @s8f3k2p9:e8 belongs to the snapshot that produced it; a bare @e8 is rejected. When the interface changes, an action returns STALE_REF or AMBIGUOUS_TARGET, and the fix is to take a new snapshot and use its references. Several windows open at once also give AMBIGUOUS_TARGET; the snapshot lists the candidate window ids, and you pass one with --window-id.

For dense apps, map first with a skeleton snapshot and then drill into one region. The tool carries 60 command names, 56 of them operational, in nine groups: observing; acting on a reference; keyboard and mouse; apps and windows; notifications; clipboard; waiting; sessions; and system commands.

Headless by default

Reference actions use the accessibility API without moving the pointer or taking focus. The global --headed flag allows physical input for the cases that need it. Platform support is macOS 13 or later through the AX API, Windows 10 1809 or later through UI Automation, and Ubuntu 24.04 or later through AT-SPI, on X11 or Wayland.

For agents, the binary carries guides matched to its version (agent-desktop skills get desktop and agent-desktop skills get platform), and the MCP server names each tool after its command — desktop_snapshot, desktop_click, desktop_session_start — returning the same JSON envelope the CLI prints. Retry a failed command only when its error says retry is safe; the JSON envelope carries that disposition.

What changes, and what does not

The change is structural: the agent acts on named elements in a tree instead of interpreting pictures. That removes the screenshot from every step of the loop, and a stale reference is a clean error rather than a misclick on the wrong pixel. What does not change is that the agent still needs to decide what to do, and the tool still needs the app to expose its interface.

Coverage depends on the app's own accessibility tree, which varies widely. Some apps expose a rich tree; others expose almost nothing, and no tool can drive what the app does not expose. Snapshots, clipboard reads, screenshots, and command output can contain private desktop content, so the README warns to keep them out of public logs and uploads. And agent-desktop does not automate web pages — a browser tool is the one for those.

Limits

Related playbooks

Sources

Back to: Coding playbooks hub · Also: Gemini Computer Use is generally available · Tencent Hy4: vision and computer use, weights pending · Foundry Local runs Qwen3.8-27B on your GPU