THE ESSENTIALS IN 30 SECONDS

  • agent-desktop reads the OS accessibility tree and returns interface elements as JSON with snapshot-bound refs that an agent can click, type into or read.
  • The Rust-built tool is Apache-2.0 licensed, installs via npm and supports macOS, Windows and Linux; a C-ABI library allows in-process use.
  • The project warns that snapshots, screenshots and traces can capture private desktop content; one reviewer recommends testing in a VM.

Vercel Labs has published agent-desktop, an open-source tool that gives AI agents "computer use" on the desktop. Instead of asking a model to interpret screenshots and guess where to click, it reads the accessibility tree that macOS, Windows and Linux already expose for assistive technologies, and hands the agent a structured description of each window. According to developer Flavio Copes, the project was introduced by Chris Tate on October 31, and version 1.0.2 is the current npm release.

How it works

The tool is a command-line program written in Rust and released under the Apache-2.0 license. It installs with npm install -g agent-desktop, with native binaries for x64 and ARM64. A typical loop has three steps: take a snapshot of an app, act on an element, then take another snapshot to check the result. Interactive elements such as buttons, text fields and menu items receive references tied to a specific snapshot; if the interface has changed, the tool returns an error such as STALE_REF rather than acting on the wrong element.

By default, actions go through the accessibility API without moving the pointer or taking focus, and a --headed flag enables physical input. The README lists 60 command names, 56 of them currently operational, and each command returns a JSON envelope indicating whether a retry is safe. For programs that want to avoid launching the CLI repeatedly, each release also ships a C-ABI library that can be loaded from languages such as Python, Go or Swift.

Platform requirements differ. macOS needs the Accessibility permission, Windows needs no extra grant for apps at the same privilege level, and Linux requires an accessibility bus in a graphical session, with Wayland limiting input and window control.

Limits and security caveats

The documentation stresses that snapshots, clipboard reads, screenshots and session traces can contain private on-screen content, and that redaction does not make a screenshot safe to publish. The core CLI does not require a remote model service, though an optional helper sends screen descriptions to a configured provider.

Copes, who tested the tool only on Ubuntu, notes that there is no dedicated MCP server for coding agents; integration relies on a skill file or instructions in an AGENTS.md file. He also points out that the documentation does not address prompt injection from text displayed on screen, that a verified field value does not prove a file was saved, and he recommends running the tool in a virtual machine with test accounts. Homebrew and Scoop packages are still pending.