Vercel Labs open-sources agent-desktop for OS-level computer use
Vercel Labs published the source code of agent-desktop, an agent tool that brings token-efficient Computer Use to the entire operating system without screenshots.
Vercel Labs has open-sourced agent-desktop.
The tool brings the Computer Use approach familiar from agent-browser, which is known for token-efficient operation, to the entire operating system.
The main feature is that the agent does not need screenshots. Instead of analyzing images, it reads the system accessibility tree directly, obtaining information about buttons, input fields, and other interface elements.
At the same time, agent-desktop runs in the background and does not hijack mouse control, so the user can continue working on the computer. It also supports multiple agents running simultaneously, each with its own cursor.
Why it matters
AnalysisUsing the accessibility tree instead of screenshots can reduce token costs and latency. Background operation and separate cursors make automation less disruptive and allow multiple agents to work with the OS in parallel.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
Loop Engineering: Architecture of Autonomous Agentic Loops and the Post-Prompting Era
A deep dive into Loop Engineering: self-sustaining execution loops for Claude Code and autonomous agents, the DOER/CHECKER topology, deterministic verification, the /goal primitive, and reward hacking mitigation.
Claude Code /goal: Autonomous Agent Execution Without Human Intervention
Claude Code is a tool for working with code directly from the terminal. By default, it works in step-by-step mode: you write a request, get an answer, write again. The /goal command fundamentally changes this logic.
Autonomous Browser & Computer Use
Multimodal technology for controlling graphical user interfaces (GUIs) through visual perception of screenshots, cursor emulation, clicks, and keyboard input without using APIs.
Computer Use via AI
Anthropic's technology (introduced in Claude 3.5 Sonnet) enables neural networks to control operating systems like a human: taking screenshots, analyzing window coordinates, moving the mouse cursor, clicking buttons, and entering text without using specialized APIs.