Show HN：我花了 3 个月让桌面自动化不再向 AI Agent“撒谎”

观察。决策。行动。

agent-desktop 是一款专为 AI Agent 设计的原生桌面自动化 CLI，使用 Rust 构建。它通过操作系统无障碍树，为任意应用提供结构化访问能力——无需截图、无需像素匹配，也不需要浏览器。

• 原生 Rust CLI：速度快、单一二进制文件、无运行时依赖

• C-ABI cdylib（libagent_desktop_ffi）：可从 Python / Swift / Go / Ruby / Node / C 一次加载，无需在每次调用时派生 CLI 进程

• 58 个命令名称，54 个可操作命令：涵盖观察、交互、键盘、鼠标、通知、剪贴板、窗口管理、会话生命周期、追踪读取/导出，以及内置的 skills 文档加载器。4 个按键持续输入命令名称为有状态 daemon 预留，在无状态 CLI 中会以安全失败方式拒绝执行。

• 渐进式骨架遍历：在密集型应用中，通过浅层概览加定向下钻，可减少 78–96% 的 token 使用量

• 快照与引用：面向 AI 优化的工作流，使用紧凑快照 ID 和限定元素引用（@s8f3k2p9:e1、@s8f3k2p9:e2）

• 默认无界面交互：引用操作使用无障碍 API，并阻止静默的焦点、光标、键盘或剪贴板副作用

• 结构化 JSON 输出：提供机器可读的响应、错误代码和恢复提示

• 适用于任何应用：Finder、Safari、系统设置、Xcode、Slack——任何具备无障碍树的应用

• 通过 CDP 与 Chromium 应用互操作：launch --cdp 会开启经验证的 DevTools 端口，因此任何支持 CDP 的框架——Playwright、Puppeteer、chrome-remote-interface、agent-browser——都能驱动网页内容；原生菜单、对话框和窗口仍通过无障碍路径处理

  npm install -g agent-desktop # downloads prebuilt binary automatically

或者无需安装：

npx agent-desktop snapshot --app Finder -i



git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/

需要 Rust 1.89+ 和 macOS 13.0+。

macOS 需要授予无障碍权限。截图还需要屏幕录制权限，而通知中心打开器则需要针对 System Events 的自动化权限。普通权限检查不会触发授权提示。请通过受限的隔离辅助程序请求缺失权限：

agent-desktop permissions --request   # request missing permissions in an isolated helper

权限字段是显式对象，例如：

{
  "accessibility": { "state": "granted" },
  "screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
  "automation": { "state": "unknown" }
}

自动化权限会报告 granted、denied 或 unknown；unknown 表示 macOS 需要弹出提示，或者在不触发提示的情况下无法探测 System Events。

每个 GitHub Release 都会发布预构建的 C-ABI cdylib（libagent_desktop_ffi），支持 macOS、Linux 和 Windows，并与 CLI tarball 一同提供。加载它并调用 agent_desktop.h 中声明的函数，即可进行进程内调用，避免每条命令都 fork-exec 一次。

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_init(3)  # verify ABI major (AD_ABI_VERSION_MAJOR) before any call
adapter = lib.ad_adapter_create()
# observe -> act: ad_snapshot -> parse a qualified ref -> ad_execute_by_ref ...
lib.ad_adapter_destroy(adapter)

完整的使用方指南——入口点、所有权、线程、错误处理、构建/链接、发布归档和验证：skills/agent-desktop-ffi/。

对于密集型应用（Slack、VS Code、Notion），请使用渐进式骨架遍历以尽可能减少 token 使用量：

# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9

# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9

# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

对于简单应用，使用完整快照即可：

agent-desktop snapshot --app Finder -i   # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9  # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report"  # insert text into a field
agent-desktop press cmd+s               # keyboard shortcut
agent-desktop snapshot -i               # re-observe after UI changes



Agent loop:  snapshot → decide → act → snapshot → decide → act → ...

追踪查看器（回读一个会话）

session_id=$(agent-desktop session start --screenshots | jq -r '.data.session_id')
export AGENT_DESKTOP_SESSION="$session_id"
agent-desktop snapshot --app Finder -i       # work inside the explicit session scope
agent-desktop click @s8f3k2p9:e5
agent-desktop trace show --limit 500         # bounded JSON timeline for agents
agent-desktop trace export --out run.html    # single-file HTML viewer (works from file://)

trace show 会以确定性方式合并所有分段文件，且无需任何权限。trace export 会将时间线和截图以 base64 形式嵌入到单个静态 HTML 文件中。不使用 --out 时，HTML 将写入会话目录（~/.agent-desktop/sessions/<id>/trace-<id>.html），而非当前目录；--out 会覆盖该路径。当启用了 artifacts: full 时，应将导出的 HTML 视同截图处理。

用于多 Agent 工作流的共享会话

每次 Agent 运行时执行一次 session start，即可创建启用追踪的会话（manifest 默认设置为 trace: on）；随后通过全局 --session <id> 或 AGENT_DESKTOP_SESSION=<id> 传入返回的 ID。该显式范围内的命令会自动在 ~/.agent-desktop/sessions/<id>/trace/ 下生成 JSONL 分段，并共享会话的最新快照命名空间——无需为每次调用都添加 --trace。

对于并发运行的独立 Agent，请为每个进程设置 AGENT_DESKTOP_SESSION=<id>。当多个 Agent 共享同一个会话 ID 时，每个 Agent 都应基于自身 snapshot 调用获得的限定引用执行操作，而不应假定命名空间中的最新快照没有变化。

没有 manifest 的裸 --session <id>（未执行 session start）仍然只限定快照命名空间，不会写入追踪文件。快照 ID 只能在选定的会话命名空间内解析；它们绝不会触发跨会话搜索。

agent-desktop session start --name release-fix          # note data.session_id
export AGENT_DESKTOP_SESSION=<session_id>
agent-desktop snapshot --app Xcode -i --compact          # uses selected session + tracing
agent-desktop wait --element @s8f3k2p9:e9 --predicate actionable --timeout 5000
agent-desktop click @s8f3k2p9:e9
agent-desktop click @e9 --snapshot s2                    # legacy bare ref, explicitly pinned
agent-desktop session end "$AGENT_DESKTOP_SESSION"
agent-desktop session gc

驱动 Chromium 应用（CDP）

对于基于 Chromium 的应用（Slack、VS Code、Discord、Obsidian、Notion），launch --cdp 会在网页内容上开启一个经验证的 Chrome DevTools Protocol 端点。任何支持 CDP 的框架都可以连接——Playwright、Puppeteer、chrome-remote-interface、agent-browser——其中推荐使用 agent-browser，因为它提供基于引用的 Agent 工作流和内置的 electron skill。无论采用何种方式，原生界面元素（菜单、对话框、窗口、截图）仍走无障碍路径。

agent-desktop launch "Obsidian" --cdp



{ "app": "Obsidian", "pid": 4821, "cdp": {
  "port": 9229,
  "http_endpoint": "http://127.0.0.1:9229",
  "websocket_url": "ws://127.0.0.1:9229/devtools/browser/<id>",
  "product": "Chrome/142.0.7444.265"
},
  "suggestion": "Next: run `agent-browser connect 9229` (preferred) or connect any CDP client such as Playwright or Puppeteer. ..." }

--cdp 需要重新启动应用——目标应用已经在运行时将返回 ACTION_FAILED。请先运行 close-app，确认进程已退出，然后再次执行 launch --cdp。

该端点绑定至 127.0.0.1。在端点保持开放期间，任何以你的用户身份运行的本地进程仍可访问它；close-app 会在关闭应用的同时结束这一暴露。

agent-desktop snapshot --app Safari -i           # accessibility tree with refs
agent-desktop snapshot --surface menu            # capture open menu
agent-desktop screenshot --app Finder            # PNG screenshot
agent-desktop find --role button --app TextEdit  # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value  # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes          # list menus, sheets, popovers, alerts

get 和 is 只解析一次引用，在可用时优先使用实时平台读取；仅在适配器不支持实时读取时才回退。

agent-desktop click @s8f3k2p9:e3                  # strict headless AX click
agent-desktop --headed click @s8f3k2p9:e3         # physical click, focus/cursor allowed
agent-desktop --headed double-click @s8f3k2p9:e3  # physical double-click
agent-desktop --headed triple-click @s8f3k2p9:e3  # physical triple-click
agent-desktop right-click @s8f3k2p9:e3            # open context menu; inspect effect before retrying
agent-desktop type @s8f3k2p9:e5 "hello world"     # insert text into element
agent-desktop set-value @s8f3k2p9:e5 "new value"  # set value directly via AX
agent-desktop clear @s8f3k2p9:e5                  # clear element value
agent-desktop focus @s8f3k2p9:e5                  # set keyboard focus
agent-desktop select @s8f3k2p9:e9 "Option B"      # select verified dropdown/list option
agent-desktop toggle @s8f3k2p9:e12                # flip checkbox or switch
agent-desktop check @s8f3k2p9:e12                 # idempotent check
agent-desktop uncheck @s8f3k2p9:e12               # idempotent uncheck
agent-desktop expand @s8f3k2p9:e15                # expand disclosure/tree item
agent-desktop collapse @s8f3k2p9:e15              # collapse disclosure/tree item
agent-desktop scroll @s8f3k2p9:e1 --direction down --amount 3  # strict headless AX scroll
agent-desktop scroll-to @s8f3k2p9:e20             # scroll element into view

（macOS，第一阶段） 默认的引用操作是严格的无界面语义操作。在有界面模式下，核心会在派发前聚焦引用所指向的确切窗口；指针命令还要求存在经过验证的目标点，而适配器负责物理派发。click、right-click、type、clear 和 scroll 采用物理优先策略；双击/三击、悬停和拖拽仅支持物理操作；展开/折叠及其他语义

OBSERVE. DECIDE. ACT.

agent-desktop is a native desktop automation CLI designed for AI agents, built with Rust. It gives structured access to any application through OS accessibility trees — no screenshots, no pixel matching, no browser required.

• Native Rust CLI : Fast, single binary, no runtime dependencies

• C-ABI cdylib (libagent_desktop_ffi): Load once from Python / Swift / Go / Ruby / Node / C instead of forking the CLI per call

• 58 command names, 54 operational commands : Observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle, trace read/export, plus a bundled skills doc loader. The four held-input names are reserved for a stateful daemon and fail closed in the stateless CLI.

• Progressive skeleton traversal : 78–96% token reduction on dense apps via shallow overview + targeted drill-down

• Snapshot & refs: AI-optimized workflow using compact snapshot IDs and qualified element references (@s8f3k2p9:e1, @s8f3k2p9:e2)

• Headless-by-default interactions : Ref actions use accessibility APIs and block silent focus, cursor, keyboard, or pasteboard side effects

• Structured JSON output : Machine-readable responses with error codes and recovery hints

• Works with any app : Finder, Safari, System Settings, Xcode, Slack — anything with an accessibility tree

• Chromium-app interop via CDP : launch --cdp opens a verified DevTools port so any framework that speaks CDP — Playwright, Puppeteer, chrome-remote-interface, agent-browser — can drive the web contents, while native menus, dialogs, and windows stay on the accessibility path

  npm install -g agent-desktop # downloads prebuilt binary automatically

Or without installing:

npx agent-desktop snapshot --app Finder -i


git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release
cp target/release/agent-desktop /usr/local/bin/

Requires Rust 1.89+ and macOS 13.0+.

macOS requires Accessibility permission. Screenshots also require Screen Recording permission, and the Notification Center opener requires Automation permission for System Events. Plain permission checks never prompt. Request missing permissions in a bounded isolated helper with:

agent-desktop permissions --request   # request missing permissions in an isolated helper

Permission fields are explicit objects, for example:

{
  "accessibility": { "state": "granted" },
  "screen_recording": { "state": "denied", "suggestion": "Grant Screen Recording permission" },
  "automation": { "state": "unknown" }
}

Automation reports granted, denied, or unknown; unknown means macOS would need to prompt or System Events could not be probed without prompting.

Every GitHub Release ships a prebuilt C-ABI cdylib (libagent_desktop_ffi) for macOS, Linux, and Windows alongside the CLI tarballs. dlopen it and call the functions declared in agent_desktop.h for in-process calls instead of fork-exec per command.

import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_init(3)  # verify ABI major (AD_ABI_VERSION_MAJOR) before any call
adapter = lib.ad_adapter_create()
# observe -> act: ad_snapshot -> parse a qualified ref -> ad_execute_by_ref ...
lib.ad_adapter_destroy(adapter)

Full consumer guide — entrypoints, ownership, threading, error-handling, build/link, release archives, and verification: skills/agent-desktop-ffi/.

For dense apps (Slack, VS Code, Notion), use progressive skeleton traversal to minimize token usage:

# 1. Shallow overview — depth-3 map, truncated containers show children_count
agent-desktop snapshot --skeleton --app Slack -i --compact
# Keep snapshot_id, for example s8f3k2p9

# 2. Drill into a region of interest (named containers get refs as drill targets)
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

# 3. Act on an element found in the drill-down
agent-desktop click @e12 --snapshot s8f3k2p9

# 4. Re-drill the same region to verify the state change
agent-desktop snapshot --root @e3 --snapshot s8f3k2p9 -i --compact

For simple apps, a full snapshot is fine:

agent-desktop snapshot --app Finder -i   # get interactive elements with refs and snapshot_id
agent-desktop click @e3 --snapshot s8f3k2p9  # click a button by ref
agent-desktop type @e5 --snapshot s8f3k2p9 "quarterly report"  # insert text into a field
agent-desktop press cmd+s               # keyboard shortcut
agent-desktop snapshot -i               # re-observe after UI changes


Agent loop:  snapshot → decide → act → snapshot → decide → act → ...

Trace viewer (read back a session)

session_id=$(agent-desktop session start --screenshots | jq -r '.data.session_id')
export AGENT_DESKTOP_SESSION="$session_id"
agent-desktop snapshot --app Finder -i       # work inside the explicit session scope
agent-desktop click @s8f3k2p9:e5
agent-desktop trace show --limit 500         # bounded JSON timeline for agents
agent-desktop trace export --out run.html    # single-file HTML viewer (works from file://)

trace show merges all segment files deterministically and requires no permissions. trace export embeds the timeline plus screenshots as base64 in one static HTML file. Without --out, the HTML is written to the session directory (~/.agent-desktop/sessions/<id>/trace-<id>.html), not the current directory; --out overrides the path. Treat exported HTML like a screenshot when artifacts: full was enabled.

Shared sessions for multi-agent workflows

Run session start once per agent run to create a trace-enabled session (manifest trace: on by default), then pass the returned ID with global --session <id> or AGENT_DESKTOP_SESSION=<id>. Commands in that explicit scope get automatic JSONL segments under ~/.agent-desktop/sessions/<id>/trace/ and share the session's latest-snapshot namespace — no --trace on every call.

For concurrent independent agents, set AGENT_DESKTOP_SESSION=<id> per process. When multiple agents share one session ID, each agent should act on the qualified refs from its own snapshot call rather than assuming the namespace's latest snapshot is unchanged.

Bare --session <id> without a manifest (no session start) still scopes the snapshot namespace only and writes no trace files. Snapshot IDs resolve only inside the selected session namespace; they never trigger a cross-session search.

agent-desktop session start --name release-fix          # note data.session_id
export AGENT_DESKTOP_SESSION=<session_id>
agent-desktop snapshot --app Xcode -i --compact          # uses selected session + tracing
agent-desktop wait --element @s8f3k2p9:e9 --predicate actionable --timeout 5000
agent-desktop click @s8f3k2p9:e9
agent-desktop click @e9 --snapshot s2                    # legacy bare ref, explicitly pinned
agent-desktop session end "$AGENT_DESKTOP_SESSION"
agent-desktop session gc

Driving Chromium apps (CDP)

For a Chromium-based app (Slack, VS Code, Discord, Obsidian, Notion), launch --cdp opens a verified Chrome DevTools Protocol endpoint on the web contents. Any framework that speaks CDP can connect — Playwright, Puppeteer, chrome-remote-interface, agent-browser — with agent-browser preferred for its ref-based agent workflow and bundled electron skill. Native surfaces (menus, dialogs, windows, screenshots) stay on the accessibility path either way.

agent-desktop launch "Obsidian" --cdp


{ "app": "Obsidian", "pid": 4821, "cdp": {
  "port": 9229,
  "http_endpoint": "http://127.0.0.1:9229",
  "websocket_url": "ws://127.0.0.1:9229/devtools/browser/<id>",
  "product": "Chrome/142.0.7444.265"
},
  "suggestion": "Next: run `agent-browser connect 9229` (preferred) or connect any CDP client such as Playwright or Puppeteer. ..." }

--cdp needs a fresh launch — an already-running target returns ACTION_FAILED. Run close-app first, confirm the process exited, then launch --cdp again.

The endpoint is pinned to 127.0.0.1. Any local process running as your user can still reach it while it stays open; close-app ends the exposure along with the app.

agent-desktop snapshot --app Safari -i           # accessibility tree with refs
agent-desktop snapshot --surface menu            # capture open menu
agent-desktop screenshot --app Finder            # PNG screenshot
agent-desktop find --role button --app TextEdit  # search by role, name, value, text
agent-desktop get @e3 --snapshot s8f3k2p9 --property value  # read element property
agent-desktop is @e7 --snapshot s8f3k2p9 --property checked # check boolean state
agent-desktop list-surfaces --app Notes          # list menus, sheets, popovers, alerts

get and is resolve the ref once, prefer live platform reads when available, and fall back only when that live read is unsupported by the adapter.

agent-desktop click @s8f3k2p9:e3                  # strict headless AX click
agent-desktop --headed click @s8f3k2p9:e3         # physical click, focus/cursor allowed
agent-desktop --headed double-click @s8f3k2p9:e3  # physical double-click
agent-desktop --headed triple-click @s8f3k2p9:e3  # physical triple-click
agent-desktop right-click @s8f3k2p9:e3            # open context menu; inspect effect before retrying
agent-desktop type @s8f3k2p9:e5 "hello world"     # insert text into element
agent-desktop set-value @s8f3k2p9:e5 "new value"  # set value directly via AX
agent-desktop clear @s8f3k2p9:e5                  # clear element value
agent-desktop focus @s8f3k2p9:e5                  # set keyboard focus
agent-desktop select @s8f3k2p9:e9 "Option B"      # select verified dropdown/list option
agent-desktop toggle @s8f3k2p9:e12                # flip checkbox or switch
agent-desktop check @s8f3k2p9:e12                 # idempotent check
agent-desktop uncheck @s8f3k2p9:e12               # idempotent uncheck
agent-desktop expand @s8f3k2p9:e15                # expand disclosure/tree item
agent-desktop collapse @s8f3k2p9:e15              # collapse disclosure/tree item
agent-desktop scroll @s8f3k2p9:e1 --direction down --amount 3  # strict headless AX scroll
agent-desktop scroll-to @s8f3k2p9:e20             # scroll element into view

(macOS, Phase 1) Default ref actions are strict headless semantic operations. In headed mode, core focuses the exact ref window before dispatch; pointer commands additionally require a verified target point, while the adapter owns physical delivery. click, right-click, type, clear, and scroll are physical-first; double/triple-click, hover, and drag are physical-only; expand/collapse and other semantic commands remain semantic. Raw coordinates never imply a target window and therefore never steal focus. See skills/agent-desktop/references/commands-interaction.md.

agent-desktop press cmd+s               # key combo
agent-desktop press cmd+shift+z          # multi-modifier
agent-desktop press escape               # single key

key-down and key-up are reserved command names and return ACTION_NOT_SUPPORTED until a stateful daemon can own the held-key lifetime.

agent-desktop --headed hover @s8f3k2p9:e3                  # move cursor to element
agent-desktop --headed hover --xy 500,300         # move cursor to coordinates
agent-desktop --headed drag --from @s8f3k2p9:e3 --to @s8f3k2p9:e8   # drag between elements
agent-desktop --headed drag --from-xy 100,200 --to-xy 400,200  # drag between coordinates
agent-desktop --headed mouse-click --xy 500,300   # click at coordinates

mouse-down and mouse-up are likewise reserved; use the atomic mouse-click or drag commands.

agent-desktop launch Safari              # launch app by name
agent-desktop launch com.apple.Safari    # launch by bundle ID
agent-desktop launch "Obsidian" --cdp    # fresh launch + verified CDP port for web contents
agent-desktop close-app Safari           # quit app
agent-desktop close-app Safari --force   # force quit (SIGTERM, then SIGKILL if needed)
agent-desktop list-apps                  # list running GUI apps
agent-desktop list-windows               # list visible windows
agent-desktop list-windows --app Finder  # windows for specific app
agent-desktop focus-window --window-id w-4521  # bring exact window to front
agent-desktop resize-window --window-id w-4521 --width 800 --height 600
agent-desktop move-window --window-id w-4521 --x 100 --y 100
agent-desktop minimize --window-id w-4521
agent-desktop maximize --window-id w-4521
agent-desktop restore --window-id w-4521

Notifications (macOS only)

agent-desktop --headed list-notifications              # open Notification Center if needed, then list
agent-desktop --headed list-notifications --app "Slack"         # filter by app
agent-desktop --headed list-notifications --text "deploy" --limit 5  # filter by text
agent-desktop --headed dismiss-notification 1 --expected-app "Slack" --expected-title "Deploy complete"
agent-desktop --headed dismiss-all-notifications                # dismiss all
agent-desktop --headed dismiss-all-notifications --app "Slack"  # dismiss all from app
agent-desktop --headed notification-action 1 "Reply" --expected-app "Slack" --expected-title "Deploy complete"

Single-notification mutations require an app or title fingerprint from the same listing. Every mutation requires --headed because it opens and focuses Notification Center. Headless listing can only observe an already-open center; headed listing may open it and restore the prior frontmost app afterward.

agent-desktop clipboard-get              # read clipboard text
agent-desktop clipboard-set "copied"     # write to clipboard
agent-desktop clipboard-clear            # clear clipboard


agent-desktop wait 500                                       # sleep 500ms
agent-desktop wait --element @s8f3k2p9:e3 --timeout 5000              # wait for element
agent-desktop wait --element @s8f3k2p9:e3 --predicate actionable      # wait until safe to act
agent-desktop wait --element @s8f3k2p9:e5 --predicate value --value ready
agent-desktop wait --window "Save" --timeout 10000           # wait for window
agent-desktop wait --text "Loading complete" --app Safari    # wait for text
agent-desktop wait --text "Done" --count 1 --app Xcode       # wait for exact match count
agent-desktop wait --notification --text "Build Succeeded"   # wait for new matching notification
agent-desktop wait --menu --timeout 3000                     # wait for menu


agent-desktop batch '[
  {"command": "click", "args": {"ref_id": "@e2", "snapshot": "<snapshot_id>"}},
  {"command": "type", "args": {"ref_id": "@e5", "snapshot": "<snapshot_id>", "text": "hello"}},
  {"command": "press", "args": {"combo": "return"}}
]' --stop-on-error

agent-desktop --session run-a batch '[
  {"command": "snapshot", "args": {"app": "Finder", "interactive_only": true}},
  {"command": "status", "session": "run-b", "args": {}}
]'


agent-desktop session start [--name LABEL] [--no-trace]  # create session; pass returned ID explicitly
agent-desktop session end [id]
agent-desktop session list
agent-desktop session gc [--older-than SECS] [--ended]
agent-desktop status                     # platform, permissions, session_id, tracing, latest snapshot
agent-desktop permissions                # check accessibility/screen-recording/automation
agent-desktop permissions --request      # request in the bounded isolated helper
agent-desktop version                    # version string
agent-desktop skills get desktop --full  # bundled agent guidance


agent-desktop snapshot [OPTIONS]

Flag	Default	Description
--app <NAME>	focused app	Filter to a specific application
--window-id <ID>	-	Filter to a specific window
-i / --interactive-only	off	Only include interactive elements
--compact	off	Omit empty structural nodes
--include-bounds	off	Include pixel bounds (x, y, width, height)
--max-depth <N>	10	Maximum tree depth
--skeleton	off	Shallow 3-level overview; truncated containers show children_count and get refs as drill targets
--root <REF>	-	Start traversal from this ref; merges into existing refmap with scoped invalidation
--snapshot <snapshot_id>	latest	Snapshot ID to use when resolving --root
--surface <TYPE>	window	window, focused, menu, menubar, sheet, popover, alert

See the versioned JSON envelope, error-code, and exit-code contract.

snapshot assigns local positions in depth-first order and emits qualified refs such as @s8f3k2p9:e1, @s8f3k2p9:e2, and @s8f3k2p9:e3. A qualified ref embeds the exact snapshot ID and needs no separate --snapshot. Legacy bare refs such as @e3 remain accepted only with an explicit --snapshot s8f3k2p9. Snapshot lookup stays inside the selected session namespace.

Interactive roles that receive refs: button, textfield, checkbox, link, menuitem, tab, slider, combobox, treeitem, cell, radiobutton, incrementor, menubutton, switch, colorwell, dockitem.

Static elements (labels, groups, containers) appear in the tree for context but have no ref.

Reliability contract:

• session start creates and returns a manifest-gated session with automatic trace segments. It does not activate later processes. Activation resolves explicit --session first, then AGENT_DESKTOP_SESSION; otherwise the command uses the global, non-session namespace.
• Bare --session <id> without a manifest scopes snapshots only — no surprise trace files for existing callers.
• Snapshot lookup is confined to the selected namespace. A session-owned snapshot requires the same explicit --session or AGENT_DESKTOP_SESSION scope.
• Ref actions re-identify targets at action time: a moved unique target can proceed, while missing or changed stable identity returns STALE_REF.
• Mutable value text is not treated as stable identity, so text fields and timers can keep resolving when the saved window, path, role, and bounds evidence still identify the same element.
• Multiple plausible targets return AMBIGUOUS_TARGET instead of choosing arbitrarily.
• Actions run an actionability preflight before dispatch: visibility, stability, enabled state, supported action, policy, and editability.
• wait --element @s8f3k2p9:e3 --predicate actionable polls until the target can be acted on.
• With an active trace-enabled session, JSONL segments land under sessions/<id>/trace/<pid>-*.jsonl automatically. --trace <path> overrides to one file; --trace-strict fails on setup and pre-action writes (post-action traces are best-effort).

Stale ref recovery:

snapshot → act → STALE_REF or AMBIGUOUS_TARGET? → wait/snapshot again → retry with the new ref

| macOS | Windows | Linux
---|---|---|---
Accessibility tree | Yes | Planned | Planned
Click / type / keyboard | Yes | Planned | Planned
Mouse input | Yes | Planned | Planned
Screenshot | Yes | Planned | Planned
Clipboard | Yes | Planned | Planned
App & window management | Yes | Planned | Planned
Notifications | Yes | Planned | Planned

cargo build                               # debug build
cargo build --release                     # optimized (<15MB)
cargo test --lib --workspace              # run tests
cargo clippy --all-targets -- -D warnings # lint (must pass with zero warnings)

See the complete FAQ for architecture, platform support, installation, refs, licensing, and support links.

Apache-2.0