AI News
商业产业 Hacker News Agent 2026-09-26T15:56 正文 ok 1281 words

🔶 Drawgent：运行在实时 Excalidraw 画布上的编程 Agent

英文标题：🔶 Drawgent: Coding agent on a live Excalidraw canvas

Drawgent 可将用户本机已安装、已登录并已配置的 Claude Code、Codex 或 opencode 连接到 Excalidraw 白板，让 Agent 直接查看、编辑并验证画布内容。开发者既可在聊天面板提出绘图请求，也可在画布中写下 `AGENT:` 指令，或通过激光区域圈定需要修改的图表部分。项目支持启动新 Agent 会话、连接已有会话、协作房间、MCP 工具和 HTTP API，并将画布场景保存在工作区的 `.drawgent/scene.json` 中。其意义在于把代码 Agent 的交互从纯文本终端扩展到可实时协作的可视化白板，但渲染仍依赖 Chrome，部分 Codex 实时连接场景尚未完成已登录环境测试。
打开原文相关 6 · 机会 1
media134 B media47.4 KB

中文

drawgent：运行在实时 Excalidraw 画布上的编程 Agent#

drawgent 将你自己的 Claude Code、Codex 或 opencode（即你的安装、登录状态、配置和代码仓库）连接到 Excalidraw 白板。你可以在聊天面板中要求它绘制图表，或在希望修改的绘图部分旁写下 AGENT: …。Agent 会查看画布（截图 + 场景数据）、实时编辑、检查结果，并将该便签标记为 DONE。

快速开始#

前提条件：已安装并登录 claude、codex 或 opencode 之一。Claude 和 Codex 桥接还需要 Node.js ≥ 18（npm）。

drawgent setup claude        # 每个 Agent 仅需一次：claude | codex | opencode
cd ~/my-repo
drawgent up                  # 在此代码仓库中启动新的 Agent 会话，并在浏览器中打开画布
drawgent up --attach         # 或者：选择一个正在运行的会话，并将画布连接至该会话

drawgent setup <agent>#

该命令会一次性检查所有条件；如果缺少某项，会给出准确的修复方法，并写入 ~/.config/drawgent/config.toml：

  1. Agent CLI。 位于 PATH 中的 claude / codex / opencode。

  2. 登录状态。 claude auth status、codex login status 或 opencode auth list。

  3. ACP 桥接。

    • opencode 原生支持 ACP（opencode acp）。
    • Claude Code 和 Codex 使用官方 ACP 适配器。这些适配器会一次性安装到 ~/.cache/drawgent/adapters（约 60 MB），_不包含_它们捆绑的 Agent 二进制文件，并指向你的 CLI（CLAUDE_CODE_EXECUTABLE、CODEX_PATH）。
    • 随后，安装程序会验证 ACP 握手。

  4. 供已连接会话使用的画布工具。 只有 Codex 需要额外改动：codex mcp add drawgent -- drawgent mcp。Claude 和 opencode 会在连接时获得这些工具。

  5. 供渲染器使用的无头 Chrome。 如果系统中已有 Chrome/Chromium，安装程序会直接使用；否则会提供以下方案：

    • 将 Chrome Headless Shell（Chrome for Testing，约 120 MB，无需 sudo）下载至 ~/.cache/drawgent/chrome，并在缺少依赖时准确说明缺少哪些系统库；
    • 使用你的包管理器安装 Chromium（apt、snap、dnf、pacman、zypper、apk、brew 或无需 sudo 的 nix）。

非交互模式：--chrome download | system | /path/to/chrome。

drawgent up 会在对应 Agent 尚未成功完成 setup，或 setup 记录的某项依赖已消失时拒绝启动。

drawgent up#

在当前目录（即工作区）中运行：

  • 在 127.0.0.1:7300 启动编辑器和 API（若端口被占用则使用下一个可用端口）；
  • 通过 ACP 启动你的 Agent 的新会话，并在该工作区中工作；
  • 打开浏览器。在无头服务器上，它会改为输出 ssh -L 命令。

场景数据保存在 .drawgent/scene.json 中，并会自动加入 git 忽略规则。

drawgent up --attach [id]#

将画布连接到一个你已经运行的会话。不指定 id 时，它会列出发现的会话（优先显示当前目录中的会话），并让你选择：

agent 发现方式 画布如何连接至它
Claude Code claude agents --json（交互式和后台会话） fork：创建一个携带完整对话的新会话，由 drawgent 通过 ACP 驱动。你的终端会话不会受到影响
opencode 本地正在监听的 opencode server：使用 opencode --port 4096（或 opencode serve）启动 TUI live：消息会进入你正在运行的会话（可在 TUI 中看到）；drawgent 会在运行时将其 MCP 工具添加到该 server；回复、工具调用以及你在 TUI 中输入的提示词都会镜像到聊天面板
Codex ~/.codex/sessions 中的会话 live：codex queue --thread <id>；回复会从该会话的 rollout 文件镜像而来。会话需要 drawgent MCP（由 setup 注册，并在 Codex 启动时加载）

使用画布#

  • 聊天面板（右侧）：发送请求，查看回复和工具调用的流式输出，批准权限提示，或按下 Stop 取消当前轮次。

  • 在画布上：在形状旁边或内部写入以 AGENT: 开头的文本，或从该便签向某个形状画一条箭头。停止输入约 2.5 秒后，指令会携带其位置、指向对象及附近内容自动触发。Agent 会将其处理为绿色的 DONE: … 便签。把它改回 AGENT: 即可再次发送。

  • 激光区域：选择 Excalidraw 的激光工具（K），圈选或涂画图表的一部分。轨迹会以红色轮廓保留在画布上，聊天面板则会打开并准备接收你的指令，同时显示一个列出区域下方内容的标签（🔴 Laser zone · API, Redis ×）。你的下一条消息会连同该区域（边界及覆盖的元素）一起发送给 Agent；Agent 完成后轨迹会消失。间隔不超过 4 秒的笔划会构成同一个区域；点击 × 可丢弃该区域。

  • 提示词会排队并逐一执行。已被处理过的排队便签会被跳过。

    drawgent up --room 'https://excalidraw.com/#room=,'

  • drawgent 会以协作者身份加入房间（🤖 Agent，其光标会跟随编辑移动）。其他人可以继续使用 excalidraw.com：他们的 AGENT: 便签会发送给你的 Agent，而 Agent 的编辑会实时显示在该网站中。

  • 流量使用房间密钥进行端到端加密。空房间会从 excalidraw 的 Firestore 存储中加载并保存。

  • 本地编辑器会镜像该房间，并仍然保留聊天面板。

其他命令#

  • drawgent mcp：通过 stdio 提供画布工具的 MCP server。Agent 会启动它；它能自行找到正在运行的 drawgent up。
  • drawgent serve：带显式 Agent 的底层 server（可使用已完成 setup 的 --agent claude,codex,opencode，或通过 name=command 指定任意 ACP Agent）；适用于脚本和容器。
  • up / serve 的选项：--port、--room、--token（API bearer 及 URL 中的 ?token=）、--permissions canvas|ask|all（默认值为 canvas：绘图工具自动批准，其他任何操作都在聊天面板中请求确认）、--data、--settle-ms。

可选：Docker#

docker compose up --build 会运行一个画布 server（drawgent + Chromium，不包含 Agent），例如用于在服务器上托管共享画布或房间桥接。Agent 永远不会被打包进来：它们始终使用你自己的环境运行。

从源代码构建#

npm ci && npm run build      # 编辑器 + 渲染器页面 -> dist/（嵌入二进制文件）
cargo install --path .       # 或者：cargo build --release

get_scene、get_screenshot（视觉；可通过 element_ids 缩放）、add_elements（Excalidraw skeleton；箭头按 id 绑定，并进行边到边路由）、add_mermaid（自动布局）、update_elements（标签跟随形状，绑定箭头重新路由）、delete_elements、clear_canvas、list_instructions、resolve_instruction、set_status。

API#

GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= · POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear · GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status · POST /api/chat {agent?, text} · WS /ws（浏览器同步 + 聊天事件）

测试#

cargo test                          # 分数索引、房间加密/帧处理
node scripts/smoke.mjs [url]        # 针对运行中的 drawgent 测试聊天轮次 + AGENT: 便签
node scripts/e2e-browser.mjs [url]  # 真实浏览器：聊天面板 + 在画布中输入的便签
node scripts/room-e2e.mjs           # 新建 excalidraw.com 房间 ↔ drawgent，双向测试
node scripts/laser-e2e.mjs          # 激光区域 → 聊天 → Agent 仅编辑该区域（需要 setup claude）

布局#

src/（Rust）：

  • main.rs：CLI。
  • setup.rs、config.rs、chrome.rs：安装、配置、渲染器安装。
  • attach.rs：会话发现和选择器。
  • agents.rs + acp.rs：ACP 驱动程序（新建 / fork 会话）。
  • live.rs：opencode / Codex 实时驱动程序。
  • hub.rs：路由、便签、聊天日志。
  • scene.rs：存储和编辑操作。
  • renderer.rs：通过 CDP 控制 Chrome。
  • mcp.rs：MCP server。
  • room.rs：excalidraw.com 客户端。
  • fractional.rs、geometry.rs、el.rs：辅助工具。

web/：编辑器（main.jsx、chat.jsx、laser.js）和渲染器页面（render.jsx）。

限制#

  • 渲染器需要 Chrome（原生渲染器已在规划中）。
  • Claude 的“连接”模式实际是 fork，因为 Claude Code 没有公开方式可向正在运行的终端会话注入消息。
  • Codex 实时连接已实现，但尚未在已登录的 Codex 环境中测试。
  • 每个工作区仅支持一个场景。图片和文件不会同步。

English / 原文

drawgent: your coding agent on a live Excalidraw canvas#

drawgent connects your own Claude Code, Codex or opencode (your install, login, config and repo) to an Excalidraw whiteboard. Ask for a diagram in the chat panel, or write AGENT: … next to the part of a drawing you want changed. The agent looks at the canvas (screenshot + scene), edits it live, checks the result, and marks the note DONE.

Quick start#

Prerequisite: one of claude, codex or opencode installed and logged in. The Claude and Codex bridges also need Node.js ≥ 18 (npm).

drawgent setup claude        # once per agent: claude | codex | opencode
cd ~/my-repo
drawgent up                  # new agent session in this repo + canvas in your browser
drawgent up --attach         # or: pick one of your running sessions and connect the canvas to it

drawgent setup <agent>#

Checks everything once, fails with the exact fix when something is missing, and writes ~/.config/drawgent/config.toml:

  1. The agent CLI. Your claude / codex / opencode on PATH.

  2. Login. claude auth status, codex login status or opencode auth list.

  3. ACP bridge.

    • opencode speaks ACP itself (opencode acp).
    • Claude Code and Codex use the official ACP adapters. They are installed once into ~/.cache/drawgent/adapters (~60 MB), without their bundled agent binaries, and pointed at your CLI (CLAUDE_CODE_EXECUTABLE, CODEX_PATH).
    • Setup then verifies the ACP handshake.

  4. Canvas tools for attached sessions. Only Codex needs a change: codex mcp add drawgent -- drawgent mcp. Claude and opencode get the tools at attach time.

  5. Headless Chrome for the renderer. Setup uses your Chrome/Chromium if you have one. Otherwise it proposes:

    • downloading Chrome Headless Shell (Chrome for Testing, ~120 MB, no sudo) into ~/.cache/drawgent/chrome, and telling you exactly which system libraries are missing, if any;
    • installing Chromium with your package manager (apt, snap, dnf, pacman, zypper, apk, brew, or nix without sudo).

Non-interactive: --chrome download | system | /path/to/chrome.

drawgent up refuses to start until setup succeeded for that agent, or if something setup recorded disappeared.

drawgent up#

Runs in the current directory (the workspace):

  • starts the editor + API on 127.0.0.1:7300 (next free port if taken);
  • starts a new session of your agent over ACP, working in the workspace;
  • opens your browser. On a headless box it prints the ssh -L command instead.

The scene is kept in .drawgent/scene.json, which is git-ignored automatically.

drawgent up --attach [id]#

Connects the canvas to a session you already run. Without an id it lists the sessions it finds (current directory first) and lets you pick one:

agent discovery how the canvas reaches it
Claude Code claude agents --json (interactive and background sessions) fork : a new session carrying the full conversation, driven by drawgent over ACP. Your terminal session is left untouched
opencode opencode servers listening locally: start the TUI with opencode --port 4096 (or opencode serve) live : messages go into your running session (you see them in your TUI); drawgent adds its MCP tools to that server at runtime; replies, tool calls and prompts you type in the TUI are mirrored into the chat panel
Codex sessions in ~/.codex/sessions live : codex queue --thread <id>; replies are mirrored from the session's rollout file. The session needs the drawgent MCP (registered by setup, loaded when Codex starts)

Using the canvas#

  • Chat panel (right side): send a request, watch replies and tool calls stream in, approve permission prompts, press Stop to cancel a turn.

  • On the canvas : write a text starting with AGENT: next to or inside a shape, or draw an arrow from the note to a shape. It fires about 2.5 s after you stop typing, with its position, what it points at and what is nearby. The agent resolves it into a green DONE: … note. Edit it back to AGENT: to send it again.

  • Laser zones : pick Excalidraw's laser (K) and circle or scribble over part of the diagram. The trace stays on the canvas as a red outline, and the chat panel opens ready for your instruction, with a chip listing what is under the zone (🔴 Laser zone · API, Redis ×). Your next message goes to the agent together with the zone (bounds and covered elements), and the trace disappears when the agent is done. Strokes within 4 s of each other form one zone; × discards it.

  • Prompts are queued and run one at a time. A queued note that was already handled is skipped.

    drawgent up --room 'https://excalidraw.com/#room=,'

  • drawgent joins the room as a collaborator (🤖 Agent, with a cursor that follows its edits). Humans can stay on excalidraw.com: their AGENT: notes reach your agent, and its edits appear there live.

  • Traffic is end-to-end encrypted with the room key. Empty rooms are loaded from and saved to excalidraw's Firestore storage.

  • The local editor mirrors the room and still has the chat panel.

Other commands#

  • drawgent mcp: stdio MCP server with the canvas tools. Agents launch it; it finds the running drawgent up by itself.
  • drawgent serve: low-level server with explicit agents (--agent claude,codex,opencode as set up, or name=command for any ACP agent); for scripts and containers.
  • Options for up / serve: --port, --room, --token (API bearer + ?token= in the URL), --permissions canvas|ask|all (default canvas: drawing tools auto-approved, anything else asked in the chat panel), --data, --settle-ms.

Optional: Docker#

docker compose up --build runs a canvas server (drawgent + Chromium, no agents), e.g. to host a shared canvas or a room bridge on a server. Agents are never bundled: they always run with your own setup.

Build from source#

npm ci && npm run build      # editor + renderer pages -> dist/ (embedded into the binary)
cargo install --path .       # or: cargo build --release

get_scene, get_screenshot (vision; zoom with element_ids), add_elements (Excalidraw skeletons; arrows bind by id and are routed edge-to-edge), add_mermaid (auto-layout), update_elements (labels follow shapes, bound arrows re-route), delete_elements, clear_canvas, list_instructions, resolve_instruction, set_status.

API#

GET /api/health · GET /api/scene · GET /api/screenshot?ids=&padding=&max= · POST|PATCH|DELETE /api/elements · POST /api/mermaid · POST /api/clear · GET /api/instructions · POST /api/instructions/{id}/resolve · POST /api/status · POST /api/chat {agent?, text} · WS /ws (browser sync + chat events)

Tests#

cargo test                          # fractional indices, room crypto/framing
node scripts/smoke.mjs [url]        # chat turn + AGENT: note against a running drawgent
node scripts/e2e-browser.mjs [url]  # real browser: chat panel + note typed on the canvas
node scripts/room-e2e.mjs           # fresh excalidraw.com room ↔ drawgent, both directions
node scripts/laser-e2e.mjs          # laser zone → chat → agent edits only that zone (needs setup claude)

Layout#

src/ (Rust):

  • main.rs: CLI.
  • setup.rs, config.rs, chrome.rs: setup, config, renderer install.
  • attach.rs: session discovery and picker.
  • agents.rs + acp.rs: ACP driver (new / fork sessions).
  • live.rs: live opencode / Codex drivers.
  • hub.rs: routing, notes, chat log.
  • scene.rs: store and edit operations.
  • renderer.rs: Chrome over CDP.
  • mcp.rs: MCP server.
  • room.rs: excalidraw.com client.
  • fractional.rs, geometry.rs, el.rs: helpers.

web/: editor (main.jsx, chat.jsx, laser.js) and renderer page (render.jsx).

Limits#

  • The renderer needs Chrome (a native renderer is planned).
  • Claude "attach" is a fork, because Claude Code has no public way to inject into a running terminal session.
  • Codex live attach is implemented, but was not yet tested against a logged-in Codex.
  • One scene per workspace. Images/files are not synced.