消费级 Agent 测试工具（CATT）

消费级 Agent 测试工具（Consumer Agent Testing Tool，简称 CATT）用于测试智能助手如何使用你的服务。支持针对 MCP 服务器、MCP 应用、网站或 HTTP API，运行基于脚本或模型驱动的 Agent。该工具可检查服务执行的操作、审查执行凭据，并在不同构建版本间对比结果。

Muse 和 Instinct 是受支持的代理仿真（surrogate emulation）目标。 其中 Muse 仍为默认目标。两者均采用共享连接器测试契约，包括路由、内存、支付、浏览器/HTTP 任务及调度。有关其数据来源与限制，请参见 Instinct 研究文档。驱动测试试验的模型与被仿真的智能助手是相互独立的。

贡献指南 | 安全政策 | 更新日志 | MIT 许可证

测试宿主说明。 CATT 是一个代理系统：它模拟的是智能助手的宿主环境，而非其私有实现。测试通过并不代表已获得与 Muse 或任何托管智能助手的兼容性认证。每个针对 Muse 的特定行为都有记录在案的依据分类：documented（有文档记录）、unofficial（非官方）、assumed（假设）或 unknown（未知）。本项目与 Meta、Instinct、OpenAI 或 Anthropic 无任何关联，亦未获得其背书。

测试轨道	命令	检查内容	状态与范围
MCP 连接器	connector-run SCENARIO	使用服务器的工具，应用账户和审批规则，保护私有变量，并检查服务记录的结果与 Agent 的回答。	Muse 和 Instinct 代理目标：`--emulation-target muse
连接器扩展	带 extensions: 的 connector-run SCENARIO	路由、内存、自定义 API 客户端、浏览器操作、钱包卡片、后台任务、不可信内容、通道限制和模拟用户。	随该轨道支持。user_simulator 属于探索性功能，无法在脚本化测试或决策契约中运行。参见扩展文档。
场景脚手架	scaffold URL --output FILE, scaffold --from-run RUN --output FILE	读取工具目录或将 Agent 运行转换为脚本化调用。不调用任何服务工具。运行前请替换所有 TODO(scaffold)。	已支持。支持单个 Streamable HTTP 连接器；不支持 OpenAPI 输入。
Muse 连接器就绪度	check URL --suite muse-guidelines	在不调用工具的情况下，根据锁定的连接器规范审查目录和可选清单。	属于已文档化的约束，并非准入凭证。参见就绪度检查。
共享代理就绪度	check URL --suite surrogate --emulation-target instinct	在不调用工具的情况下，检查目录、白名单、Schema 语法以及选定的投影。	针对任一目标的 CATT 契约；参见共享契约。
个人 Agent 协议	check ORIGIN --suite poppy	读取 poppy.json。配合清单，探测会话、登录以及在公司宿主上声明的一项操作。	草案 0.1 版本；尚无生产环境 Muse 凭据。
Muse 小部件注册	check FILE --suite gadget	根据锁定的 SDK 限制检查 link.register 消息和命令规范。不建立网络连接。	属于已文档化的限制，并非准入凭证。SDK 条款声明其非开发者平台。
MCP 协议契约	check TARGET	检查 Schema、版本协商、会话和操作契约；运行锁定的合规测试套件。	已支持。通用 MCP 凭据；参见就绪度检查。
嵌入式 MCP 应用	surface-run SCENARIO	在 Chromium 中渲染应用，并检查表单、工具调用、资源和服务响应。Agent 驱动程序从场景的脚本化步骤中进行选择。	已支持。通用 MCP Apps 宿主，非 Muse。参见宿主范围说明。
网站	browse POLICY, run --surface browser	根据请求规则对无障碍快照执行操作。人工接管会暂停运行（退出码 3）；CLI 不提供回传接管控制。	原生 Chromium，无操作系统级隔离。浏览器场景仅使用打包的预订门户。
HTTP API	run --surface api-tools, inspect, check	将审批绑定到每个提议的调用中，并检查后端结果。	支持映射配置。你的 API 还需要环回部署和专用控制适配器。参见 API 场景和 API 试验。
凭据审查	report, export, replay, diff	审查报告、脱敏导出项、验证回放并对比不同运行版本。	已支持。回放和对比功能需要封存的运行记录。
开发扩展	--development-extensions COMMAND	CLI 二进制文件、隔离浏览器、云端 Worker、本地 Glimmer、生成的 OpenAPI 和 Prism Mock。	不在正式支持的发布版本范围内。参见开发扩展说明。

连接器轨道记录了每个 Muse 特定行为的依据。工具类别和审批提示遵循已记录的规范指南。muse-cli 投影和模板客户端遵循非官方镜像；原生投影被视为通用的基线假设。中立的 cli 投影作为 CATT 的处理模式同时支持两个目标。Instinct 运行拥有独立的来源记录，并将共享宿主算法标记为假设。原生/CLI MCP 是 CATT 的测试适配器；目前尚未发现公开的 Instinct MCP 集成契约。

在无需模型和账户的情况下，针对 Instinct 运行本地预订示例：

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json \
  --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"

在场景中设置 emulation_target: instinct 或传入 --emulation-target instinct 参数。两个目标均使用 native 或 cli 投影以及通用 MCP 客户端。Instinct 默认采用纯文本回复，且不执行 Muse 规范检查。其本地钱包固件覆盖了支付行为；而真实支付通道、可信网络协议、Vault 以及 Files 仍处于共享契约之外。

接下来包含三个快速入门指南。连接器测试轨道和常用 MCP 服务器快速入门均在本地回环上运行，无需账户；Coverage Cat 沙箱快速入门则调用托管沙箱。“测试你的服务”章节将它们整合为工作流，文档索引中列出了完整页面。

系统要求 Python 3.12 或更高版本（CI 在 Linux 上测试 3.12，在 macOS 上测试 3.14）、uv，以及 Node.js 20 或更高版本及 npm（安装的包声明要求 >=20；已在 25.9 测试）。记录的发布凭据基于 macOS。从克隆的代码仓库中，安装 Python 和 Node 依赖项以及浏览器轨道所需的 Chromium 构建：

uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium

连接器轨道、就绪度检查和脚本化运行仅需 uv sync。Node 和 Chromium 仅用于 MCP Apps 和浏览器测试轨道。

在 Linux（Debian 或 Ubuntu）上，请改用系统库安装 Chromium。Linux 环境未包含在记录的发布凭据中：以下命令与 CI 工作流一致，但尚未在此处得到合格验证。

sudo apt-get update && sudo apt-get install -y nodejs npm   # 或从 nodesource 或 nvm 安装 Node 20+
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
  node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium

有三项功能依赖于 macOS Seatbelt（sandbox-exec），在 Linux 上会报告不支持或受阻，而非无限制运行：--offline 就绪度模式、run --surface api-tools 底层的网络隔离验证 Worker，以及本地模型内存配置。连接器轨道、脚本化运行、就绪度检查以及基于 Docker 的 stdio 目标不依赖该机制。

使用 doctor 检查环境配置：

uv run consumer-agent-testing-tool doctor

它会根据安装包声明的最低要求检查 Node，比对 tooling/payment-ui 依赖是否符合固定版本，并检查 .cache/ms-playwright 下的 Chromium 及其 Headless Shell。它还会报告可选的协议工具链，且仅在开发轨道或配置了 stdio MCP 与 CLI 目标时检查 Docker。即使现有配置或 --model-profile 选中了模型，该命令也不会发起任何模型请求。每个失败或受阻的检查项都会在 stderr 上打印一行原因，若属于配置问题，还会输出修复该问题的命令。

如需显式运行提供方预检（会发送两次简短的模型请求并可能产生费用），可添加 --model-preflight：

uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight

脚本化运行无需模型账户，且所有三个快速入门中的脚本化命令均无需配置文件。Agent 运行则需要模型配置文件和 API Key；“模型提供商”章节介绍了如何创建。请通过环境变量提供凭据，避免将其写入场景文件或共享凭据中。默认设置文件 consumer-agent-testing-tool.yaml 存储在本地且已被 git 忽略，不会随项目分发任何 profile。

快速入门：连接器测试轨道

本快速入门完全在本地回环网络运行，无需任何外部账户。完成环境配置后，在第二个终端窗口启动类似于 Cedar Stays 的测试服务。准备就绪后它会输出一行包含其 URL 的 JSON；按下 Ctrl-C 可停止运行：

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none

在不调用模型的情况下回放示例的脚本化调用：

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json

该脚本依次执行搜索房间、获取报价、下单预订并给出最终回答。宿主会从场景的 private: 变量中填入客人的电子邮件（模型永远无法看到该信息），脚本化用户则根据场景的 permissions: 规则响应每张审批卡片。运行完成后，宿主会检查固件的私有预言机（Oracle，确认是否有且仅有一条预订记录，且无重复扣费），并通过 get_booking 检查预订状态。同时还会验证每次发起的调用是否与其审批卡片相匹配（approval.dispatches_match, approval.disclosure_bound），以及最终回答中没有虚假陈述并完整转述了总价与确认信息（answer.no_false_claims, answer.relay_complete）。

该命令将输出运行目录与 Connector report: .../report.html，并在所有必需检查均通过时返回退出码 0。各次运行记录将完整封存在 runs/connector-<id>/ 目录下。如需对外分享，请使用 export 命令的导出结果，切勿直接分享运行目录，因为其 private/ 文件夹中保存有宿主私有变量。

如需执行基于 Agent 的运行，可通过“模型提供商”中的 init 命令创建 agent.yaml，在终端 shell 中配置 OPENROUTER_API_KEY，然后运行：

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

系统会首先执行模型预检，并打印其自身的凭据路径。接着，Agent 将看到四个被声明为原生函数的允许工具（宿主私有的电子邮件已从其 Schema 中移除），并自行填写其他调用参数。此过程会产生模型推理费用。若要在不修改场景文件的前提下更改运行策略：使用 --projection cli 可通过单个 exec 命令向任意目标提供相同的工具；--exposure search-only 可通过 tool_search 动态加载工具参数；--output-schema strict 会像官方 MCP SDK 一样拒绝违反工具 outputSchema 的结果；--trials N 可运行 N 次独立试验并输出 pass^k 成功率指标。运行锁定记录会在 overrides 项下保存 projection、exposure、client-profile 和 output-schema 的覆盖项。

connector-run 的退出状态码规范如下：当所有必要检查均通过时退出码为 0；当检查未通过、回答包含虚假陈述或达到 Agent 上限时退出码为 1；当出现环境配置、测试框架套件或报告生成错误时退出码为 2；当测试试验受阻或不受支持（例如存在待核对的不确定写操作、不可观测的检查项或客户端 profile 限制）时退出码为 3。若要基于你自己的服务器开始接入，可运行 scaffold；若要无模型重放已记录且可共享的 Agent 试验，可使用 scaffold --from-run 将其转换为调用文件。连接器测试轨道参考文档详细介绍了场景、命令参数及审查凭据。示例目录列出了每个内置场景及其对应的固件启动命令，详细信息可参考[测试轨道扩展](/Coverage-Cat/consumer-agent-testing-tool/blob/main/docs/connector

Consumer Agent Testing Tool (CATT)

Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.

Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.

Contributing | Security | Changelog | MIT license

A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented, unofficial, assumed or unknown. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.

Lane	Command	What it checks	Status and scope
MCP connectors	connector-run SCENARIO	Use the server's tools, apply account and approval rules, protect private values, and check the service's recorded outcome and the agent's answer.	Muse and Instinct surrogates: `--emulation-target muse
Connector extensions	connector-run SCENARIO with extensions:	Routes, memory, custom API clients, browser actions, wallet cards, background jobs, untrusted content, channel limits and simulated users.	Supported with the lane. user_simulator is exploratory and cannot run in scripted tests or decision contracts. See extensions.
Scenario scaffolding	scaffold URL --output FILE, scaffold --from-run RUN --output FILE	Read a tool catalog or turn an agent run into scripted calls. Calls no service tools. Replace every TODO(scaffold) before running.	Supported. One Streamable HTTP connector; no OpenAPI input.
Muse connector readiness	check URL --suite muse-guidelines	Review the catalog and optional manifest against the pinned connector guidelines without calling tools.	Documented constraints, not admission. See readiness checks.
Shared surrogate readiness	check URL --suite surrogate --emulation-target instinct	Check the catalog, allowlist, schema syntax and selected projection without calling tools.	CATT's contract for either target; see shared contract.
Personal Agent Protocol	check ORIGIN --suite poppy	Read poppy.json. With a manifest, probe sessions, sign-in and one declared operation on the company's hosts.	Draft 0.1; no production Muse evidence.
Muse gadget registration	check FILE --suite gadget	Check a link.register message and command specs against the pinned SDK limits. Opens no connection.	Documented limits, not admission. The SDK's terms say it is not a developer platform.
MCP protocol contracts	check TARGET	Check schemas, version negotiation, sessions and operation contracts; run the pinned conformance suite.	Supported. Generic MCP evidence; see readiness checks.
Embedded MCP apps	surface-run SCENARIO	Render the app in Chromium and check forms, tool calls, resources and service responses. An agent driver chooses from the scenario's scripted steps.	Supported. Generic MCP Apps host, not Muse. See host scope.
Websites	browse POLICY, run --surface browser	Act on accessibility snapshots under request rules. Takeover pauses the run (exit 3); the CLI has no handback.	Native Chromium, without OS isolation. Browser scenarios use only the packaged reservation portal.
HTTP APIs	run --surface api-tools, inspect, check	Bind approval to each proposed call and check backend outcomes.	Supported with a mapping. Your API also needs a loopback deployment and private control adapter. See API scenarios and API trials.
Evidence	report, export, replay, diff	Inspect reports, redact exports, verify playback and compare runs.	Supported. Playback and diff require sealed runs.
Development extensions	--development-extensions COMMAND	CLI binaries, isolated browsers, cloud workers, local Glimmer, generated OpenAPI and Prism mocks.	Outside the supported release. See development extensions.

The connector lane records the basis for each Muse-specific behavior. Tool classes and approval prompts follow documented guidelines. The muse-cli projection and template client follow unofficial mirrors; the native projection is an assumed generic baseline. The neutral cli projection supports both targets as a CATT treatment. Instinct runs have a separate source record and label the shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter; no public Instinct MCP integration contract was found.

Run the local booking example under Instinct without a model or account:

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json \
  --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"

Set emulation_target: instinct in your scenario or pass --emulation-target instinct. Both targets use native or cli projection and the generic MCP client. Instinct defaults to plain-text replies and does not run Muse guideline checks. Its local wallet fixture covers payment behavior; live payment rails, trusted-network protocol, Vault and Files remain outside the shared contract.

Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and the documentation index lists every page.

You need Python 3.12 or newer (CI tests 3.12 on Linux and 3.14 on macOS), uv, and Node.js 20 or newer with npm (the installed packages declare >=20; tested with 25.9). The recorded release evidence covers macOS. From a clone of this repository, install the Python and Node dependencies and the Chromium build the browser lanes use:

uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js install chromium

The connector lane, the readiness checks and the scripted runs need only uv sync. Node and Chromium are for the MCP Apps and browser lanes.

On Linux (Debian or Ubuntu), install Chromium with its system libraries instead. Linux is not covered by the recorded release evidence: the commands below match the CI workflow but have not been qualified here.

sudo apt-get update && sudo apt-get install -y nodejs npm   # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" \
  node tooling/payment-ui/node_modules/playwright/cli.js install --with-deps chromium

Three features rely on macOS Seatbelt (sandbox-exec) and report unsupported or blocked on Linux rather than running unconfined: the --offline readiness mode, the network-denied validator workers behind run --surface api-tools, and the local-model memory profile. The connector lane, the scripted runs, the readiness checks and the Docker-backed stdio targets do not depend on it.

Check the setup with doctor:

uv run consumer-agent-testing-tool doctor

It checks Node against the floors the installed packages declare, the tooling/payment-ui packages against their pinned versions, and Chromium and its headless shell under .cache/ms-playwright. It reports the optional protocol tooling, and checks Docker only for development lanes or configured stdio MCP and CLI targets. It makes no model requests, even when an existing configuration or --model-profile selects a model. Each failed or blocked check prints one line on stderr with its reason and, for a setup problem, the command that fixes it.

To explicitly run the provider preflight, which sends two short model requests and may incur charges, add --model-preflight:

uv run consumer-agent-testing-tool doctor --config agent.yaml --model-preflight

Scripted runs need no model account, and the scripted commands in all three quickstarts need no configuration file. Agent runs need a model profile and an API key; Model providers shows how to create one. Supply credentials through environment variables, and keep them out of scenarios and shared evidence. consumer-agent-testing-tool.yaml, the default settings file, is local and git-ignored, and no profile is shipped.

Quickstart: connector lane

This quickstart stays on loopback and needs no account. Complete Setup, then start the Cedar Stays-like test service in a second terminal. It prints one JSON line with its URLs when it is ready; Ctrl-C stops it:

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none

Replay the example's scripted calls, with no model:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml \
  --driver scripted --script examples/connector/fixture-booking.calls.json

The script searches rooms, gets a quote, books and gives a final answer. The host fills the guest's email from the scenario's private: values, which the model never sees, and the scripted user answers each approval card from the scenario's permissions: rules. After the run, the host checks the fixture's private oracle (one confirmed booking, no duplicate charges) and the booking's status through get_booking. It also checks that each sent call matched its approval card (approval.dispatches_match, approval.disclosure_bound) and that the final answer made no false claim and relayed the total and the confirmation (answer.no_false_claims, answer.relay_complete).

The command prints the run directory and Connector report: .../report.html, and exits 0 when every required check passed. Runs are sealed under runs/connector-<id>/. Share the output of export, not the run directory: its private/ folder holds host-private values.

For an agent run, create agent.yaml with the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

The model preflight runs first and prints its own evidence path. The agent then sees the four allowed tools as native functions, with the host-private email removed from their schemas, and writes the other arguments itself. This uses paid model inference. To change the treatment without editing the scenario, --projection cli offers the same tools through one exec command for either target, --exposure search-only loads their parameters through tool_search, --output-schema strict refuses results that break a tool's outputSchema as the official MCP SDK does, and --trials N runs independent trials and prints pass^k. The run lock records the projection, exposure, client-profile and output-schema overrides under overrides.

connector-run exits 0 when every required check passed; 1 when a check failed, the answer made a false claim or an agent limit was reached; 2 for a setup, harness or reporting error; and 3 when the trial is blocked or unsupported, for example by an uncertain write to reconcile, an unobservable check or a client-profile limit. To start from your own server, run scaffold; to replay a recorded, shareable agent trial without a model, turn it into a calls file with scaffold --from-run. The connector lane reference covers scenarios, flags and evidence. The examples list each shipped scenario with its fixture command, and lane extensions covers routes, custom connectors, the browser sub-agent, wallets, schedules and the other extensions.

Quickstart: a popular MCP server

The connector lane works with any MCP server over Streamable HTTP, not only the bundled test service. This quickstart points it at "Everything" (@modelcontextprotocol/server-everything, MIT), the reference server the MCP project maintains for client authors. Its weather tool returns simulated data, so it needs no account or key. Start it in a second terminal; the first run downloads the pinned release from npm:

PORT=3001 npx -y @modelcontextprotocol/server-everything@1.0.0 streamableHttp

See what the host infers from the live catalog. scaffold runs the handshake and tools/list, calls no tool, and writes a scenario skeleton with suggested tool classes from the server's annotations:

uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool check http://127.0.0.1:3001/mcp --suite muse-guidelines

The second command reviews the catalog statically against the pinned Muse connector guidelines. Then replay the shipped example, which asks for the weather in Chicago, first with native tools and then through the Muse-style command-line projection over the unofficial Muse template client:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --projection muse-cli --client-profile muse-template-unofficial-2026-10 \
  --driver scripted --script examples/connector/everything-weather-cli.calls.json

Both exit 0. The scenario offers the model three of the server's tools; a reviewed allowlist keeps the rest off its list, including get-env, which returns the server process's environment variables, and gzip-file-as-resource, which fetches any URL it is given. The server has no private oracle, so the outcome check calls get-structured-content itself after the agent stops, and the answer checks look for the temperature and conditions the server returned. With a model profile, let an agent do the same task three times and print pass^k (paid inference). The scenario declares no reset URL, so connector-run warns that the trials share the server's state; this weather tool keeps none:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml \
  --config agent.yaml --trials 3

check also runs readiness suites that read catalogs and documents instead of running an agent. With the connector-lane quickstart's fixture still running, review its catalog and the example manifest against the pinned Muse connector guidelines:

uv run consumer-agent-testing-tool check http://127.0.0.1:8765/mcp --suite muse-guidelines \
  --manifest examples/connector/readiness.yaml

Suite	Command	Exit codes
muse-guidelines	check URL --suite muse-guidelines [--manifest FILE]	0 no fail finding, 1 fail findings, 2 setup error, 3 the client profile cannot reach the server
muse-guidelines drift	check --suite muse-guidelines --drift-check	0 unchanged, 1 drifted, 2 not fetched
poppy	check ORIGIN --suite poppy [--manifest FILE]	0 no fail finding, 1 fail findings, 2 setup error, 3 blocked
gadget	check FILE --suite gadget	0 no fail finding, 1 fail findings, 2 the document cannot be read
contract (default)	check TARGET [--interface-profile PROFILE]	0 every required check passed, 1 a check failed, 2 setup, harness or reporting error, 3 blocked or unsupported

The guidelines review lists the server's tools and calls none; --drift-check fetches the live guidelines, and only when given. Poppy discovery only reads. Its probes run only with --manifest and act on the company's own hosts: they start sessions, sign in the declared test account and confirm one declared operation. The gadget suite reads one local file. The contract suite checks a target named in the configuration file (--config). For a kind: mcp target without declared contract_cases, its operation-contracts check is unsupported and the run exits 3.

A clean readiness result is not an admission. Meta reviews connectors by hand, Poppy is draft 0.1 with no production Muse evidence, and the Gadget SDK's terms say it is not a developer platform. Readiness checks describes the rules, manifests, --offline behavior and the mcp-contract-2026-07-28 profile.

Quickstart: Coverage Cat sandbox

From a source checkout or unpacked source distribution, complete Setup, then run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted

The example creates a fictional shopper's draft in Coverage Cat's hosted sandbox at https://www.coveragecat.com/api/consumer/mcp. It needs no Coverage Cat account, operator credentials or model API key.

The first umbrella_consumer_prefill call sets sandbox: true, as specified in the consumer MCP manifest. That argument activates the server's sandbox. The scenario's test_only: true flag merely records the test author's intent.

The host fixes the tool arguments and keeps the returned access token private. It permits two tools, umbrella_consumer_prefill and consumer_intake_issues, and checks that:

1. Prefill returns sandbox: true.
2. Follow-up retains sandbox: true.
3. Follow-up returns the original application ID.
4. The server reports application_submitted: false.

Each run creates a new draft and stops before submission. A failed or missing sandbox check stops the workflow. These checks cover draft creation and follow-up, not payment widgets, browser behavior or independent backend checks.

To let a model choose the steps, create agent.yaml with the init command in Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml \
  --driver agent --config agent.yaml

The agent driver is a scripted action menu : Spark Open picks one ready scenario step per turn, and the host runs it with the scenario's arguments. The model never sees the server's tools, schemas or raw results, only the result fields the scenario exposes, and the evidence is labeled scripted action menu. The host supplies the test inputs and checks the results. This version uses paid model inference through Together, with ZDR enabled and fallback disabled by default.

Verified 15 October 2026: both scripted and Spark Open agent runs passed all four sandbox checks. Both model responses identified Together; both requests used the configured ZDR policy. See the recorded quickstart results.

Each command prints an outcome line, such as Surface run completed (exit 0), then Surface report: .../report.html. Open the report to inspect the workflow and its checks. The same directory contains result.json for the structured result and events.jsonl for the redacted event timeline. Agent runs also store model and tool records under agent/.

Replace the placeholders below with the run directories printed by your commands:

uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID

Share the generated export. Keep the full run folder private. While a run can resume, checkpoint.private.json holds the credentials, tool results and host inputs it needs; a completed run, or one that failed with exit 1 and left no write unconfirmed, keeps only a scrubbed record.

To test your own MCP service, copy the example and replace its endpoint, allowed origins, tool policies, test data and response checks. Surface workflows explains UI steps, the agent driver and resuming runs.

Extend through submission and sandbox checkout

The checkout example adds review, submission, Monoline offer selection, consent, bind follow-up, document upload and simulated payment-method setup. Its 14 steps contain 62 assertions. All seven tools use the consumer umbrella MCP, and every service step must return the same sandbox intake.

Current integration status, 15 October 2026: blocked at submission. Coverage Cat returns quote fields that its MCP output schema excludes. It also returns minimum_auto_limits as a string where the schema requires an object. The host rejects the response. The draft quickstart passes, but the full checkout test has not passed end to end. See the failure report and separate diagnostics.

Run this command to reproduce the failure or retest after the service contract is corrected:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
  --driver scripted \
  --replies examples/coveragecat-sandbox-checkout-replies.json

The reply files supply a fictional shopper's answers : approval to submit, credit consent, the Monoline $2M offer at $389, sufficient auto limits, declined excess UM/UIM, and “No” to both carrier questions. These fixtures do not authorize transactions for a real shopper. A changed quote or missing or negative approval stops the next write. The two generated declarations PDFs are labelled SYNTHETIC TEST ONLY.

Once submission works, the example pauses with exit code 3 at sandbox_document_review_poll. Only at that checkpoint , wait at least the reported 60 seconds and resume the printed run:

sleep 60
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml \
  --driver scripted --resume runs/surface-YOUR_RUN_ID \
  --replies examples/coveragecat-sandbox-checkout-after-review.json

The host pauses for your reply but does not measure how long you wait. The after-review fixture approves the simulated payment callback.

For an agent-driven run, replace --driver scripted with --driver agent --config agent.yaml on both commands. A resumed run must keep its original scenario, driver and model configuration.

The sandbox supplies mock payment credentials. The example sends the returned seti_sandbox_... with the fixture pm_sandbox_consumer_agent_testing_tool, then requires payment_method_on_file, waiting_on_carrier, and no charge confirmation. Billing checks inspect setup metadata rather than rendered fields. No Stripe form opens, no card details are collected, and no fallback URL is used. This test establishes neither premium collection nor active insurance. Test the embedded payment UI and Stripe test-mode setup separately.

The quickstarts above are the pieces of one workflow. For an MCP server you own:

1. Write a scenario. scaffold URL --output my-connector.yaml reads the live catalog (--auth-env VAR for an API key, or --auth-from FILE with a scenario that holds an OAuth block) and writes a skeleton. Replace every TODO(scaffold): the user's goal in their own words, the allowlist of tools a reviewed connector would expose, tool classes your documentation settles, approval rules for the scripted user, host-private values such as the user's email, and checks that read the real outcome, from a private state URL (an oracle) or one of your read tools.
2. Check readiness. check URL --suite muse-guidelines --manifest FILE reviews the catalog and your declarations against the pinned connector guidelines. A clean result is a documented constraint, not an admission.
3. Pin the happy path. Replay a scripted calls file with --driver scripted; it needs no model and belongs in CI. After an agent trial you trust, scaffold --from-run RUN --output FILE turns it into one.
4. Run agent trials. --trials N with a model profile gives pass^k. Run the same scenario under each treatment a Muse-like host may use: --projection muse-cli, --exposure search-only, --client-profile muse-template-unofficial-2026-10 and --output-schema strict. The run lock records each one.
5. Add realism. Lane extensions add competing connectors and connect-on-request, payment with single-use cards, background jobs, untrusted content, channel limits and a simulated user.
6. Compare builds. diff compares two sealed runs and exits 1 on an outcome regression. Share the output of export; the run directory's private/ folder holds host-private values.

For an MCP app with its own UI, start from the Coverage Cat scenario and surface-run. For a website, use browse with a browser policy, or the connector lane's browser extension. For an HTTP API, write a mapping for run --surface api-tools, or test whether an agent can build its own client from your docs with the custom_connector extension. Run a consumer test covers the other lanes.

Create a profile with init. This one uses Muse Spark Open (meta/muse-spark-open) through OpenRouter, routed only to Together, with zero data retention (ZDR) required and fallback disabled; set OPENROUTER_API_KEY in your shell before an agent run:

uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open \
  --routing-provider together --config agent.yaml

init also stages, beside the configuration, the API scenarios, the browser scenarios and the mapped-API example. It stages only scenarios the supported CLI can run, and refuses providers that have only a configuration template (openai, anthropic, gemini, azure-openai, vertex-gemini, openai-compatible) unless --development-extensions comes before init. init --provider none --config scripted.yaml writes a configuration for scripted commands that need one. init refuses to overwrite an existing file.

Profiles that init writes are labeled specified/unverified. doctor --model-preflight, run and the agent drivers of connector-run, surface-run and browse print a warning for any profile not labeled supported, and the run lock records the label. Model IDs that contain contributor select a mode in which the provider may train on your prompts; such a profile needs allow_training_on_prompts: true (init --allow-training-on-prompts). See model providers for every adapter, its checks and its failure labels.

OpenRouter model and privacy settings

Consumer Agent Testing Tool requests zero data retention (ZDR) by default , even when routing.zdr is omitted. To change this, edit the selected profile in consumer-agent-testing-tool.yaml or the file passed to --config:

model_profiles:
  openrouter-dev:
    provider: openrouter
    api: chat-completions
    model_id: meta/muse-spark-open
    auth:
      mode: api_key
      env: OPENROUTER_API_KEY
    routing:
      only: [together]
      allow_fallbacks: false
      require_parameters: true
      data_collection: deny
      zdr: true  # Set to false to disable this tool's per-request ZDR requirement.
defaults:
  model_profile: openrouter-dev

Use unquoted YAML booleans: true or false. Consumer Agent Testing Tool applies the setting to every OpenRouter request, including requests that continue after a tool result, and records it in the routing evidence.

Setting zdr: false removes this request-level requirement. It leaves data_collection: deny, the pinned provider and the ban on fallbacks in place. Your OpenRouter account or guardrails may still enforce ZDR. Request logs show the policy sent; they cannot independently verify the provider's retention practices. See OpenRouter's ZDR documentation.

Checked 15 October 2026: OpenRouter listed Together for Spark Open in both its model and ZDR endpoint catalogs. A live tool-call test passed with the routing settings above. Both responses identified Together, and the model used the tool result correctly. See the test record.

To choose another provider, set routing.only or pass --routing-provider to init. Requests never switch providers or models silently.

Runs use Inspect's dedicated openrouter provider, which replays earlier reasoning in structured form rather than as text. Each response must name the pinned provider. Responses do not name endpoint variants, so a pin such as nebius/fp4 is recorded as variant unverified. Streaming is not supported. See model providers.

With zdr: true, only ZDR-listed endpoints are eligible. On 15 October 2026, Muse Spark 1.3 and Muse Wonder were served only by Meta on OpenRouter and were not ZDR-listed, so no ZDR-compliant OpenRouter route existed for them. A request for them is expected to fail, but no live run has recorded the error. When OpenRouter's error cites the data policy, it is labeled no endpoint satisfies the routing/ZDR policy. With routing.only: [together], the init default, it may instead be labeled routing policy: no allowed provider.

Sources: Meta announcement, OpenRouter model, provider endpoints, and ZDR endpoints.

The standalone smoke test and older consumer helper read the same --config and --model-profile, including privacy settings. Profiles that init writes leave reasoning effort unset.

.venv/bin/python scripts/openrouter_smoke.py --config agent.yaml \
  --model-profile openrouter-dev

This command tests the model connection with synthetic tool data. It needs an API key and an available route to the pinned provider. It makes no requests to the service you plan to test.

init --provider meta writes a profile for Meta's Model API:

uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml

The profile uses the Responses API at https://api.meta.ai/v1 and reads the key from MODEL_API_KEY or META_API_KEY; if both are set, they must agree. Add --api chat-completions for Chat Completions, or --key-env NAME to use another variable, which is then the only one read. The adapter uses the OpenAI SDK from the openrouter extra.

The host refuses requests that Meta rejects, such as a forced tool choice, before sending them (exit 3). Meta adds steering context to every prompt on its servers, so captured requests are incomplete, and every Meta run records this as an observation gap. Streaming is not supported, and the API is pay-as-you-go. See Meta Model API. No live Meta Model API run has been recorded; the adapter is tested against local fixtures.

This section covers the embedded MCP apps (surface-run), browser (browse) and HTTP API lanes. To let an agent use your MCP server's own tools, use the connector lane.

A scenario defines the endpoint, allowed origins, tools, private arguments, UI actions, consent requirements and result checks, and optionally an API key for the MCP connection (auth). Use endpoints and synthetic transactions you have permission to test.

consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver scripted
consumer-agent-testing-tool surface-run path/to/scenario.yaml --driver agent --config agent.yaml
consumer-agent-testing-tool surface-run path/to/scenario.yaml --resume runs/existing-run
consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json

The host declares MCP Apps support as a client, finds each tool's UI through its tools/list definition and renders it in a native browser. With --driver agent, the model picks among the scenario's steps (a scripted action menu) and never sees the server's tools, raw results or private inputs. Service checks determine the result; a rendered form or a model's claim of success cannot prove that a transaction completed.

browse drives a restricted browser through accessibility snapshots and element references, with no selectors, scripts or CDP. Its policy file sets the start url, the permitted origins and request rules, snapshot_max_chars (default 20,000), history_snapshots (2) and dialogs (agent, accept or dismiss), as in examples/browser-policy.json. Long snapshots are paged with {"kind": "snapshot", "offset": N}. Refused or failed actions come back to the agent as observations with a status, and dialogs and file choosers are answered with kind: dialog or kind: upload. With --driver agent, the provider preflight runs before the browser opens. The result's observations block counts truncated snapshots and refused actions. When the agent requests a takeover, the run pauses (exit 3). The CLI has no handback, so control does not return to the agent in that run. In native mode, WebSockets opened by dedicated workers and WebRTC TCP relays are not enforced. The policy field webmcp: true, off by default, lists page-registered WebMCP tools in the snapshot and lets the agent call them with kind: webmcp; that lane is labeled "generic WebMCP page tools, not Muse" (basis: assumed), and its evidence notes that Meta documents WebMCP for its AI glasses, a Meta product that is not Muse. See restricted browser behavior.

HTTP API tests use explicit mappings with the scenario runner. Mappings can send declared request headers and put the target credential in a custom API-key header. In API and browser trials, invalid tool proposals are returned to the model as tool errors, and exhausted limits fail the trial; see API scenarios. To test whether an agent can build its own client from an API's documentation instead, use the connector lane's custom_connector extension. Prism, Schemathesis and generated-OpenAPI tools are available only as development extensions. Their five packages with unresolved original license notices are excluded from the supported release, not treated as cleared.

surface-run follows the exit codes in spec §18:

Exit	Meaning
0	Completed
1	Failed: an assertion, a service rejection, a UI interaction failure or an agent error
2	Setup, infrastructure, integration or reporting failure
3	Blocked or unsupported: waiting for a reply or approval, an unconfirmed write, a limit, or missing coverage

Before Surface report: ... it prints one line: Surface run <status> (exit N) at <stage> [<class>]: <reason>. result.json records the same failure_class and failure_stage, and whether the run is resumable. A completed run, or one that failed with exit 1, is final unless the failure left a write unconfirmed: that write may have taken effect, so the run stays resumable to reconcile it. Blocked runs and setup failures can resume.

run and browse use the same four codes, except that exhausting a turn, tool-call or time limit is a failure (exit 1). Only Inspect operator and custom limits, and in run a deadline that passed before an attempt's first generation, block instead (exit 3). run prints each evidence path on stdout. browse prints the final observation and its evidence path when the run completes or pauses for takeover, and nothing on stdout otherwise. For any outcome other than completed, both also print one line on stderr: <label> <status> (<class>) at <check>: <reason>. Evidence: <path>. The label names the step, for example Model preflight, Trial 2/10 of stale_total, Resumed trial of stale_total or Browse (doctor uses Doctor check). A run with a failed or blocked trial, or with --trials above 1, also prints a summary line with the counts. With --trials above 1 it adds the suite pass^k, the mean over scenarios, for example Run completed: 0 failed, 0 blocked and 4 completed of 4 trials; pass^1 1.00, pass^2 1.00. Evidence: .... The suite's metrics also break the trials down by scenario (statistical decisions). run DIR refuses (exit 2), before the preflight, a directory whose scenario files share an id.

connector-run prints the same stderr line, labeled for example Connector trial 1/1 of fixture-booking-happy-path; its codes are listed in the connector-lane quickstart. With --trials N, one code stands for all trials: 2 when any trial's evidence is not shareable, else 1 when any trial failed, else 2 when any hit a setup, harness or verification error, else 3 when any was blocked, else 0. The Connector trials: line on stderr gives pass^k and names the trials that are not shareable or ended in an error. The readiness suites' codes are in Readiness checks.

What the MCP Apps host implements

The embedded-app lane is generic MCP Apps host evidence (MCP Apps 2026-01-26), for hosts such as ChatGPT or Claude. It is not Muse evidence: Muse documents no third-party UI. Each result.json says so under mcp_apps_host.

The host implements:

• the client capability (io.modelcontextprotocol/ui with text/html;profile=mcp-app), UI discovery from the tool's tools/list definition, and HTML resources as text or base64 blobs;
• tool-input with the call's arguments, tool-result, and a hostContext (theme, display mode, locale, time zone, platform, container width and maximum height, and tool information);
• a content security policy built from the resource's _meta.ui.csp, or the specification's restrictive default when the resource declares none, and the declared camera, microphone, geolocation and clipboard-write permissions;
• in the native renderer, a sandbox proxy on an origin separate from the host page. With browser_origins, the host page also has Chromium's local-network-access permission, every redirect hop is checked again (GET or HEAD to an allowlisted origin only), and each direct write is recorded as an unconsented host_action before it is sent;
• view tools/call requests for the step's allowlisted tools, and resources/read and resources/list for the step's callback_resources;
• ui/message, delivered once to agent runs in the next observation a model reads, as app_messages; the view is told when the host does not deliver it. ui/update-model-context, ui/open-link, display-mode, log and teardown requests are recorded, and the latest model-context update reaches agent runs as app_context;
• size changes, ui/resource-teardown before the view closes, and tool-cancelled when the view never received the call's result;
• _meta.ui.visibility, for view calls and for tool steps;
• MCP 2026-07-28 servers, when the scenario sets version_negotiation: auto. The default is the 2025 initialize handshake;
• an API key on the MCP connection (auth), as a bearer token, a custom header or a query parameter;
• a legacy window.openai bridge.

It does not implement a model turn for a view's message, resource templates, _meta.ui.domain, display modes other than inline, partial tool input, host-context change notifications, OAuth on the MCP connection, or consent for a view's direct writes in the native renderer. The legacy bridge and the isolated renderer have no sandbox proxy. See MCP Apps host.

Inspect retained evidence

consumer-agent-testing-tool report runs/existing-run
consumer-agent-testing-tool export runs/existing-run
consumer-agent-testing-tool replay runs/existing-run
consumer-agent-testing-tool diff runs/baseline runs/candidate

Replay checks the evidence seals and stops on a mismatch. It never substitutes a live request. To test the service again, start a new run. Reports mark missing measurements as unknown and keep partial totals distinct from complete totals.

Reports open with the verdict, which names the failing step and check, then the checks and host capabilities, and a per-step timeline that shows what the model received in full. Then come a connectors table for runs with several connectors, the surface views (what the service published, what the model was offered, what it received at each step, and what only the developer sees), the metrics, and the viewers and linked evidence: agent and preflight runs, their Inspect logs and the inspect view command, and, locally, a captured Playwright trace. Gaps and limits follow, then the run lock and the full recorded result, each collapsed; see report layout. report on an agent sub-run of a sealed run returns its sealed report unchanged.

Evidence files and exports hide credentials, registered secrets, contact details and other personal fields such as names and addresses. Model views keep the user's own data visible: they hide only credentials and declared secrets, plus browser password fields and, in a resumed browser scenario, text typed in an earlier attempt. The provider boundary refuses any model request that still contains a registered credential. Export also applies the run's own private values, which are never exported, and refuses (exit 2) when a staged file, or a member of an Inspect log or other archive, still holds one. See redaction.

diff compares two sealed runs, such as surface, browser or connector runs. A change to the package source counts as a run-lock change, so compare runs made from the same checkout. It exits 0 when they are comparable with no outcome regression, and 1 when an outcome regressed: a worse status, or a passing check that no longer passes or is missing. It exits 3 when they are not comparable, because of a run-lock change outside --treatment or an unsealed or unverifiable run. Generated IDs and service timestamps are normalized; run locks are compared exactly. --ignore POINTER (repeatable) excludes a JSON pointer under /outcomes, /capabilities or /milestones, and --treatment /server declares an intentional server build change. See replay and diff.

For more detail, see the documentation index, connector lane, lane extensions, readiness checks, consumer tests, browser diagnostics, measurement coverage, model timing, and release extensions, which includes development history.

Scripted connector runs need no model and no key, and they replay the same calls every time, so they work as a pull-request check. connector-run can start the service itself and write reports a CI system reads:

uv run consumer-agent-testing-tool connector-run my-connector.yaml \
  --driver scripted --script my-connector.calls.json \
  --start "python -m my_service --port {port}" \
  --junit reports/connector.xml --summary-json reports/connector.json

• --start COMMAND runs the service on a free loopback port. The command is split like a shell command but not run in a shell.
  • {port} in the command is that port. The service also gets it as PORT and the scenario as ${PORT}.
  • The tool waits until --ready-url, or the scenario's first connector endpoint, answers HTTP. It waits at most --ready-timeout seconds (default 30).
  • When the run ends, the service's process group gets SIGTERM, then SIGKILL after 5 seconds.
  • Its output goes to runs/service-<id>.log, outside the sealed run. That log isn't redacted, so pass secrets to the service through the environment. The run lock records the command with credentials redacted.
  • A service that exits early or never becomes ready exits 2.
• --junit FILE writes one test suite per trial and one test case per check. Blocked checks are <skipped>. check --suite muse-guidelines accepts it too.
• --summary-json FILE writes the verdict, exit code, checks, trials, pass^k and run paths. Its format is schemas/summary-v1.schema.json. --summary-json - prints only that JSON on stdout and moves the other lines to stderr.
• Both reports are also written when the command stops with an error. See CI outputs and services.

The exit codes in Outcomes and exit codes decide the step. A 3 (blocked or unsupported) is not a pass. examples/ci/connector-lane.yml is a GitHub Actions template with two jobs. On pull requests, it runs the scripted calls and the readiness review. Nightly or on demand, it runs agent trials under each Muse-like treatment, and only when a model key secret is set. .github/workflows/ci.yml runs this repository's tests and shipped examples on macOS and Linux. Coding agents should start with AGENTS.md. All matrix jobs feed the required CI check; Linux failures are not allowed to pass silently. Workflow validation, tracked-file hygiene, secret scanning and schema freshness are checked too.

Plain pytest runs the supported collection:

tests/conftest.py gives each session a temporary directory, home and cache under .cache/pytest/, removes them after a passing session, and removes the surface and browser run directories its tests create under runs/. The full collection takes about twenty minutes and makes no network request beyond loopback and no model request. pytest --development-extensions also collects the experimental tests, which may need other runtimes, tools and external evidence. Async cleanup and unhandled thread exceptions fail tests instead of remaining warnings. See CONTRIBUTING.md for the review workflow and the launch checklist for the private-to-public transition.

The supported release is local-consumer-v1 : the connector lane and its extensions, the readiness checks, consumer MCP apps, the restricted browser, mapped HTTP APIs, transactional consent, continuation, evidence and replay, measurements, the distribution, and the OpenRouter and Meta Model API adapters. Real-Muse calibration, cloud provisioning, local Glimmer, browser OS isolation and live Bedrock qualification are outside it; see release scope for the full boundary.

Release evidence is generated per release and is not committed. docs/release-gates.json is the committed policy: the gates, the scope and the exclusions. The release workflow runs on a v* tag. It records every gate on macOS, requires the audit to pass, and attaches three things to the GitHub release:

• the wheel and the source archive, with signed build provenance;
• release-evidence-<commit>.tar.gz, which holds release/release-gates.json (the recorded ledger) and release/evidence/.

Use consumer-agent-testing-tool release-audit to check release readiness. It reads release/release-gates.json when evidence has been recorded or extracted there. Without it, it reads the committed policy and reports every gate unverified. Exit code 0 means release-ready forlocal-consumer-v1; exit code 3 means at least one required gate needs attention. The evidence binds a digest of the source, so a source change or an expired gate (seven days after recording) cannot inherit an earlier passing result. To audit a published release, check out its tag, extract its evidence archive at the repository root and run the audit before the gates expire.

Day to day, you don't need any of this. Run the tests for what you change, and let CI run the full collection on each pull request.

A separate live Spark Open test passed through Together with ZDR enabled. It verified native tool calls, continuation after a tool result, and the requested routing policy. It did not test streaming, independently verify retention practices, or establish compatibility with every consumer workflow. The Coverage Cat checkout failure described above remains an open integration issue.

The audit requires five gates:

Gate	Evidence required
Regression	Every test in the default supported collection passes, with no skips
Distribution	Fresh offline installation, original dependency notices, artifact integrity and packaged consumer workflows
Transactional consent	Passing consent, continuation and effect checks in the current regression
Replay/diff	Passing evidence-integrity and comparison checks in the current regression
Surrogate parity	Exact Muse/Instinct native/CLI cases, memory lifecycle, routing, payment, readiness and extension coverage

To record evidence locally instead, for example before tagging, run the same steps as the workflow, in this order. verify_distribution.py regenerates the dependency notices, which are part of the source fingerprint, so the distribution comes before the regression. The regression takes about twenty minutes. Everything is written under release/, which git ignores:

.venv/bin/python scripts/verify_distribution.py
.venv/bin/python scripts/record_distribution.py
.venv/bin/python scripts/run_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_regression.py release/evidence/regression-run.xml
.venv/bin/python scripts/record_release_tests.py release/evidence/regression-run.xml
consumer-agent-testing-tool release-audit

run_regression.py refuses an existing report path, so remove release/ or pick a new name to record again. It records workspace-relative paths and writes the home directory as ~ in the log.

Distribution checks use caches in the workspace and the installed browser. They need no Docker VM, cloud worker or model credentials. If a dependency cache is missing, populate it separately with scripts/verify_distribution.py --prepare-cache; that setup step uses the network. The installation evidence covers macOS only.

The audit verifies source fingerprints, artifact hashes, required checks and expiry dates. It returns 0 when every required gate passes, or 3 otherwise. Excluded capabilities remain labelled as excluded. Qualification inputs explains what each gate needs.

The repository retains worker, isolation, local-model, calibration and generated-OpenAPI experiments, and the CLI/skill lane for a developer's own CLI binary (run --surface cli-skill), which needs Docker and an oracle that returns the reservation fixture's snapshot schema. The normal CLI rejects these paths. To use them, put --development-extensions before the subcommand and install the separate development-extensions dependency group and any required tooling. These experiments are outside the supported release. The supported way to give an agent a CLI over an MCP server is connector-run --projection muse-cli.

The optional Node API tools have known dependency advisories. Do not use them with untrusted inputs while those findings remain open.

The original specification and earlier test records document the project's history; they do not define the scope of this release.

MIT; see LICENSE. The license does not cover the Coverage Cat and CATT names or the logo; see TRADEMARKS.md. Third-party components keep their own licenses; the distribution ships their notices under third_party/.