Show HN：Agentsnap——面向 AI Agent 的快照测试工具

面向 AI Agent 的确定性快照测试。

完整文档，包括指南和 API 参考，见 iamfaham.github.io/AgentSnap。

agentsnap 会在一次**黄金运行（golden run）**期间记录 Agent 的 LLM 调用和工具调用，并生成一个需要提交到版本控制的快照文件。此后的每次运行中，它都会重放相同输入，并从四个维度将新调用轨迹与快照进行比较：

维度	检查内容	方法
结构	工具调用的名称和顺序	对工具序列计算 Levenshtein 编辑距离
参数	工具调用参数	使用 deepdiff（如已安装）或普通字典差异对比，支持配置忽略字段
模型工具	_模型自身_请求了哪些工具（而不只是代码实际执行了什么）	对 tool_requests 计算 Levenshtein 编辑距离，并逐项比较请求参数差异
语义	LLM 响应和最终输出	通过 all-MiniLM-L6-v2 计算余弦相似度，或使用 LLM judge 获得更高准确度

若任一维度的偏差超过阈值，agentsnap 将抛出 AgentRegressionError，并附带结构化差异报告。

---

Agent 的行为会悄无声息地退化。一次提示词微调、一次模型替换、一个工具接入了错误参数——不会抛出异常，不会让 CI 失败，而你往往会在 Agent 悄悄开始给出更差答案时，才在生产环境中发现问题。

agentsnap 提供两种模式，分别适用于不同任务：

• Replay，每个 PR 都运行——使用录制的响应进行重放，而非调用真实 API。确定性、零成本，可捕获代码回归问题，例如提示词编辑、工具接线损坏及调用次数变化。
• Live，每晚运行——使用当前模型发起真实 API 调用，捕获只有模型本身发生变化时才会出现的漂移。

通过 replay 模式捕获一次提示词修改，无需 API 调用：

Agent regression in 'replay'
============================

[ARGS] llm_call[0].messages:
  messages: [{'content': 'Answer concisely: What is Python?', ...}] ->
            [{'role': 'user', 'content': 'You are a pirate. Answer: ...'}]

[SEMANTIC] llm_call[0]: 100% PASS
[SEMANTIC] output: 100% PASS

Failed checks: ['llm_requests']

---

系统会要求你选择语义比较后端：

选项	所需条件	最适用场景
[1] LLM judge（默认）	API 密钥（OpenRouter、OpenAI、Anthropic 或自定义服务）	事实型 Agent，准确度最高
[2] 离线嵌入模型	无需任何凭据——下载约 22 MB 模型，可在任意环境运行	任意机器，无需 API 密钥
[3] 本地 LLM judge	即将推出	性能较强的本地机器，无需云服务

向导会将你的选择保存至 pyproject.toml，并将 API 密钥（如有）保存至 .env。密钥绝不会写入 pyproject.toml。它还会将 __agent_snapshots__/.last_run/ 加入 .gitignore（如有必要会创建该文件），并提供在 tests/test_agentsnap_example.py 中生成示例快照测试的选项。

agentsnap check   # 可随时验证你的配置

3 — 录制你的 Agent（无需修改代码）

PatchSet 会在类级别修补所有已安装的 LLM SDK——无论原始客户端在何处创建，都会被自动捕获。无需封装客户端：

from agentsnap import PatchSet, AgentRecorder
import anthropic

# 你现有的 Agent——保持不变
def my_agent(question):
    client = anthropic.Anthropic()   # 原始客户端，无需包装器
    return client.messages.create(...).content[0].text

# 首次运行：录制黄金快照
with PatchSet():
    with AgentRecorder("my_agent") as rec:
        result = my_agent("What is Python?")
        rec.output = result
# 写入 __agent_snapshots__/my_agent.json——请提交该文件

4 — 在后续运行中断言

from agentsnap import PatchSet, AgentAsserter

with PatchSet():
    with AgentAsserter("my_agent") as a:
        result = my_agent("What is Python?")
        a.output = result
# 如果行为发生漂移，则抛出 AgentRegressionError

5 — 使用 pytest fixture（最简单）

snapshot.run() 会在首次调用时自动录制，并在后续每次运行时自动断言——无需切换。加入 agentsnap_instrument，即可自动启用 PatchSet：

def test_my_agent(snapshot, agentsnap_instrument):
    with snapshot.run("my_agent") as s:
        result = my_agent("What is Python?")   # 原始客户端——自动捕获
        s.output = result


pytest
# 或者为一个会话中的每个测试启用 PatchSet：
pytest --agentsnap-instrument

---

每次断言可在以下两种模式之一运行：

模式	LLM 调用	可捕获的问题	最适用场景
live（默认）	真实 API	模型／行为漂移	每晚运行、发布前测试
replay	无——重放录制的响应	代码回归：提示词编辑、工具接线、循环逻辑	每个 PR / CI

在 replay 模式中，每次 LLM 调用的录制响应都会重新提供给 Agent——无需 API 密钥、没有成本、不会出现不稳定测试。比较会转向请求侧：如果代码发送了不同的提示词、发起了不同数量的 LLM 调用，或者改变了工具调用序列，agentsnap 就会让测试失败。

# 单个测试
with AgentAsserter("my_agent", mode="replay") as a: ...

# 整个测试套件
pytest --agentsnap-replay        # 强制使用 replay
pytest --agentsnap-live          # 强制使用 live


[tool.agentsnap]
mode = "replay"   # 将 replay 设为项目默认模式

在 replay 模式下，工具调用仍会真实执行。传入 replay_tools=True，也可从录制内容中对工具调用进行 stub 处理（完全没有副作用）。

注意事项：

• Replay 需要使用 agentsnap >= 0.2.0 录制的快照（其中包含 raw_response）。旧版快照会抛出 SnapshotFormatError——请通过 pytest --agentsnap-record 重新录制。
• Replay 目前支持 Anthropic、OpenAI、Groq 和 OpenRouter。其他提供商会抛出 ReplayError——请对这些测试使用 live 模式。
• 使用场景（scenario）时，必须在 replay 模式中显式传入 scenario=（由于测试主体运行前便会读取快照，因此无法使用输入自动哈希）。
• 如果重放后的最终输出与黄金输出并非字节级完全一致，为其评分就需要语义后端——请安装并配置嵌入模型后端（pip install agentsnap[offline]，然后在 agentsnap init 中选择选项 2），或配置 judge（AGENTSNAP_JUDGE_API_KEY）。
• 异步客户端（AsyncAnthropic、AsyncOpenAI）同样会被拦截——replay 的无网络保证同时覆盖同步和异步客户端，包括异步流。唯一尚未覆盖的缺口是流式 OpenAI Responses API（responses.create(stream=True)），该调用会直接透传且不被录制。参见 examples/async_agents.py。

AnthropicAdapter 和 OpenAIAdapter 会对 stream=True 调用进行 tee 处理（Groq 和 OpenRouter 通过继承 OpenAIAdapter 获得这一能力）：数据块会原样转发给 Agent，同时组合后的响应会被录制以供 replay，格式为 raw_response={"__stream__": True, "chunks": [...]}。

在 replay 模式中，录制的数据块会被重建为真实 SDK 的 chunk/event 对象，并以增量形式重新 yield——Agent 的消费方式与实时流完全一致，且不会产生 API 调用。一次流式调用的录制内容不能作为非流式请求重放（反之亦然）——这种情况会抛出 ReplayError，并给出明确的“shape mismatch”错误信息。

尚不支持：client.messages.stream() 上下文管理器辅助方法，以及流式 OpenAI Responses API 调用。Mistral 仍会对每个调用强制使用 stream=False。完整的可运行演示参见 examples/streaming.py，异步客户端版本参见 examples/async_agents.py。

如果一个流从未被迭代、也从未关闭，它会在 recorder/asserter 退出时自动完成收尾；但仍建议及时消费或关闭流，以便事件按调用顺序出现。

---

除了代码实际执行的工具外，agentsnap 还会捕获模型决定调用哪些工具。每个非流式 Anthropic/OpenAI llm_call 事件都会记录一个 tool_requests 列表——即模型返回的 tool_use 块，每项格式为 {"name": ..., "args": {...}}。Groq 和 OpenRouter 因继承自 OpenAIAdapter，也自动获得该能力。

在断言阶段，agentsnap 会比较模型请求的工具序列（而不仅是代码实际执行的工具）。如果其发生变化，则 model_tools 检查失败；如果请求的是同一个工具但参数不同，则 model_tool_args 检查失败——报告中显示为 [MODEL TOOLS] ...。即使代码自身的工具调用逻辑没有改动，它也可以捕获模型悄悄选择了与黄金运行不同的工具的情况，例如模型更新、提示词注入或提供商侧回归。

[MODEL TOOLS] Model-requested tool sequence changed (edit distance 1): ['search'] -> ['delete_file']

[ARGS] model_tool:search->delete_file[0]:
  args: {'query': 'capital of France'} -> {'path': '/etc/passwd'}

Failed checks: ['model_tools', 'model_tool_args']

向后兼容：仅当差异比较的双方中每一个 llm_call 事件都携带 tool_requests 时，才会启用该比较。请注意，这一条件以整个调用轨迹为范围，而非按单个事件判断：轨迹中任意一个流式调用或非 Anthropic/OpenAI 调用，都会禁用整个运行的模型工具检查。旧的黄金快照（在此功能推出前录制）不会因新增检查项而失败。

当前范围：非流式 Anthropic 和 OpenAI 调用，以及通过继承支持的 Groq/OpenRouter。尚未捕获流式 tool_use 的组装结果。

完整可运行演示见 examples/model_tools.py。

---

提供商	Adapter	拦截方法
Anthropic	AnthropicAdapter	.messages.create()
OpenAI	OpenAIAdapter	.chat.completions.create()
Google Gemini	GeminiAdapter	.models.generate_content()
Cohere	CohereAdapter	.chat()
Mistral	MistralAdapter	.chat.complete()
Groq	GroqAdapter	.chat.completions.create()
OpenRouter	OpenRouterAdapter	.chat.completions.create()
LangGraph	LangGraphAdapter	.invoke() + 通过回调获取节点级 LLM/工具事件
任意可调用对象	ToolAdapter	直接调用

按需安装提供商 SDK：

pip install agentsnap[google]    # google-genai
pip install agentsnap[cohere]    # cohere
pip install agentsnap[mistral]   # mistralai
pip install agentsnap[groq]      # groq
pip install agentsnap[all-providers]

---

支持你的框架

框架会在内部创建自己的 SDK 客户端，因此无需进行包装——PatchSet 会修补底层 SDK 类（包括同步和异步 Anthropic/OpenAI chat，以及 OpenAI Responses API），因此任何构建于这些 SDK 之上的框架都能被自动捕获。

框架	方式	已通过 CI 验证
Pydantic AI	PatchSet——异步 OpenAI/Anthropic 客户端	✅
OpenAI Agents SDK	PatchSet——Responses API	✅
LangChain	PatchSet——同步 + 异步 chat	✅
LangGraph	使用 LangGraphAdapter 获取节点级事件，或使用 PatchSet	✅（已有支持）
CrewAI	通过 LiteLLM 的 OpenAI 兼容同步路径工作	已文档说明，未通过 CI 验证

通用模式——仅包装框架调用，其他内容无需修改：

from agentsnap import PatchSet
from agentsnap.core.asserter import AgentAsserter

with PatchSet():
    with AgentAsserter("my_framework_agent") as a:
        a.output = my_pydantic_ai_agent.run_sync("What is Python?").output

注意事项：

• 流式 OpenAI Responses API 运行（responses.create(stream=True)）在当前版本会直接透传且不被录制——非流式 Responses 调用以及所有 chat-completions 流式调用（同步和异步）均可被录制和重放。
• 模型工具检查（见下文）受整个调用轨迹的条件限制：如果轨迹中的任意调用是流式调用或非 Anthropic/OpenAI 提供商调用，整次运行的 model_tools/model_tool_args 比较都会被跳过。

真实框架验证测试位于 tests/frameworks/ 中（使用 frameworks marker，并由 pytest.importorskip 保护；通过单独的 CI 任务运行，安装依赖为 .[dev,frameworks]）。这些测试使用离线 mock transport 驱动每个框架的真实代码路径，并对 agentsnap 录制的调用轨迹进行断言。

---

使用 adapters（替代方案）

如果你偏好显式拦截，可以使用相应 adapter 包装 SDK 客户端，而不是使用 PatchSet。这适用于希望明确指定捕获范围，或直接控制 Agent 代码的场景：

from agentsnap import AgentRecorder
from agentsnap.adapters.anthropic import AnthropicAdapter

client = AnthropicAdapter(anthropic.Anthropic())   # 显式包装器

with AgentRecorder("my_agent") as rec:
    result = my_agent(client, "What is Python?")
    rec.output = result

*注意：

Deterministic snapshot testing for AI agents.

Full documentation, including guides and API reference, lives at iamfaham.github.io/AgentSnap.

agentsnap records your agent's LLM and tool calls during a golden run and produces a committed snapshot file. On every subsequent run it replays the same inputs and compares the new trace against the snapshot across four dimensions:

Dimension	What it checks	How
Structural	Tool call names and order	Levenshtein edit distance on the tool sequence
Arguments	Tool call arguments	deepdiff (if installed) or plain dict diff, with configurable ignored fields
Model tools	Which tool the model itself requested (not just what your code executed)	Levenshtein edit distance on tool_requests, plus per-request argument diffs
Semantic	LLM responses and final output	Cosine similarity via all-MiniLM-L6-v2, or an LLM judge for higher accuracy

If any dimension drifts beyond its threshold, agentsnap raises AgentRegressionError with a structured diff report.

---

Agents regress silently. A prompt tweak, a model swap, a tool wired to the wrong argument — nothing throws an exception, nothing fails CI, and you find out in production when the agent quietly starts giving worse answers.

agentsnap gives you two modes for two different jobs:

• Replay, on every PR — recorded responses are replayed instead of calling a real API. Deterministic, zero cost, catches code regressions (prompt edits, broken tool wiring, changed call counts).
• Live, nightly — real API calls against the current model, catching drift that only shows up when the model itself changes.

A prompt edit caught by replay mode, no API call required:

Agent regression in 'replay'
============================

[ARGS] llm_call[0].messages:
  messages: [{'content': 'Answer concisely: What is Python?', ...}] ->
            [{'role': 'user', 'content': 'You are a pirate. Answer: ...'}]

[SEMANTIC] llm_call[0]: 100% PASS
[SEMANTIC] output: 100% PASS

Failed checks: ['llm_requests']

---

Asks you to choose a semantic comparison backend:

Option	What it needs	Best for
[1] LLM judge (default)	API key (OpenRouter, OpenAI, Anthropic, or custom)	Factual agents, highest accuracy
[2] Offline embeddings	Nothing — ~22 MB model download, runs anywhere	Any machine, no API key
[3] Local LLM judge	(coming soon)	Strong local machine, no cloud

The wizard saves your choice to pyproject.toml and your API key (if any) to .env. Keys are never written to pyproject.toml. It also adds __agent_snapshots__/.last_run/ to .gitignore (creating the file if needed) and offers to scaffold an example snapshot test at tests/test_agentsnap_example.py.

agentsnap check   # verify your setup at any time

3 — Record your agent (no code changes needed)

PatchSet patches all installed LLM SDKs at the class level — any raw client created anywhere is captured automatically. No need to wrap your clients:

from agentsnap import PatchSet, AgentRecorder
import anthropic

# your existing agent — untouched
def my_agent(question):
    client = anthropic.Anthropic()   # raw client, no wrapper needed
    return client.messages.create(...).content[0].text

# First run: records the golden snapshot
with PatchSet():
    with AgentRecorder("my_agent") as rec:
        result = my_agent("What is Python?")
        rec.output = result
# Writes __agent_snapshots__/my_agent.json — commit this file

4 — Assert on future runs

from agentsnap import PatchSet, AgentAsserter

with PatchSet():
    with AgentAsserter("my_agent") as a:
        result = my_agent("What is Python?")
        a.output = result
# Raises AgentRegressionError if behavior drifted

5 — Use the pytest fixture (simplest)

snapshot.run() auto-records on first call and auto-asserts on every run after — no switching needed. Add agentsnap_instrument to activate PatchSet automatically:

def test_my_agent(snapshot, agentsnap_instrument):
    with snapshot.run("my_agent") as s:
        result = my_agent("What is Python?")   # raw client — captured automatically
        s.output = result


pytest
# or enable PatchSet for every test in a session:
pytest --agentsnap-instrument

---

Every assert can run in one of two modes:

Mode	LLM calls	Catches	Best for
live (default)	Real API	Model/behavior drift	Nightly runs, pre-release
replay	None — recorded responses are replayed	Code regressions: prompt edits, tool wiring, loop logic	Every PR / CI

In replay mode the recorded response for each LLM call is fed back to your agent — no API key, no cost, no flakes. The comparison flips to the request side : agentsnap fails the test if your code sends different prompts, makes a different number of LLM calls, or changes the tool sequence.

# per test
with AgentAsserter("my_agent", mode="replay") as a: ...

# whole suite
pytest --agentsnap-replay        # force replay
pytest --agentsnap-live          # force live


[tool.agentsnap]
mode = "replay"   # make replay the project default

Tool calls still execute for real in replay mode. Pass replay_tools=True to stub them from the recording too (no side effects at all).

Notes:

• Replay needs snapshots recorded with agentsnap >= 0.2.0 (they include raw_response). Older snapshots raise SnapshotFormatError — re-record with pytest --agentsnap-record.
• Replay currently supports Anthropic, OpenAI, Groq, and OpenRouter. Other providers raise ReplayError — use live mode for those tests.
• With scenarios, pass scenario= explicitly in replay mode (input auto-hash is not available because the snapshot is read before the test body runs).
• If the replayed final output isn't byte-identical to the golden, scoring it needs a semantic backend — install and configure the embeddings backend (pip install agentsnap[offline], then agentsnap init option 2) or configure a judge (AGENTSNAP_JUDGE_API_KEY).
• Async clients (AsyncAnthropic, AsyncOpenAI) are intercepted too — replay's no-network guarantee covers both sync and async clients, including async streams. The one remaining hole is the streamed OpenAI Responses API (responses.create(stream=True)), which passes through unrecorded. See examples/async_agents.py.

AnthropicAdapter and OpenAIAdapter tee stream=True calls (Groq and OpenRouter inherit this from OpenAIAdapter): chunks are forwarded to your agent unmodified while the assembled response is recorded for replay, with raw_response={"__stream__": True, "chunks": [...]}.

In replay mode the recorded chunks are rebuilt into real SDK chunk/event objects and yielded back incrementally — your agent consumes them exactly like a live stream, with zero API calls. A recording made from a streaming call cannot replay as a non-streaming request (or vice versa) — that raises ReplayError with a clear "shape mismatch" message.

Not yet supported: the client.messages.stream() context-manager helper, and streamed OpenAI Responses-API calls. Mistral still forces stream=False on every call. See examples/streaming.py for a full runnable walkthrough, and examples/async_agents.py for the async-client version.

A stream that is never iterated and never closed is finalized automatically at recorder/asserter exit, but consuming or closing it promptly is still recommended so events appear in call order.

---

Beyond the tools your code actually executes, agentsnap also captures which tool the model decided to call. Every non-streaming Anthropic/OpenAI llm_call event records a tool_requests list — the tool_use blocks the model returned, each as {"name": ..., "args": {...}}. Groq and OpenRouter get this for free since they subclass OpenAIAdapter.

On assert, agentsnap compares the model's requested tool sequence (not just what your code executed) and fails model_tools if it changed, or model_tool_args if the same tool was requested with different arguments — surfaced in the report as [MODEL TOOLS] .... This catches a model quietly choosing a different tool than the golden run even when your code's own tool-calling logic is untouched (a model update, a prompt injection, a provider-side regression).

[MODEL TOOLS] Model-requested tool sequence changed (edit distance 1): ['search'] -> ['delete_file']

[ARGS] model_tool:search->delete_file[0]:
  args: {'query': 'capital of France'} -> {'path': '/etc/passwd'}

Failed checks: ['model_tools', 'model_tool_args']

Backward compatible: the comparison only engages when every llm_call event on both sides of the diff carries tool_requests. Note this gate is trace-wide, not per-event: a single streamed call or non-Anthropic/OpenAI call anywhere in the trace disables the model-tools check for the whole run. Old goldens (recorded before this feature) never fail from the new surface.

Scope today: non-streaming Anthropic and OpenAI calls, plus Groq/OpenRouter via inheritance. Streamed tool_use assembly is not captured yet.

See examples/model_tools.py for a full runnable walkthrough.

---

Provider	Adapter	Intercepts
Anthropic	AnthropicAdapter	.messages.create()
OpenAI	OpenAIAdapter	.chat.completions.create()
Google Gemini	GeminiAdapter	.models.generate_content()
Cohere	CohereAdapter	.chat()
Mistral	MistralAdapter	.chat.complete()
Groq	GroqAdapter	.chat.completions.create()
OpenRouter	OpenRouterAdapter	.chat.completions.create()
LangGraph	LangGraphAdapter	.invoke() + node-level LLM/tool events via callbacks
Any callable	ToolAdapter	direct call

Install provider SDKs as needed:

pip install agentsnap[google]    # google-genai
pip install agentsnap[cohere]    # cohere
pip install agentsnap[mistral]   # mistralai
pip install agentsnap[groq]      # groq
pip install agentsnap[all-providers]

---

Works with your framework

Frameworks build their own SDK clients internally, so there's nothing to wrap — PatchSet patches the underlying SDK classes (sync and async Anthropic/OpenAI chat, plus the OpenAI Responses API), so any framework built on top of them is captured automatically.

Framework	How	CI-verified
Pydantic AI	PatchSet — async OpenAI/Anthropic clients	✅
OpenAI Agents SDK	PatchSet — Responses API	✅
LangChain	PatchSet — sync + async chat	✅
LangGraph	LangGraphAdapter for node-level events, or PatchSet	✅ (existing)
CrewAI	Works via LiteLLM's OpenAI-compatible sync path	documented, not CI-verified

The universal pattern — wrap the framework call, nothing else changes:

from agentsnap import PatchSet
from agentsnap.core.asserter import AgentAsserter

with PatchSet():
    with AgentAsserter("my_framework_agent") as a:
        a.output = my_pydantic_ai_agent.run_sync("What is Python?").output

Caveats:

• Streamed OpenAI Responses-API runs (responses.create(stream=True)) pass through unrecorded this iteration — non-streaming Responses calls and all chat-completions streaming (sync + async) are recorded and replayable.
• The model-tools check (see below) is gated trace-wide: if any call in the trace is a streamed call or a non-Anthropic/OpenAI provider, the whole run's model_tools/model_tool_args comparison is skipped.

Real-framework verification tests live in tests/frameworks/ (marker frameworks, pytest.importorskip-guarded, run via a separate CI job with .[dev,frameworks] installed) — they drive each framework's real code path through an offline mock transport, asserting on agentsnap's recorded trace.

---

Using adapters (alternative)

If you prefer explicit interception, wrap your SDK client with the matching adapter instead of using PatchSet. Useful when you want to be explicit about what is captured or when you control the agent code directly:

from agentsnap import AgentRecorder
from agentsnap.adapters.anthropic import AnthropicAdapter

client = AnthropicAdapter(anthropic.Anthropic())   # explicit wrapper

with AgentRecorder("my_agent") as rec:
    result = my_agent(client, "What is Python?")
    rec.output = result

Note: Do not combine adapters with PatchSet on the same client — both interceptors will fire and events will be recorded twice.

---

API key for the LLM judge (optional)

The LLM judge uses a small language model to compare outputs instead of embeddings — more accurate for factual content.

agentsnap resolves the API key automatically — you do not need a separate key. It checks in this order:

1. AGENTSNAP_JUDGE_API_KEY — explicit override, always wins
2. The provider-specific key that matches judge_base_url:

judge_base_url contains	Key used automatically
openrouter.ai (default)	OPENROUTER_API_KEY
api.openai.com	OPENAI_API_KEY
anthropic.com	ANTHROPIC_API_KEY
api.groq.com	GROQ_API_KEY
api.mistral.ai	MISTRAL_API_KEY
api.cohere.com	COHERE_API_KEY

Once any matching key is found, the snapshot pytest fixture enables the LLM judge automatically — no code changes needed in tests.

To use a different provider, change judge_base_url in pyproject.toml and set the matching env var:

export OPENAI_API_KEY=sk-...


[tool.agentsnap]
judge_base_url = "https://api.openai.com/v1"
judge_model    = "gpt-4o-mini"

Project settings (pyproject.toml)

[tool.agentsnap]
judge_model        = "openai/gpt-4o-mini"
judge_base_url     = "https://openrouter.ai/api/v1"
semantic_threshold = 0.92   # final agent output (strict)
llm_threshold      = 0.75   # intermediate LLM responses (tolerant)
mode               = "live"   # "live" (default) or "replay"

These can also be set as pytest ini options:

[tool.pytest.ini_options]
agentsnap_judge_model        = "openai/gpt-4o-mini"
agentsnap_judge_base_url     = "https://openrouter.ai/api/v1"
agentsnap_semantic_threshold = "0.92"
agentsnap_llm_threshold      = "0.75"

---

AgentRecorder(test_name, snapshot_dir="__agent_snapshots__", model="unknown")

Context manager. Intercepts all adapter calls and writes a snapshot on clean exit.

with AgentRecorder("name", model="claude-haiku-4-5") as rec:
    rec.input_data = {"query": "hello"}   # optional metadata
    result = my_agent(wrapped_client, ...)
    rec.output = result

AgentAsserter(test_name, snapshot_dir, semantic_threshold, llm_threshold, ignored_fields, embed_fn, judge)

Context manager. Reads the snapshot, intercepts calls, runs the three-layer diff on exit. If no snapshot exists yet, automatically switches to record mode and writes the golden run.

Parameter	Default	Description
semantic_threshold	0.92	Min similarity for final output
llm_threshold	0.75	Min similarity for intermediate LLM responses
ignored_fields	None	Tool arg keys to exclude from argument diff
embed_fn	None	Custom embedding function (for testing)
judge	None	LLMJudge instance; overrides embedding comparison

with AgentAsserter("name", semantic_threshold=0.95, ignored_fields=["timestamp"]) as a:
    result = my_agent(wrapped_client, ...)
    a.output = result

Context manager that monkey-patches all installed LLM SDK classes so any client — wrapped or unwrapped — is captured by an active AgentRecorder or AgentAsserter.

from agentsnap import PatchSet

with PatchSet():
    # all anthropic.Anthropic(), openai.OpenAI(), etc. clients are auto-captured
    ...

LLMJudge(api_key, model, base_url)

Uses an LLM to score semantic equivalence instead of embeddings. Returns a 0.0–1.0 score and a one-sentence reason explaining any difference.

from agentsnap import LLMJudge

judge = LLMJudge(api_key="sk-or-...", model="openai/gpt-4o-mini")
judge = LLMJudge.from_env()   # returns None if no key found

with AgentAsserter("name", judge=judge) as a:
    ...

Auto-wired from [tool.agentsnap] and environment variables. No imports needed.

def test_agent(snapshot):
    # Auto mode: records first time, asserts every run after
    with snapshot.run("name") as s:
        s.output = run_agent(...)

    # Explicit record
    with snapshot.record_agent("name") as rec:
        rec.output = run_agent(...)

    # Explicit assert
    with snapshot.assert_agent("name") as a:
        a.output = run_agent(...)

    # Per-test overrides
    with snapshot.assert_agent("name", judge=False) as a:          # force embeddings
        a.output = run_agent(...)
    with snapshot.assert_agent("name", semantic_threshold=0.98) as a:
        a.output = run_agent(...)

Every snapshot.run() / record_agent() / assert_agent() use feeds a terminal summary section ("agentsnap snapshots") printed at the end of the pytest run. Known limitation: under pytest-xdist, this summary is per-worker and is not aggregated across workers — run without -n if you need the full picture.

Flag	Description
--agentsnap-record	Force re-record all snapshots, overwriting existing goldens
--agentsnap-instrument	Auto-patch all installed LLM SDKs (zero-instrumentation mode)
--agentsnap-replay	Force replay mode for every test in the session
--agentsnap-live	Force live mode for every test in the session

pytest --agentsnap-record        # re-record everything
pytest --agentsnap-instrument    # capture raw clients without adapters
pytest --agentsnap-replay        # force replay mode
pytest --agentsnap-live          # force live mode

agentsnap_instrument fixture

Standalone fixture for zero-instrumentation capture within a single test:

def test_agent(snapshot, agentsnap_instrument):
    with snapshot.run("name") as s:
        client = anthropic.Anthropic()   # no adapter needed
        s.output = my_agent(client, "query")

Exception	When raised
AgentRegressionError(message, diff_report)	Behavior drifted beyond threshold
SnapshotNotFoundError(test_name)	No snapshot found (only from direct SDK use; AgentAsserter auto-records instead)
AdapterNotWrappedError	Unwrapped client used inside a recording context without PatchSet

AgentRegressionError.diff_report is a DiffReport dataclass with structural_diff, argument_diffs, semantic_scores, semantic_reasons, and failed_checks.

---

agentsnap init                                     # interactive setup wizard — choose backend and save config
agentsnap check                                    # verify current backend is working (exits 0/1)
agentsnap list                                     # list all snapshots
agentsnap status                                   # pass/fail/stale status for every snapshot (CI-friendly, exits 0/1)
agentsnap diff __agent_snapshots__/my_agent.json   # pretty-print a snapshot
agentsnap update my_agent                          # show diff and approve last run as new golden
agentsnap update my_agent --yes                    # approve without confirmation prompt
agentsnap update --all                             # batch-approve every failing or new snapshot

---

{
  "version": "1.1",
  "recorded_at": "2026-01-01T00:00:00+00:00",
  "model": "claude-haiku-4-5",
  "input": { "query": "What is Python?" },
  "trace": [
    { "step": 0, "type": "llm_call", "messages": [...], "response": "...", "tokens": 350, "raw_response": {...} },
    { "step": 1, "type": "tool_call", "name": "search", "args": {"query": "Python"}, "result": "..." }
  ],
  "output": "Python is a high-level programming language..."
}

Golden snapshots live in __agent_snapshots__/ and are committed to git. The .last_run/ subdirectory is written on every assert run and is gitignored — it is only used by agentsnap update.

---

CI integration (GitHub Actions)

name: Agent regression tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - name: Install
        run: pip install -e ".[dev]"

      - name: Run agent snapshot tests
        run: pytest tests/ -v
        env:
          # Optional: enables LLM judge for higher-accuracy semantic comparison
          AGENTSNAP_JUDGE_API_KEY: ${{ secrets.AGENTSNAP_JUDGE_API_KEY }}

Snapshots are committed to the repo. CI only runs the asserter — no real agent API calls needed unless your tests explicitly make them.

Dogfooding / continuous real-API validation

agentsnap validates itself against live provider APIs: a monthly-gated live-validation workflow runs examples/run_all.py --real, and an sdk-drift workflow checks the hermetic suite against the latest unpinned provider SDKs. Both are maintainer-facing and never gate a PR — see CONTRIBUTING.md for how to arm/run them.

---

How to approve an intentional regression

When you intentionally change agent behavior (new prompt, model upgrade, new tool):

# 1. Run tests — they fail, new trace saved to .last_run/
pytest tests/test_my_agent.py

# 2. Approve — shows a diff and prompts for confirmation
agentsnap update my_agent

# 3. Commit the new baseline
git add __agent_snapshots__/my_agent.json
git commit -m "approve: updated golden after Sonnet upgrade"

For multiple failures at once, run agentsnap status to see what changed, then agentsnap update --all to batch-approve every failing or new snapshot in one pass.

---

Two independent thresholds control the semantic layer:

Threshold	Default	Applies to
semantic_threshold	0.92	Final output — the agent's actual answer
llm_threshold	0.75	Intermediate llm_call[n] responses — tolerates natural phrasing variance

Tune per-test:

# Critical factual agent — hold output tightly
with AgentAsserter("rag_agent", semantic_threshold=0.97) as a: ...

# Creative agent — allow more paraphrasing
with AgentAsserter("writer_agent", semantic_threshold=0.75) as a: ...