基于 OpenTelemetry 的 AI Agent 可观测性

最后更新：2026 年 10 月 19 日 · 18 分钟阅读

AI Agent 不仅是调用模型并返回响应，更负责编排整个工作流：为 LLM 挑选工具、评估安全护栏（Guardrails）、在多轮 LLM 调用之间路由，以及在对话轮次间管理上下文。这意味着单个用户请求就可能派生出数十个内部步骤，每一步都是潜在的故障点、延迟源或意外成本来源。

AI Agent 可观测性是指捕获这些执行流程的实践，使你能够理解 Agent 做了什么、为什么选择该路径以及其性能表现如何。在实践中，这意味着追踪 Agent 运行的执行路径，捕获每一步的输入和输出，并监测成本、Token 使用量、延迟和错误率。

AI Agent 链路追踪（Tracing）构成了 AI Agent 可观测性的基石。没有追踪，调试 AI Agent 就会沦为一场盲目猜测的游戏：开发者只能凭空推测究竟是哪个事件触发了 Agent 工作流中的哪项决策，以及其背后的原因。

一个请求可能在 HTTP 层面成功返回，但 Agent 却给出了错误答案、在某个步骤耗时过长，或者触发了重复的 LLM 或工具调用。Agent 可观测性为工程师提供了调查这些问题并跨运行对比行为表现的关键依据。

我们来深入探究其中的几类典型问题。

单次 LLM 调用本身就具备非确定性：相同的 Prompt 在不同次运行中可能会产生不同的输出。Agent 会将这种非确定性放大，因为它们把多次 LLM 调用串联在一起，上一步的输出直接决定了下一步的输入。

这意味着偏差并不仅存在于单一节点，而是在整个工作流中不断累积复合。第一步中模型稍有差异的响应，可能导致 Agent 在第二步选择完全不同的工具，进而改变传递给第三步的上下文，依此类推。如果不捕获完整的执行路径，几乎不可能复现某次特定的 Agent 运行。

传统软件在返回错误结果时可能并不抛出异常。在 Agent 场景下，这类故障更加难以察觉，因为每次运行的路径可能都不相同：工作流看似顺利完成且无报错，但 Agent 却选错了工具、检索了过期的上下文，或者自信地返回了一个完全错误的回答。

缺乏可观测性时，这些静默失败（Silent Failures）只有在用户投诉或下游系统消费了错误输出时才会暴露。对 Agent 循环中的每一个决策进行追踪——例如选择了哪个工具、当时有哪些可用上下文、模型实际返回了什么——是评估 Agent 的“成功运行”在逻辑上是否真正正确的唯一可靠途径。

成本与延迟在设计上具有不可预测性

Agent 在运行时自主决定调用多少次工具以及与 LLM 进行多少轮往返交互。一个简单的用户查询可能一次 LLM 调用即可解决，而另一个类似的查询可能因 Agent 的推理逻辑而触发多次工具调用和模型调用。

这使得成本和延迟天然不可预测。可观测性使工程师能够追踪不同运行中的 Token 消耗、工具调用频率以及单步延迟，从而及早发现高成本模式。例如，如果两个价格档次不同的模型在特定工作负载下的表现相当，工程师便可以切换到更便宜的选项——前提是拥有支持这一对比决策的数据支撑。

OpenTelemetry (OTel) 是 CNCF 的毕业项目，旨在为应用程序插桩以输出中立于供应商的遥测数据（主要是 Trace、Metric 和 Log）。使其特别契合 Agent 应用的关键在于：它的可见性不仅覆盖 AI Agent 本身，还延伸至整个请求生命周期——用户调用了哪个端点、执行了哪些数据库查询、是否入队了后台任务、应用程序是否发起了下游 API 调用？

OpenTelemetry 允许开发者对整个应用程序（包括其 AI Agent）进行插桩。

SigNoz Cloud 是一个 OpenTelemetry 原生的可观测性平台，在单一界面中统一了 Trace、Metric 和 Log。

• 注册 一个免费的 SigNoz Cloud 账号。
• 按照文档说明 生成 Ingestion Key。
• 准备好你的 Workspace 区域（例如 us、in、eu）和 Ingestion Key，以便后续配置应用。

克隆并进入示例应用目录：

bash git clone --filter=blob:none --sparse https://github.com/SigNoz/examples.git git -C examples sparse-checkout set python/otel-ai-agent-observability-demo cd examples/python/otel-ai-agent-observability-demo

创建并激活 Python 虚拟环境：

bash python3.12 -m venv .venv source .venv/bin/activate

安装应用程序和 OpenTelemetry Python 依赖：

bash pip install -r requirements.txt

requirements.txt 文件 锁定了以下 OTel 依赖包：

• opentelemetry-distro：提供便捷机制来配置 Python 自动插桩中最常见的选项。
• opentelemetry-exporter-otlp：安装将遥测数据发送到任何 OpenTelemetry 兼容后端 所需的 OTLP Exporter。
• opentelemetry-instrumentation-genai-openai 与 opentelemetry-instrumentation-genai-openai-agents：来自独立的 OpenTelemetry GenAI 代码仓库 的最新 OpenAI Agents 与 OpenAI Python API 插桩包。

接下来，运行 opentelemetry-bootstrap 检测主流 Python 库（如 FastAPI Web 框架 和 HTTPX 客户端库）并安装对应的插桩依赖包：

bash opentelemetry-bootstrap --action=install

残留的 OpenAI 插桩包提示

opentelemetry-bootstrap 目前默认会安装已弃用的 opentelemetry-instrumentation-openai-v2 包。 不过，由于我们的 requirements.txt 显式安装了更新的 opentelemetry-instrumentation-genai-* 库，OpenTelemetry 在运行时会解析使用较新的包。

你可以保留该 v2 包，也可以选择将其卸载。

使用 opentelemetry-instrument 启动应用程序，以对应用及 AI Agent 工作流进行自动插桩：

bash OPENAI_API_KEY=""
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest..signoz.cloud:443"
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key="
OTEL_SERVICE_NAME="otel-ai-agent-observability-demo"
OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=dev"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
opentelemetry-instrument fastapi run --port 8085 --workers 1

以下是这些 OpenTelemetry 环境变量 的配置作用：

• OTEL_SERVICE_NAME="otel-ai-agent-observability-demo"：在 SigNoz Cloud 中唯一标识该应用。
• OTEL_RESOURCE_ATTRIBUTES：向所有导出的遥测数据附加元数据（版本 1.0.0，环境 dev），用于在 OpenTelemetry 后端组织和筛选遥测数据。
• OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only：将模型 Prompt、补全结果及工具定义作为 Span 属性记录在 LLM 模型 Span 上。

将日志与 Trace 关联

通过 opentelemetry-bootstrap 安装的日志插桩库会自动将应用日志与 Trace 关联。这样便无需配置 OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED 环境变量，该变量在较新的 OpenTelemetry 版本中已被废弃。

FastAPI 服务在 8085 端口启动。请求根路径 / 将返回健康检查响应：

bash (.venv) ❯ curl "http://127.0.0.1:8085/" {"message":"OpenTelemetry NBA agent demo is running"}

演示程序现已正常运行。在深入了解 Agent 代码之前，我们先来看看用于捕获 AI Agent 工作流的 OpenTelemetry 规范与插桩包。

理解 OpenTelemetry 不断演进的 GenAI 规范

早期的插桩讨论仅聚焦于“LLM 追踪”，而 OpenTelemetry 现已将 AI 遥测统一纳于更广泛的“生成式 AI”（GenAI）范畴下，采用以 gen_ai.* 为前缀的语义约定（Semantic Conventions）。

由于 GenAI 模式和 Agent 框架的迭代速度远快于传统 HTTP 或数据库库，OpenTelemetry 创建了专用代码仓库 open-telemetry/opentelemetry-python-genai，以便维护者独立开发和发布这些插桩库。

这次演进引入了更为清晰的分层作用域架构（Split-scope Architecture）：

1. opentelemetry.instrumentation.genai.openai_agents：对 Agent 编排框架进行插桩，针对工作流执行（invoke_workflow）和 Agent 路由（invoke_agent）生成 Span。
2. opentelemetry.instrumentation.genai.openai：配套插桩包，用于插桩底层模型调用（chat），捕获 Token 使用量（gen_ai.usage.input_tokens、gen_ai.usage.output_tokens）、模型参数和 Prompt 上下文。

早期的 Agents 插桩方案 将工作流追踪与模型调用追踪合并在单个包中。

此外，由于 opentelemetry-bootstrap 尚未将这些专用的 GenAI 依赖包纳入其自动发现清单，因此必须通过 requirements.txt 或 pyproject.toml 单独安装并维护它们。

开发者必须手动安装最新的 GenAI 插桩库，因为它们尚未在 opentelemetry-bootstrap 中注册。

GenAI 插桩包的成熟度各异

不同 GenAI 插桩包的功能覆盖度存在差异，尤其是在 Agent 移交（Handoffs）与护栏（Guardrails）等特性方面。建议显式锁定插桩依赖包版本，并在升级依赖时仔细检查变更日志。

NBA Agent 如何使用工具、护栏与会话

了解了 OpenTelemetry 如何分离 Agent 编排与模型调用之后，我们来看一下 NBA Agent 工作流背后的实现代码。如果你想直接生成并检查 Trace，可以跳至下一节。

应用由 FastAPI 提供 HTTP 端点，OpenAI Agents SDK 管理 Agent 循环，OpenAI Python API 库处理底层的模型调用。这种分工正是我们需要同时对 Agents SDK 和 API 库进行插桩的原因：前者捕获工作流，后者捕获每一次模型调用。

我们的 Web 应用提供了一个 NBA 记者 Agent，用于回答 NBA 相关问题（如常规新闻 general 或总决赛 finals）、进行网络调研、计算球队胜率，并通过服务端会话持久化保留多轮对话历史。

agent_service.py 文件 定义了 Agent 配置以及供 FastAPI 端点调用的执行函数。

我们使用系统提示词、工具、输入护栏和指定的模型来初始化 Agent。该应用采用了 gpt-6-luna（定义在 app/config.py 中的 OPENAI_MODEL = "gpt-6-luna"），它能提供稳定的多轮推理和工具调用格式化表现：

python NBA_AGENT = Agent( name="NBA_Reporter", instructions=NBA_INTERACTIVE_PROMPT, tools=[WebSearchTool(), calculate_win_percentage], input_guardrails=[nba_content_guardrail], model=OPENAI_MODEL, )

calculate_win_percentage 函数工具为 Agent 提供了一种确定性计算和展示球队胜率的方式，消除了算术幻觉的风险：

python @function_tool def calculate_win_percentage(wins: int, losses: int) -> str: """Calculates the winning percentage for an NBA team given their wins and losses."""

total_games = wins + losses
if total_games == 0:
    return ".000"

return f"{wins / total_games:.3f}".lstrip("0")

为防止滥用或偏离主题的提问，我们定义了输入护栏。当用户 Prompt 与篮球相关关键词不匹配时，该函数会返回一个 tripwire_triggered=True 的 GuardrailFunctionOutput：

python @input_guardrail() def nba_content_guardrail( context: RunContextWrapper[None], agent: Agent, input_data: str | list, ) -> GuardrailFunctionOutput: """Ensures the user query is relevant to basketball/NBA."""

keywords = [
    "nba",
    "basketball",
    "player",
    ... # 此处略去部分代码

AI Agent Observability with OpenTelemetry

Last Updated: October 19, 202618 min read

AI agents don't just call a model and return its response. Instead, they orchestrate entire workflows that involve selecting tools for LLMs to use, evaluating guardrails, routing between LLM calls, and managing context across conversation turns. This means that a single user request can spawn dozens of internal steps, each one a potential point of failure, latency, or unexpected cost.

AI agent observability is the practice of capturing these execution flows so you can understand what the agent did, why it chose that path, and how it performed. In practice, this means tracing an agent run's execution path, capturing inputs and outputs at each step, and tracking cost, token usage, latency, and error rates.

AI agent tracing builds the foundation for AI agent observability. Without tracing, debugging an AI agent becomes a game of trying to guess which event triggered what decision in the agent workflow, and why.

A request can return successfully while the agent gives an incorrect answer, takes too long on a particular step, or triggers duplicate LLM or tool calls. Agent observability gives engineers the evidence to investigate these problems and compare behavior across runs.

Let's look at some of these problems in more detail.

A single LLM call is already non-deterministic: the same prompt can produce different outputs across runs. Agents amplify this because they chain multiple LLM calls together, where the output of one step determines the input of the next.

This means variance doesn't just exist at one point. Rather, it compounds across the workflow. A slightly different model response in step one can cause the agent to select a completely different tool in step two, which changes the context passed to step three, and so on. Reproducing a specific agent run becomes nearly impossible without capturing the full execution path.

Traditional software can also return incorrect results without throwing an error. With agents, these failures can be harder to catch because each run can take a different path: the workflow completes without any error, but the agent selected the wrong tool, retrieved stale context, or returned a response that is confidently incorrect.

Without observability, these silent failures only surface when a user complains or when downstream systems consume bad output. Tracing each decision in the agentic loop, such as which tool was selected, what context was available, what the model actually returned, is the only reliable way to evaluate whether an agent's "successful" run was actually logically correct.

Cost and latency are unpredictable by design

Agents decide at runtime how many tools to call and how many LLM roundtrips to make. A simple user query might resolve in one LLM call, while a similar query might trigger several tool calls and model invocations depending on the agent's reasoning.

This makes cost and latency inherently unpredictable. Observability lets engineers track token usage, tool call frequency, and per-step latency across runs to identify expensive patterns early. For example, if two models with different pricing tiers perform comparably for a given workload, engineers can switch to the cheaper option — but only if they have the data to make that comparison.

OpenTelemetry (OTel) is a CNCF graduated project for instrumenting applications to emit telemetry data (primarily traces, metrics, and logs) in a vendor-neutral format. What makes it uniquely suited for agentic applications is that your visibility extends beyond the AI agent to the entire request: what endpoint did the user call, what DB queries ran, were any background jobs enqueued, did the application make any downstream API calls?

OpenTelemetry enables developers to instrument entire applications, including their AI agents.

SigNoz Cloud is an OpenTelemetry-native observability platform that unifies traces, metrics, and logs in a single interface.

• Sign up for a free SigNoz Cloud account.
• Follow the documentation to generate an ingestion key.
• Keep your workspace region (e.g., us, in, eu) and ingestion key handy for the application configuration.

Clone and navigate to the demo application:

Copy

git clone --filter=blob:none --sparse https://github.com/SigNoz/examples.git
git -C examples sparse-checkout set python/otel-ai-agent-observability-demo
cd examples/python/otel-ai-agent-observability-demo

Create and activate a Python virtual environment:

Copy

python3.12 -m venv .venv
source .venv/bin/activate

Install the application and OpenTelemetry Python dependencies:

Copy

pip install -r requirements.txt

The requirements.txt file pins the following OTel packages:

• opentelemetry-distro: Provides a convenient mechanism to configure some of the more common options for Python auto-instrumentation.
• opentelemetry-exporter-otlp: Installs the OTLP exporters needed to deliver telemetry data to any OpenTelemetry-compatible backend.
• opentelemetry-instrumentation-genai-openai and opentelemetry-instrumentation-genai-openai-agents: Modern OpenAI Agents and OpenAI Python API instrumentation packages from the dedicated OpenTelemetry GenAI repository.

Next, run opentelemetry-bootstrap to detect widely used Python libraries (like the FastAPI web framework and HTTPX client library) and install their corresponding instrumentation packages:

Copy

opentelemetry-bootstrap --action=install

Dangling OpenAI instrumentation package

opentelemetry-bootstrap currently installs a deprecated opentelemetry-instrumentation-openai-v2 package by default.
However, because our requirements.txt explicitly installs the modern opentelemetry-instrumentation-genai-* libraries, OpenTelemetry resolves to the newer packages at runtime.

You can safely leave the v2 package alone or uninstall it.

Start the application using opentelemetry-instrument to automatically instrument our application, including the AI agent workflow:

Copy

OPENAI_API_KEY="<your-openai-api-key>" \
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<your-region>.signoz.cloud:443" \
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>" \
OTEL_SERVICE_NAME="otel-ai-agent-observability-demo" \
OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=dev" \
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only \
opentelemetry-instrument fastapi run --port 8085 --workers 1

Here is what these OpenTelemetry environment variables configure:

• OTEL_SERVICE_NAME="otel-ai-agent-observability-demo": Uniquely identifies the application in SigNoz Cloud.
• OTEL_RESOURCE_ATTRIBUTES: Attaches metadata (version 1.0.0, environment dev) to all exported telemetry. This is used to organize and filter telemetry in the OpenTelemetry backend.
• OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only: Captures model prompts, completions, and tool definitions as span attributes on LLM model spans.

Correlating logs to traces

Application logs are automatically correlated with traces by the opentelemetry-bootstrap-installed logging instrumentation. This eliminates the need to set the OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED environment variable, which has been deprecated in the more recent OpenTelemetry releases.

The FastAPI server starts on port 8085. Calling / returns a quick health check response:

Copy

(.venv)  ❯ curl "http://127.0.0.1:8085/"
{"message":"OpenTelemetry NBA agent demo is running"}

The demo is now running. Before exploring the agent code, let’s look at the OpenTelemetry conventions and instrumentation packages used to capture AI agent workflows.

Understanding OpenTelemetry’s evolving GenAI conventions

While early instrumentation discussions focused solely on "LLM tracing", OpenTelemetry standardizes AI telemetry under the broader umbrella of Generative AI (GenAI), using semantic conventions prefixed with gen_ai.*.

Because GenAI patterns and agentic harnesses evolve far more rapidly than traditional HTTP or database libraries, OpenTelemetry created a dedicated repository, open-telemetry/opentelemetry-python-genai, to enable maintainers to independently develop and publish these libraries.

This migration introduced a cleaner, split-scope architecture :

1. opentelemetry.instrumentation.genai.openai_agents : Instruments the agent orchestration harness, emitting spans for workflow execution (invoke_workflow) and agent routing (invoke_agent).
2. opentelemetry.instrumentation.genai.openai : A companion package that instruments the underlying model calls (chat), capturing token usage (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens), model parameters, and prompt context.

The earlier Agents instrumentation combined workflow and model-call tracing in a single package.

Additionally, because opentelemetry-bootstrap has not yet added these dedicated GenAI packages to its automated discovery list, you must install and track them separately through your requirements.txt or pyproject.toml files.

Developers must install the latest GenAI instrumentation libraries manually as they aren't yet registered with opentelemetry-bootstrap.

Varying maturity across GenAI instrumentation packages

Coverage varies across GenAI instrumentation packages, especially for agent features such as handoffs and guardrails. It's recommended that you pin your instrumentation packages explicitly and check their changelogs when upgrading dependencies.

How the NBA agent uses tools, guardrails, and sessions

Now that we've seen how OpenTelemetry separates agent orchestration from model calls, let's look at the code behind the NBA agent's workflow. Skip to the next section if you want to jump directly to generating and inspecting traces.

FastAPI serves our HTTP endpoints, the OpenAI Agents SDK manages the agent loop, and the OpenAI Python API library handles the underlying model calls. This separation is why we instrument both the Agents SDK and the API library: one captures the workflow, while the other captures each model call.

Our web application exposes an NBA Reporter agent that answers questions on NBA topics (e.g., general news or finals), performs web research, calculates team win percentages, and retains conversation history across turns via server-managed session persistence.

The agent_service.py file defines the agent configuration and the execution function used by our FastAPI endpoint.

We initialize the agent with a system prompt, tools, input guardrails, and model selection. The application uses gpt-6-luna (OPENAI_MODEL = "gpt-6-luna" in app/config.py), which delivers consistent multi-turn reasoning and tool formatting:

Copy

NBA_AGENT = Agent(
    name="NBA_Reporter",
    instructions=NBA_INTERACTIVE_PROMPT,
    tools=[WebSearchTool(), calculate_win_percentage],
    input_guardrails=[nba_content_guardrail],
    model=OPENAI_MODEL,
)

The calculate_win_percentage function tool gives the agent a deterministic way to calculate and display team win percentages without the risk of arithmetic hallucinations:

Copy

@function_tool
def calculate_win_percentage(wins: int, losses: int) -> str:
    """Calculates the winning percentage for an NBA team given their wins and losses."""
 
    total_games = wins + losses
    if total_games == 0:
        return ".000"
 
    return f"{wins / total_games:.3f}".lstrip("0")

To prevent misuse or off-topic queries, we define an input guardrail. When a user prompt does not match relevant basketball keywords, the function returns a GuardrailFunctionOutput with tripwire_triggered=True:

@input_guardrail()
def nba_content_guardrail(
    context: RunContextWrapper[None],
    agent: Agent,
    input_data: str | list,
) -> GuardrailFunctionOutput:
    """Ensures the user query is relevant to basketball/NBA."""
 
    keywords = [
        "nba",
        "basketball",
        "player",
        ... # truncated for brevity
    ]
 
    # extract the last user message from chat history
    if isinstance(input_data, list):
        latest_user_message = next(
            (
                item.get("content", "")
                for item in reversed(input_data)
                if isinstance(item, dict) and item.get("role") == "user"
            ),
            "",
        )
        input_query = latest_user_message
    else:
        input_query = input_data
 
    input_query = input_query.lower()
    is_relevant = any(keyword in input_query for keyword in keywords)
 
    if len(input_query) < 5 or is_relevant:
        return GuardrailFunctionOutput(tripwire_triggered=False, output_info=None)
 
    return GuardrailFunctionOutput(
        tripwire_triggered=True,
        output_info={
            "reason": "The request is off-topic. Please ask questions relevant to NBA or basketball."
        },
    )

Setting tripwire_triggered=True instructs the Agents SDK to abort execution and raise an InputGuardrailTripwireTriggered exception. We intercept this in a FastAPI exception handler to record the exception event directly on the active OpenTelemetry span before returning an HTTP 400 response:

Copy

@app.exception_handler(InputGuardrailTripwireTriggered)
async def handle_guardrail_block(
    request: Request,
    exc: InputGuardrailTripwireTriggered,
) -> JSONResponse:
    guardrail_msg = exc.guardrail_result.output.output_info
    span = trace.get_current_span()
 
    span.record_exception(exc)
    return JSONResponse(status_code=400, content={"detail": guardrail_msg})

The run_agent_turn function completes our implementation. The agent uses the OpenAIConversationsSession object to persist and gather conversation context across turns.

If no session_id was passed, the session object triggers the creation of a new conversation session. In a follow-up request, the user can attach this ID to ensure the agent has context of the conversation thus far.

def run_agent_turn(
    topic: str,
    user_message: str | None,
    session_id: str | None,
) -> dict:
    nba_topic = _validate_topic(topic)
    prompt = build_nba_turn_prompt(nba_topic, user_message)
 
    # if no session ID was given, the sdk internally creates a session_id during the turn
    # subsequent calls which pass the ID maintain the conversation context
    session = OpenAIConversationsSession(conversation_id=session_id)
    result = Runner.run_sync(NBA_AGENT, prompt, session=session)
 
    message = sanitize_agent_message((result.final_output or "").strip())
    ... # truncated for brevity
 
    return {
        "topic": topic,
        "session_id": session.session_id,
        "message": message,
        "model": OPENAI_MODEL,
        "usage": usage,
    }

Let’s send the agent a few requests to generate traces, then inspect how tool calls, conversation history, and guardrail failures appear in SigNoz Cloud.

Let’s start by asking the NBA Reporter agent a question about the Lakers' win percentage in the 2025-26 season, as well as LeBron and his future with the team. This query asks three distinct questions, setting us up for follow-up questions in the next turn, and the win percentage ask should trigger the calculate_win_percentage tool.

The agent response to queries includes the session_id to continue the conversation across turns.

The agent responds first with the win percentage calculation — the format matches our tool's response shape. It makes a prediction about the upcoming NBA season, and informs us that LeBron is no longer with the LA Lakers.

The response JSON also contains the model name, token usage details at a glance, and the session_id to continue the conversation in follow-up requests.

Let's now add the session_id to our request payload and ask why LeBron switched teams. We'll refer to LeBron as he to test the agent's context retrieval capabilities.

The agent retrieves conversation history to continue the conversation across turns.

The agent correctly identifies he as LeBron in the context of the conversation, and outlines why he moved to a new team. It also returns the same session_id in the response, further confirming continuation of the previous conversation.

Finally, let’s ask an off-topic question to see whether the input guardrails trigger to prevent users from abusing our agent:

The AI agent triggers a guardrail and aborts the workflow to prevent abuse.

Because the prompt lacks basketball-related keywords, the tripwire fires immediately. This helps the agent abort execution before executing tools or calling the model, preventing unneeded API spend and latency.

Now let's visualize in SigNoz Cloud the multi-step trace that invoked tool use:

Walkthrough of the OpenTelemetry-instrumented multi-step AI agent trace.

Modern OpenTelemetry GenAI instrumentation structures the request into a clean, hierarchical span tree:

Copy

POST /agent/turn                                     [fastapi]              Server   26.00s
├─ POST /agent/turn http receive                     [asgi]                 Internal
├─ POST /v1/conversations                            [httpx]                Client
├─ GET  /v1/conversations/{id}/items?order=asc       [httpx]                Client
├─ invoke_workflow Agent workflow                    [genai.openai_agents]  Internal 24.41s
│  ├─ POST /v1/conversations/{id}/items              [httpx]                Client
│  └─ invoke_agent NBA_Reporter                      [genai.openai_agents]  Internal 23.71s
│     ├─ chat gpt-6-luna                             [genai.openai]         Client    3.94s
│     │  └─ POST /v1/responses                       [httpx]                Client    3.84s
│     ├─ execute_tool calculate_win_percentage       [genai.openai_agents]  Internal
│     ├─ execute_tool calculate_win_percentage       [genai.openai_agents]  Internal
│     ├─ POST /v1/conversations/{id}/items           [httpx]                Client
│     ├─ chat gpt-6-luna                             [genai.openai]         Client   18.21s
│     │  └─ POST /v1/responses                       [httpx]                Client   17.90s
│     └─ POST /v1/conversations/{id}/items           [httpx]                Client
├─ POST /v1/traces/ingest  ×3                         [httpx]                Client
└─ POST /agent/turn http send  ×2                    [asgi]                 Internal

This trace highlights several key structural details of the agentic loop:

1. Full-stack request lifecycle : The trace begins at the FastAPI boundary (POST /agent/turn). Before the agent workflow executes, child HTTP spans show the session layer fetching conversation history (GET /v1/conversations/{id}/items).
2. Dedicated agentic scopes : The agent orchestration harness is captured under opentelemetry.instrumentation.genai.openai_agents via the invoke_workflow and invoke_agent spans, defining the agent's decision boundary.
3. Iterative tool execution loop : Rather than a single model completion, the trace maps the full multi-step loop:
   • Initial model call ( chat gpt-6-luna, 3.94s): The model processes the turn, detects the win percentage query, and emits tool call instructions.
   • Tool execution ( execute_tool calculate_win_percentage ×2): The agent loop executes our tools, followed by an HTTP span persisting the tool results to the conversation session.
   • Final synthesis call ( chat gpt-6-luna, 18.21s): The model receives the tool outputs and generates the comprehensive user-facing response.
4. Standardized GenAI attributes : Selecting either chat gpt-6-luna span reveals attributes for:
   • Provider and model : gen_ai.provider.name, gen_ai.request.model, and gen_ai.response.model.
   • Operation and outcome : gen_ai.operation.name and gen_ai.response.finish_reasons.
   • Token usage : gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

Clarifying the duplicate tool calls

You must have noticed the duplicate calculate_win_percentage tool calls. Several previous runs with the same prompt only triggered a single tool call, as expected given the trivial workload. In this run, the agent loop triggered a second tool call, and its input arguments tell the story: gen_ai.tool.call.arguments: "{"wins":0,"losses":0}".

The tool call was made with technically valid but impractical input and received an equally impractical result: gen_ai.tool.call.result:".000". Although this didn't affect the final result, this further goes to show the non-deterministic nature of AI agents, and underlines why we need observability for AI workloads.

Inspecting message content and privacy controls

Because we set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only, the instrumentation populates gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, and gen_ai.tool.definitions directly as span attributes. This allows engineers to inspect the exact prompts, system instructions, and model completions for each turn during debugging.

In environments with sensitive data or strict PII compliance, prompt content capture can be turned off entirely by setting:

Copy

export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=no_content

This retains model timing, token counts, and span hierarchies while omitting raw message payloads and input/output values from the exported telemetry.

Handling guardrail failures in traces

When an off-topic query triggers the input guardrail tripwire, the agent stops execution before calling the model.

In our FastAPI exception handler, intercepting InputGuardrailTripwireTriggered and invoking span.record_exception(exc) captures the exception event on the root span.

Root spans capture the guardrail triggered exception events.

Monitoring agent performance and metrics

While individual traces let you debug specific agent executions, understanding production behavior requires aggregated metrics across turns, models, and sessions.

The OpenTelemetry GenAI instrumentation automatically emits standard metrics—such as gen_ai.client.operation.duration and token usage counters—alongside your traces. SigNoz Cloud captures this telemetry in a dedicated AI Observability interface that aggregates operational health, latency percentiles, and agent loop behaviors in one unified view.

SigNoz AI Observability correlates latency percentiles, finish reasons, and tool execution metrics across your AI agent runs.

Executive KPIs and contextual drilldowns

The top-level overview surfaces high-level operational indicators across your agent infrastructure:

• Total cost and token volume : Tracks spend and token consumption across services to help identify expensive workloads.
• Latency percentiles and TTFT : Tracks tail latency and Time to First Token (TTFT) for streaming responses.
• LLM call volume and error rates : Monitors traffic spikes and model failure rates across services.

Use the dashboard's trace and log drilldowns to track the corresponding telemetry responsible for the numbers. Break down individual metrics by attributes such as model name or service, then combine those breakdowns with dashboard-level filters to answer questions like:

• Which model cost you the most for a particular service in the given time frame?
• How does LLM latency compare between staging and production over time?
  • Could differences between your team's test requests and customer requests explain the gap?

Evaluating model latency and response times

Multi-turn reasoning and tool selection introduce variable latency across agent interactions. The LLM latency by model table and time-series charts track execution performance across percentiles:

• Percentile breakdowns : Compare p50, p90, and p95 latencies by model to see whether slow responses affect typical requests or only the tail.
• Interactive time-series : Drill into traces around a latency spike to inspect how much time was spent in model calls and tool execution.

The Finish reasons panel shows how model invocations end:

• ["stop"] : Indicates the model finished generating a final response to the user.
• ["tool_calls"] : Indicates the model requested one or more tool executions, prompting the agent loop to execute the tool and start another turn.

Spikes in ["tool_calls"] relative to ["stop"] can be signals of recursive agent behavior, repetitive tool retries, or agents struggling to synthesize an answer.

Tool call performance and span distribution

The Tool calls section aggregates telemetry from the tools invoked by the agent:

• Tool call rate and error tracking : Shows how often each tool runs and fails, helping identify repeated calls or unreliable tools.
• Tool duration : Shows which tools contribute the most latency to an agent run.
• Top span names : Compares agent, model, and tool span counts. Unexpected differences, such as a large gap between agent spans and LLM spans, can guide further trace inspection.

As AI applications evolve from simple model prompts into autonomous, multi-turn agents, traditional debugging approaches fall short. Agent workflows are non-deterministic, span multiple tool calls, and rely on dynamic context that can silently degrade in quality.

Instrumenting your agents with OpenTelemetry delivers end-to-end visibility across every layer:

• Full request context : Correlate incoming HTTP requests, session storage lookups, agent decision loops, and downstream API calls in a single trace.
• Vendor-neutral GenAI standards : Leverage official OpenTelemetry semantic conventions (gen_ai.*) to standardize metrics, attributes, and span structures without proprietary lock-in.
• Unified debugging in SigNoz Cloud : Combine traces, metrics, and logs in one interface to diagnose hallucinated tool calls, identify latency bottlenecks, and govern token spend across production runs.

If you’re building agentic applications, sign up for a free SigNoz Cloud trial to start observing your AI agents with OpenTelemetry today.