Agent 化监控：系统运维与可观测性的新范式
英文标题：🔶 Agentic Monitoring
本文基于 Anthropic 团队使用专注单一组件的 Claude“死忠粉（superfans）”Agent 开展监控的实践，指出由 Agent 主导的系统监控正成为现代软件工程的必然趋势。面对代码量指数级激增而传统可观测性门槛与成本居高不下的现状，作者展示了利用模型针对发布与配置推送等变更点进行自主生成监控规范并低成本排查的实践方案。测试表明，该方案不仅能以极高精度和召回率发现隐蔽回归，甚至能通过自动化分析实现 50% 内存节约与 30% 延迟优化；随着模型成本大幅降低，Agent 化监控有望从被动分析走向主动自愈，重塑未来的 SRE 运维体系。
中文
Anthropic 最近的一篇博客文章中包含了一小段话，虽然它完全不是那篇文章的主旨，但自从读到后就一直在我的脑海中盘旋：
迈向全新水准的卓越运维已成为可能。我们团队大量使用 Claude Tag，它帮助我们监控系统的各个组件。例如，我们创建了若干 Slack 频道，让 Claude 极度专注地盯紧单一组件的健康状况。我们把这些 Claude 称为“死忠粉”（superfans）。我们有 scion-daemon 死忠粉、scion-cache 死忠粉、Antique 合并队列死忠粉等等。它们能够访问我们的遥测数据和日志，表现得就像一位经验丰富的工程师，闲来无事就不断刷新图表、搜寻任何异常。我们对所有系统依然设有传统的告警，但我们的“死忠粉”能够发现微妙且新颖的问题，相比之下让那些传统告警显得维度非常单一。
这显而易见是软件系统运维的前进方向。过去几周我一直在思考用 Agent 进行监控的问题，而这段话让我的一些思考彻底具象化了。
现代软件系统就像编写它们的 LLM 一样，正变得越来越偏向“自然生长”而非“刻意设计”。诚然，人类工程师通常仍然在对系统进行高层设计并勾勒形态；诚然，软件系统（目前而言）比 LLM 具备好得多的可解释性——因为只要你愿意，随时可以坐下来阅读代码。然而，随着编写和交付的代码量呈指数级增长，任何系统的人类运维者所拥有的“实际审视过代码”的知识储备都在不断缩减。
系统监控从来就不容易。市面上有一整个可观测性 SaaS 产业——Sentry、Datadog、Splunk、Honeycomb 等等，不一而足。可观测性价格不菲。做好可观测性需要专业的人类技能集，并依赖于在大规模运行真实系统中所积累的经验。这种技能基本上只能通过在工业界实战中获得。这是一个需要花费数年时间来培养初级工程师的领域。
正如 2026 年软件行业中看似发生的一切一样，由于 Agent 的出现，系统监控正迎来阶跃式的变化。我……很惊讶居然没看到更多人撰文讨论这一点，尤其是在当前“Agent 狂热”的环境下。它就摆在大家眼前，而且切实有效。虽然已经有一大批初创公司在尝试做“AI SRE”，但一线从业者的写作却少之又少。
但你其实并不需要一个完整的“AI SRE”来提升卓越运维水平，也不需要为此购买现成的 SaaS 产品。到了 Opus 5.5 / Astra 级别的模型，只要稍加用心，完全有可能打造出一套“开箱即用（just works）”的优秀 Agent 驱动监控系统。
案例研究：变更点监控
我将把这个讨论落脚于一个我非常关注的问题上：生产环境中针对变更点的监控，比如二进制文件发布和配置推送。其核心思路是，你希望围绕系统变更的关键节点进行专门、密集的监控；这些变更点更容易引发回归，因此如果密切监控这些事件，就能限制其影响面。引用 Google SRE 手册中那句老生常谈的话：“约 70% 的故障都源于在线系统的变更”。
这是一个棘手问题（wicked problem）。我从 2023 年起就以某种形式在着手解决它，却一直难以取得实质突破。通过与行业内各方交流，我了解到有一些能“做好”变更点监控的方法，但这些方法在很大程度上依然依赖人类做出正确的行为（例如勤勉地为指标/告警打标签、定义精准的信号等），并且需要巨大的组织资源投入。
更具体地说，变更点监控作为平台工程的一部分，之所以难以实现“起飞”，是因为：
- 你的第一反应可能是瞄准那些运维习惯良好、积极性极高的用户。然而，这些团队的发布本身通常已经相当安全，你并没有把整个系统的运维基准线提高多少。你在这里兜售的其实是减少苦工（toil reduction），这固然“锦上添花”，但本身并不是一阶维度的安全提升。对于高积极性团队来说，高精度更容易达到，但这意昧着变更点监控系统在召回率和延迟上是在与传统告警“竞争”。
- 另一种方式是面向广泛的服务群体。换言之，覆盖面铺得极广，在选定的一组通用健康指标上让“所有服务”都默认接入。在实践中，绝大多数通用健康指标都带有不可消除的噪音。这种噪音会导致极差的精度和告警疲劳。
- 在 LLM 出现之前，精度与延迟之间也存在明显的权衡。处于这一光谱上的有：结构化日志（精准、缓慢）、指标（粗粒度、快速）以及告警（更加粗粒度，召回率相当差，精度也很差，但“聊胜于无”）。
令人震惊的是，Agent 基本上“直接”解决了这个问题。Agent 乐于执行大量查询，能针对每次变更动态编写查询，并且能不厌其烦地扫描日志。结果就是，你可以用极低成本兼顾优秀的精度与召回率。在这套配置的基础版本中，你只需要能够将代码库（例如理解配置如何被消费）以及可观测性技术栈（例如查看并监控线上生产状态）接入到 Agent 即可。
我已经针对这套流程快速打磨了一个原型：由 Agent 基于服务代码提出监控计划，一套脚手架（scaffold）来执行该计划，再由 Agent 对触发的监控项进行调查，并具备向上升级转交人工的路径。即使在这一早期原型中，我也能在自己毫无了解的系统中检测到回归问题，其精度和延迟在一年之前是绝无可能实现的。
在对此进行迭代的过程中，我发现纯 Agent 循环（pure agent loop）并不是 Agent 化监控的最佳形态。相反，提供一个让 Agent 编写“如何监控”规范（spec）的脚手架工具（harness），搭配一个运行这些规范的确定性系统，能产生清晰得多的信号，同时大幅降低 Token 消耗。如果说这种模式中存在什么“独门秘诀”，那就是降噪启发式规则、引导 Agent 编写优质规范的 Prompt，以及一套足够大、可用来验证进一步调优的带标注评测集（eval set）。
Agent 在结构上非常契合监控场景
在这一模式流行起来之前，需要发生两次观念转变：
- Agent 实际上能比人类更好地完成监控工作。
- 监控 Agent 需要脚手架，正如软件开发生命周期的其余部分必须为了 Coding Agent 进行适应性调整一样。监控领域目前还没有迎来它的“Claude Code 时刻”。
一旦完成这些思维转变，监控 Agent 相比人类运维人员就拥有了几项“非对称优势”：
- 全天候 24/7 追踪一切的能力。如果 Token 很便宜，那么你所关注指标的任何微小回归、任何可疑的日志行都可以被深入调查——而不仅仅是那些以往能越过“人类注意力门槛”的事件。Agent 不需要“日不落轮岗（follow the sun）”。
- 发散并同时调查多种排障路径的能力。当多个问题同时爆发时，Agent 可以并行化展开调查，并迅速进行协同。这已经成为我处理事故排查时的首选方案：直接启动一批 Agent 去调查数条潜在线索或事故原因推测。目前这还是由人类驱动的，但未来完全没有理由不能由 Agent 自主发起。你可以弹性扩容出一支事故响应团队。
- 我们已经看到，这对于安全事件正变得至关重要。Agent 团队/集群/蜂群正在被用于网络攻击，而防御者也开始建立类似的防御能力（例如 OpenAI 的 Daybreak Blue、Anthropic 的 Project Glasswing、Google 的 Fairwind）。
- 比人类更快响应问题的能力。Agent 阅读代码的速度远快于人类。Agent 也能以快得多的速度查询并理解指标和日志信号。这种速度并不总是能转化为更快的事故检测时间——对系统有深刻理解的人类在寻找正确信号和直觉判断根因方面可能仍然更快。但只要这种隐性知识能够被记录下来并提供给 Agent，Agent 最终在速度上必将战胜人类。
- 由此推论，对于人类运维者经验尚浅的系统，Agent 的“理解耗时（time to understanding）”将会显著更短。
“迈向全新水准的卓越运维已成为可能。”
更加勤勉负责：你不会花钱雇一个全职软件工程师去通读一个服务的全部 Git commit，挑出需要监控的地方，并专门为这些变更制作短期的快速触发告警。然而，现在只要每百万 Token 几美分的成本，这不仅已经可行，而且将成为行业标准。
持续提升性能：Agent 可以监控性能，查看堆转储（heap dump）、堆栈轨迹（stack trace）和 CPU 分析（CPU profile），并根据所发现的模式提出优化方案。如有必要，可以将提出的改进置于 A/B 测试中，就像监控对服务的其他变更一样进行监控。
在一个实例中，我观察到几次 Astra/Opus 调查会话就在一个人类已经投入大量精力优化的通用热路径库中发现了 50% 的堆内存节省和 30% 的延迟降低。一旦你给一个足够强大的 LLM 提供一台虚拟机和可运行的基准测试，当前一代 LLM 的“爬山寻优（hill-climbing）”能力是令人惊叹的。
采取行动：到目前为止，我讨论的大多是“只读”监控系统。当然，Agent 并不一定只能是“只读”的。有了合适的脚手架和护栏，未来的 Agent 绝对有能力采取一线行动来排解事故。范围可以涵盖从边界明确的运维任务（如调整实例规格或轮换计算池），到确定性动作（如回滚配置更改或二进制版本），再到开放式任务（如定位 Bug 根因、提出修复补丁并在线重新部署）。显然，你交给 Agent 的控制权越多，风险就越大；但考虑到过去 12 个月内 Agent 能力的扩展速度，可以负责任地移交给 Agent 的写路径工作注定会增加。真正具备自愈能力系统的愿景令人振奋。
成本不再是瓶颈。Luna/Haiku 级别的模型现在已经非常擅长“盯图看表（graph-starers）”。一家规模尚可的科技公司可能只需要每天约 100 美元的成本，就能监控其整个集群。初创公司规模的集群所需成本更是微不足道。相比于人力成本，乃至相比于运行可观测性基础设施本身的开销，这笔费用简直微乎其微。
构建良性循环：如果做得好，Agent 化监控将创造一个正向反馈闭环：Coding Agent 让你能够交付更多代码，监控 Agent 让你能够支撑更多系统，这反过来又释放了人类的时间去指导交付更多工作。人类将越来越多地把时间花在“建造工厂（software factory）”或“打理花园”上。
Agent 化监控值得投资。不久之后，如果不使用 Agent 监控你的系统，将会被视作不负责任的行为。在行业其余部分反应过来之前，现在动手起步将为你赢得不可多得的非对称优势。
English / 原文
A recent Anthropic blog post contained a short paragraph, not at all the point of the post, which has been buzzing around my brain since I read it:
A new level of operational excellence is possible. Our team uses Claude Tag heavily, which has helped us monitor each component of the system. For example, we created Slack channels where we tasked Claude with being obsessively focused on the health of a single component. We call these Claudes “superfans.” We have a scion-daemon superfan, a scion-cache superfan, an Antique merge queue superfan, and so on. They have access to our telemetry and logs, and act like an experienced engineer who has nothing better to do than to refresh charts and look for anything strange. We also have traditional alerts for all of our systems, but our superfans can detect subtle and novel issues that make these alerts seem one-dimensional by comparison.
This is so obviously the path forward for software systems operations. I’ve been thinking about monitoring with agents for the past few weeks, and this paragraph crystallized some of those thoughts.
Modern software systems, like the LLMs that write them, are becoming more “grown” than “designed”. Yes, human engineers often still give the high-level design and shape to the system, and yes, software systems (for now) are much more interpretable than LLMs – since you can just sit down and read the code, if you are so inclined. However, with the volume of code being written and shipped increasing exponentially, the amount of actual eyes-on-code knowledge that any human operator of a system has is decreasing.
Systems monitoring was never easy. There is a whole industry of observability SaaS companies – Sentry, Datadog, Splunk, Honeycomb, to name a few. Observability is expensive. Doing it well requires a specialized human skill set and relies on building experience operating real systems at scale. One can really only develop these skills by working in industry. This is a domain that takes years to train junior engineers on.
As with seemingly everything in the software industry in 2026, system monitoring is poised for a step change due to agents. I… am surprised I have not seen more people write about this, especially in the current milieu of “agent frenzy”. It’s right there in front of us all, and it works. There have been a bunch of startups trying to do the “AI SRE”, but very little writing from practitioners.
But you don’t really need a full “AI SRE” to improve operational excellence, nor do you need a built-out SaaS product to do so. As of the Opus 5.5 / Astra class of models, with a bit of care, it’s entirely possible to make a good agent-driven monitoring system that “just works”.
Case study: change-point monitoring
I’ll ground this discussion in a problem that is near to my heart: production monitoring of change points, like binary releases and config pushes. The idea is that you want to do specialized, intensive monitoring around critical points of change to a system; these change points are more likely to trigger a regression, and thus if you monitor these events closely, you can limit its impact. To quote the clichéd line from the Google SRE book, “~70% of outages are due to changes in a live system”.
This is a wicked problem. I’ve been working on it in some form since 2023, and it’s been hard to get traction on. From talking with folks across the industry, I’ve learned there are ways to do change-point monitoring “well”, but these methods largely still rely on humans to do the right thing (e.g. tagging metrics/alerts diligently, defining precise signals, etc.) and require significant organizational investment.
More specifically, achieving “takeoff” for change-point monitoring as a piece of platform engineering is hard because:
- Your first instinct may be to target highly motivated users who already have a good ops posture. However, these teams’ rollouts are already mostly safe, and you don’t actually raise the ops waterline much. The thing you’re selling here is toil reduction , which is “nice to have” but is not itself a first-order safety improvement. Precision is easier to achieve for highly motivated teams, but this means the change-point system is “competing” on recall and latency with traditional alerts.
- You can alternatively target the broad population of services. In other words, you go super wide and opt in “all services”, on a chosen subset of universal health metrics. In practice, most of these general health metrics are irreducibly noisy. This noise results in poor precision and alert fatigue.
- Prior to LLMs, there was a clear precision/latency tradeoff as well. Along this spectrum are: structured logs (precise, slow), metrics (coarse, fast), and alerts (even coarser, recall is pretty bad and precision is pretty bad, but “at least it’s something”).
Shockingly, agents largely “just” solve this problem. Agents are happy to run copious queries, can write per-change queries on the fly, and will scan logs ad nauseam. As a result, you can get both good precision and recall for cheap. In the basic version of this setup, you just need the ability to connect your codebase (e.g. to reason about config consumption) and your observability stack (e.g. to view and monitor the live production state) to the agent.
I’ve spiked a prototype of exactly this: an agent that proposes a monitoring plan based on the service code, a scaffold that executes that plan, and an agent that investigates triggered monitors, with an escalation path to humans. Even in this early prototype, I’ve been able to detect regressions in systems I have zero understanding of with precision and latency that would have been impossible a year ago.
In my iteration on this, I’ve found that a pure agent loop is not the optimal shape for agentic monitors. Rather, a harness that allows the agent to write a spec of how to monitor, along with a deterministic system for running those specs, produces significantly cleaner signals with dramatically lower token usage. To the extent there is “secret sauce” in this pattern, it’s in the noise-reduction heuristics, the prompts that get the agent to write good specs, and a labeled eval set large enough to validate further tweaks.
Agents are structurally well suited to monitoring
Two mindset shifts need to happen before this pattern can catch on:
- Agents actually can do a much better job at monitoring than humans.
- Monitoring agents need scaffolding, the same way the rest of the software development lifecycle had to adapt for coding agents. There hasn’t been a “Claude Code moment” for monitoring yet.
Once you make these shifts, monitoring agents have several “unfair advantages” over human operators:
- Ability to track everything, 24/7. If tokens are cheap, then any regression in a metric you care about and any suspicious log line can be investigated – not just the events that would previously have risen above the “human attention cutline”. Agents don’t need to “follow the sun”.
- Ability to fan out and investigate several paths toward resolving a production problem. Agents can parallelize investigations when multiple things go wrong at once, and coordinate quickly. This is already my go-to for incident triage: I just launch a bunch of agents to investigate several potential threads or theorized incident causes. Right now, this is human-driven, but there’s really no reason it couldn’t be agent-initiated in the future. You can elastically scale up a fleet of incident responders.
- We’re already seeing that this is becoming critical for security incidents. Teams/fleets/swarms of agents are being used for offensive cyberattacks, and defenders are beginning to build up similar defensive capabilities (e.g. OpenAI’s Daybreak Blue, Anthropic’s Project Glasswing, Google’s Fairwind).
- Ability to respond faster to a problem than can a human. Agents can read code faster than humans. Agents can also query and digest metrics and log signals significantly faster. This speed doesn’t always translate into faster incident detection time; a human with a deep understanding of a system may still be faster at finding the right signal and intuiting the root causes. But to the extent that tacit knowledge can be recorded and made available to an agent, the agent will eventually beat the human on speed.
- By extension, for systems where the human operator isn’t deeply experienced, the “time to understanding” is going to be shorter for an agent.
“A new level of operational excellence is possible.”
Be more diligent: You wouldn’t pay a full-time software engineer to look through all the Git commits for a service, pick out things to monitor, and make special short-term fast-trigger alerts just on those changes. However, for a couple cents per million tokens, it’s possible now, and will become an industry standard.
Continuously improve performance: Agents can monitor performance, look at heap dumps and stack traces and CPU profiles, and propose optimizations from the patterns they find. Put the proposed improvements behind A/B tests if necessary, and monitor as you would any other change to the service.
In one instance, I observed that a few sessions of Astra/Opus investigation discovered 50% heap savings and 30% latency reduction in a common hot-path library that humans had already spent nontrivial effort to optimize. Once you give a sufficiently powerful LLM a VM and benchmarks to run, the “hill-climbing” ability of current-generation LLMs is remarkable.
Take action : Thus far, I’ve mostly been discussing “read-only” monitoring systems. Agents, of course, needn’t be “read-only”. With the right harness and guardrails in place, there’s definitely a future in which agents will be able to take front-line actions to remediate incidents. This could span anything from well-scoped operational tasks like resizing instances or rotating compute pools, to deterministic actions like rolling back a config change or binary, to open-ended tasks like root-causing bugs, proposing fixes, and redeploying on-the-fly. Obviously there is risk the more control you hand-over to the agent, but considering how much agent capabilities expanded over the course of the past 12 months, the amount of write-path work that can be responsibly handed over to agents is bound to increase. The promise of truly self-healing systems is exciting.
Cost is no longer a limitation. Luna/Haiku-class models are now competent graph-starers. A moderately large tech company could likely monitor its entire fleet for on the order of $100/day. Startup-scale fleets would be a small fraction of that. Compared to human labor, or even the cost of the infra that runs observability, this is trivial.
Create a virtuous cycle: If done well, agentic monitoring creates a positive feedback loop: coding agents allow you to ship more, monitoring agents allow you to support more systems, which in turn frees up human time to direct more things to ship. Humans increasingly spend time “building the factory” or “tending the garden”.
Agentic monitoring is worth investing in. It will soon be reckless not to have agents monitoring your systems. Until the rest of the industry catches up, getting started now affords an unfair advantage.