Claude 值班：Claude Tag 如何成为 Anthropic CI/CD 故障的首位响应者
英文标题：Claude on call: How Claude Tag serves as Anthropic’s first responder for CI/CD failures
Anthropic 表示，Claude Tag 已在过去数月中担任其 CI/CD 故障的值班首位响应者，在近期每起有态势报告的事故中均撰写首份报告，通常在 15 分钟内发布初步分析。该系统通过 Slack、MCP Connectors、GitHub 技能文件和 `lessons.md` 记忆库，实现告警检测、并行排障、修复建议、验证与交接。Claude 在事故开启后的首份基于证据分析报告的中位产出时间为 14 分钟，最快可在 4 分钟内指出根因。Anthropic 还发布了开源 on-call setup kit，帮助开发团队将自身事故历史转化为分诊手册，并在事故频道中部署可诊断、升级和持续学习的 Claude。
10.5 KB
100.2 KB
341.2 KB
119.2 KB
103.7 KB
32.0 KB
403 B
11.6 KB
13.4 KB
12.5 KB
10.2 KB
47.7 KB
中文
使用我们的配置工具包，搭建你自己的 Claude 值班系统 。
几周前，我正在值班，同事在晚上 10 点通过 Slack 给我发消息：一个新服务中大约有 44 个测试没有触发。
过去，我会放下手头的事，坐到笔记本电脑前，无奈地叹口气，然后开始长达一小时的调查和修复过程。但现在，我的工作流完全不同了：我会拉入 @Claude，问问它看到了什么。
在这个案例中，Claude 发现这些测试是在当天早些时候某个功能开关被打开后消失的，并且确认可以安全地回滚。我让同事回滚该开关。3 分钟后，Claude 在 Slack 上提醒我，确认跳过规则确实已经移除，错误率也回到了基线水平。
过去数月，Claude Tag 一直是 Anthropic CI/CD 故障的值班首位响应者。这不仅改善了我们的社交生活，也让每起 CI 事故都能立即获得首位响应者：在近期每起拥有态势报告的事故中，Claude 都撰写了第一份情况报告，通常会在 15 分钟内发布首份分析。
本文将介绍我们构建了什么、它如何工作，以便你也能自行搭建，不再惧怕轮到自己值班。
我们的 Claude 值班配置
在逐一介绍事故响应流程的各阶段之前，我先概述一下我们的整体配置，以便你在了解后续细节时掌握全局。
一个值班 Agent 需要具备记忆，以记住已经完成的工作；需要连接和访问权限，以便调查、理解并采取行动；需要日程安排，以便知道何时恢复工作；还需要指令，以便知道该做什么。
Claude Tag 是我们值班 Agent 的核心。Claude Tag 在值班 Slack 频道中保存记忆，并提供在事故期间按轮次下达指令的界面。Claude 还会实时响应值班频道及其他频道中的事件。例行任务的调度，即 Claude 定期执行的操作，也在这个频道中通过自然语言提示完成，例如“每周一美国东部时间上午 9:00 运行 CI 交接”。
Claude Tag 拥有自己的服务账号，并可访问 Anthropic CI 工程师所需的工具，例如 Datadog 或 Grafana。管理员只需为该频道一次性完成此配置（具体方法见此处）。
除了值班频道外，我们还将 Claude 配置为监控其他相关频道——这些频道同样已将 Claude Tag 添加为成员——以便它获取服务告警、配置变更或 PR 更新等额外上下文。
常驻指令以技能形式存储在 Markdown 文件中，并提交到 GitHub 仓库。这样，多名团队成员可以共同迭代这些指令，我们也能像管理代码一样管理变更。其中还包含路由指令、策略，以及作为自我改进闭环一部分的经验教训日志等关键信息。
这套配置耗时是数小时，而非数天。我们在 GitHub 上创建了通用的值班配置工具包，可帮助你开始搭建类似的 Agent。它会将团队自身的事故历史转化为分诊手册，并最终让你在事故频道中拥有一个只读 Claude，用于诊断、升级和学习。你可以查看它如何基于一个虚构团队的历史运行，整个过程约需 10 分钟。
用 TL;DR 方式总结，步骤如下：
- 你需要订阅 Claude Team 或 Claude Enterprise 计划。
- 组织所有者需要通过 Claude Tag 将 Claude 添加到值班 Slack 频道。
- 组织所有者还需要协助将值班 Slack 频道中的 Claude 连接至适当的连接器、GitHub 仓库，并配置 Claude Code Remote。
- 将 Claude 添加到你的事故频道，并指示它监控事故并立即进行分诊。
现在，让我们深入了解这项转变在事故每个阶段的具体表现。
检测
Claude 不仅改变了你响应事故的方式，也改变了你最初发现事故的方式。此前，事故检测主要存在两种失败模式。
人类很难始终具备前瞻性，设置完美的规则和完美的阈值。当你没有足够数据分析流量模式时，这尤其困难。
为此，在新服务上线的最初几天，我们让 Claude 分析数据和收到的告警，以建议增加规则，并微调过宽或过窄的规则。
事故检测的第二种主要失败模式是告警疲劳：检查和审核每一条触发的告警非常枯燥。然而，Claude 不会像人类一样感到疲劳。
Claude 会监控每个告警频道中的所有相关告警，并根据 根 oncall.md 文件中的标准，判断某个问题能否等到早晨处理，还是需要呼叫值班人员。例如，在完成数据分析和调优后，该文件中的一条规则可以是：“如果错误率超过 2% 的时间长于 5 分钟，且当前并非已知部署窗口，则呼叫值班人员；否则将其写入 lessons.md。”
Claude 值班告警流程还可通过另外两种方式触发：
- CI 团队成员可以在值班频道中报告问题，开篇提到的 44 个测试缺失便属于这种情况；或者
- 公司内任何人都可以通过内部页面发起事故。如果其被标记为 CI 基础设施事故，系统便会为该事故创建一个 Slack 频道，我们的值班 Claude 会接手处理。
这里的关键结论是：告警流程是确定性的，而值班升级同时包含确定性路径和 Agentic 路径。
分诊
让 Claude 过滤告警噪声是一回事，但真正节省时间的是调查环节。事故开启后，Claude 发布首份基于证据的分析报告所需时间的中位数为 14 分钟；在最快的案例中，它会在首份报告中于 4 分钟内指出根因。
当告警被升级为事故时，Claude 往往已经在我们的 Slack 频道中准备好了可供审阅的、基于证据的假设。Claude Tag 会启动一个动态工作流，由编排 Agent 启动执行子 Agent，对每个依赖项和事实来源进行调查。
对我们而言，这些来源包括 Grafana、日志存储、PagerDuty、GitHub、Kubernetes 和 Slack 事故频道，全部通过 MCP Connectors 接入。Claude 可以并行追踪多条线索，从而帮助降低 MTTR（平均修复时间）。
执行 Agent 会将调查结果回报给编排 Agent，后者会综合信息，并以连贯的 SITREP（情况报告）形式呈现。
编排 Agent 和执行 Agent 并非盲目搜索。它们由调查技能引导，并可查阅针对每类 Bug 的更详细参考 Markdown 文件。
例如，一个用于影子差异 Bug 的 617 行调查技能，编码了我在典型调查中采取的每个步骤。我是在一次事故中与 Claude 逐轮排障时构建它的，之后让它根据这段经历创建了该文件。
Lessons.md 同样会引导 Claude 进行排障。这个 Markdown 文件持续记录我们已经解决的每起事故：发生了什么、根因是什么、如何修复，以及值得记住的陷阱。Claude 会自行自动向其中追加内容。每次新的调查都会从阅读它开始，因此 Claude 的第一个假设会从近期发生过的事情出发。
如果同一种模式反复出现足够多次，我们就会将其提升为调查技能本身。我最喜欢的一条记录是 Claude 写给我的。我曾在检查指标前就根据配置文件作出假设，而 lessons.md 文件现在写道：“先查询数据，再提出理论。配置告诉你可能出什么问题；指标告诉你实际出了什么问题。”
即使拥有这些工具和上下文，Claude 也并非总能第一次就判断正确。人类的直觉和经验依然重要。Claude Tag 让团队能够以多人协作模式排查事故。我们中的任何人都可以实时引导调查，或共同添加新的假设。
解决
如果 Claude 可以升级和排查告警，它也能修复告警吗？这个问题的答案因团队而异，但以下是我们的做法。
我们团队的大多数部署都在功能开关之后进行。我在 Claude Code 中创建了一个独立 Agent，它拥有我的权限，能够在这些功能开关之后执行渐进式部署。
我们的发布流程第一阶段通常由 Claude 管理金丝雀流量、监控问题，并自动上调或下调某个功能开关。这个主题完全可以单独写一篇文章，因此这里不再展开。
Claude Tag 还能通过以下其他解决路径帮助我们的团队：
- 告知我们是否需要排空或隔离 Kubernetes 集群的某些部分；
- 针对需求激增，提供扩展部分基础设施的操作指引（这种情况很少见，但当 Claude 明确给出可用于缓解问题的措施时非常有帮助）；以及最常见的，
- 以 PR 形式提供修复方案，值班人员可审查、合并并部署，以快速解决问题。
验证、沟通与交接
Claude 使用与调查阶段相同的许多 MCP Connectors 和工具，验证修复是否如预期生效。作为 oncall.md 中常驻指令的一部分，它会将事后复盘写入 lessons.md，并生成交接 SITREP。
为了跨多起事故传达完整情况，我们创建了一个名为 ci-weather 的 Agent。它汇总每个事故 Slack 频道的信息、构建指标、合并队列统计数据和部署延迟，随后在公司内任何人都可阅读的一个公共频道发布新闻编辑室风格的报告。现在，当工程师需要判断是否应暂缓合并，或者试图回答“CI 出了什么问题？”时，可以查阅该频道，而不必询问我们。
坦诚地说，我们需要多次迭代报告格式。Claude 可以一次性生成一个用于产出状态报告的技能，但真正让报告易读的是团队特有的偏好。这是人类沟通，而不是管道工程。
最后，Claude 虽然会在 lessons.md 中为自己保留日志，但我们也希望每周一为人类产出交接报告。Claude 会生成每日和每周摘要，使一名团队成员能够从另一名成员停下的地方继续工作。
从监控事故到监控事故响应系统
我们的软件工程师平均每季度交付的代码量，是其在 2021 年至 2025 年间的8 倍。尽管我们仍保持很高的质量标准——每个 PR 都有指定的人类负责人，每项变更都需要获批才能合并，每项变更都需通过同一组 CI 关卡——但要跟上 Agentic 编码，唯一的方式就是采用 Agentic CI。
Claude 吸收了我工作中繁琐的部分、非工作时间的打扰和事故沟通工作，使我能够专注于真正能推动系统可靠性提升的中长期架构变更。
我们构建的系统最好的部分在于，它并不显得零散。我们的值班流程仍在 Slack 中，但现在 Claude 已加入这个频道。
如何开始：
- 你需要订阅 Claude Team 或 Claude Enterprise 计划。
- 组织所有者需要通过 Claude Tag 将 Claude 添加到值班 Slack 频道。
- 组织所有者还需要协助将值班 Slack 频道中的 Claude 连接至适当的连接器、GitHub 仓库，并配置 Claude Code Remote。
- 将 Claude 添加到你的事故频道，并指示它监控事故并立即进行分诊。
使用我们的配置工具包，搭建你自己的 Claude 值班系统 。
本文由 Anthropic 技术团队成员 Sachin Malhotra 撰写，Anthropic 员工 Michael Segner 亦有贡献。
English / 原文
Set up your own Claude on-call with our setup kit .
A few weeks ago, I was on-call and my colleague Slacked me a message at 10pm: roughly 44 tests on a new service weren’t firing.
In the past, I would have stopped what I was doing, sat down with my laptop, sighed wearily, and began an hour-long investigate-and-fix process. But now, my workflow is entirely different: I pull in @Claude, and ask what it sees.
In this case, Claude found the tests disappeared when a feature flag got turned on that morning, and also that it would be safe to revert. I asked my colleague to revert the flag. Claude pinged me on Slack 3 minutes later to verify the skip rules had indeed been removed and the error rate was back to baseline.
For the last several months Claude Tag has been the on-call first responder for CI/CD failures at Anthropic. Not only has this helped with our social lives, it has given every CI incident an instant first responder: Claude authored the first situation report in every recent incident that had one, typically publishing its first analysis within 15 minutes.
In this article we’ll walk through what we built and how it works so you can build it yourself and stop dreading your turn in the rotation.
Our Claude on call setup
Before we go into each stage of the incident response process, I’ll provide a general overview of our setup here so you have the big picture in mind as we fill in the details.
An on-call agent needs memory so it remembers what’s been done; connections and access so it can investigate, understand, and act; schedules so it knows when to get back to work; and instructions so it knows what to do.
Claude Tag is the backbone of our on-call agent. Claude Tag holds memory across our on-call Slack channel and the interface to provide per-turn instructions during an incident. Claude also acts in real time to events in the on-call channel and others. The scheduling of routines, or the regular actions Claude takes, happens on this channel as well with natural language prompts like “run CI handoff every Monday at 9:00am EST.”
Claude Tag has its own service account and access to the tools an Anthropic CI engineer needs such as Datadog or Grafana. This was set up one time by an administrator for the channel (here’s how).
In addition to the on-call channel, we set up Claude to watch other relevant channels that also have Claude Tag as a member so it can get additional context like service alerts, configuration changes, or updates on PRs.
Standing instructions are in markdown files as skills, committed in a GitHub repository. This way multiple teammates can iterate on them and we can manage changes just like we do code. It also includes key information like routing instructions, policies, and a log of lessons learned as part of a self-improvement loop.
This setup took us hours, not days. We created a generalized on-call setup kit in GitHub that can help get you started with a similar agent.**** It transforms your team's own incident history into triage playbooks and leaves you with a read-only Claude in your incident channel that diagnoses, escalates, and learns. You can watch it run against a fictional team's history in about ten minutes.
To summarize the steps TL;DR fashion
- You’ll need a Claude Team or Claude Enterprise plan
- The organization owner needs to add Claude to the on call Slack channel via Claude Tag
- The org owner also needs to help connect Claude in the on-call Slack channel to the appropriate connectors, GitHub repo, and set up Claude Code Remote.
- Add Claude to your incident channel and instruct it to monitor for incidents and immediately triage
Now, let’s dive into the details of what this transformation looks like at each step of an incident.
Detection
Claude doesn’t just transform how you respond to incidents, it transforms how you detect them in the first place. Previously, there were two major failure modes for detecting incidents.
It's hard for humans to have the foresight to set perfect rules with perfect thresholds all the time. It's especially difficult when you don't have enough data to analyze traffic patterns.
To address this, we have Claude analyze the data and incoming alerts for the first few days of a new service to suggest additional rules and to fine-tune any that are overly broad or narrow.
The second major failure mode for detecting incidents was alert fatigue: checking and vetting every alert that fires is tedious. However, Claude doesn’t get fatigued the same way a human does.
Claude monitors every relevant alert in each alert channel and goes through the criteria in the root oncall.md file to determine if it can wait until the morning or if the on-call needs a page. For example, once tuned from analyzing the data, a rule in the file could be, “If the error rate is greater than 2% for longer than 5 minutes AND it's not a known deploy window, page the on-call otherwise write it to lessons.md.”
There are two other ways the Claude on-call alert process can trigger:
- A member of the CI team can report an issue in the on-call channel, as was the case in the opening example of 44 missing tests; or
- Anyone in the company can open an incident through an internal page. If it’s marked as a CI infrastructure incident then a Slack channel is provisioned for that incident and our on-call Claude picks it up.
The key takeaway here is that the alerting process is deterministic, while on-call escalation has both deterministic and agentic paths.
Triage
It's one thing to have Claude filter through the alert noise, but the real savings comes from the investigation. Claude posts its first evidence-grounded analysis a median of 14 minutes after an incident opens, and in the fastest cases names the root cause within 4 minutes in its first report.
When an alert has been escalated to an incident, Claude is often ready in our Slack channel with a hypothesis grounded in evidence that we can review. Claude Tag kicks off a dynamic workflow with an orchestration agent that spins up executor subagents to investigate each dependency and source of truth.
For us that’s Grafana, our log store, PagerDuty, GitHub, Kubernetes and Slack incident channels–all wired up via MCP Connectors. Claude can chase multiple leads in parallel, helping to reduce MTTR (mean time to resolution).
Executors report the findings back to the orchestration agent which synthesizes and surfaces the information in a coherent SITREP.
The orchestrator and executor agents aren’t searching blind. They are guided by an investigation skill with more detailed reference markdown files for each bug class.
For example, a 617 line investigation skill for shadow divergence bugs encodes every step I take during a typical investigation. I built it by troubleshooting with Claude turn-by-turn during one of the incidents and then had it create the file from that experience.
Lessons.md also guides Claude’s troubleshooting. This markdown file is a running log of every incident we've resolved: what happened, the root cause, the fix, and the gotcha worth remembering. Claude appends to it on its own automatically. Every new investigation starts by reading it, so Claude's first hypothesis starts with what has happened recently.
If the same pattern shows up enough times, we promote it into the investigation skill itself. My favorite entry is one Claude wrote about me. I'd made an assumption from a config file before checking the metrics, and the lessons.md file now states, "query the data first, then theorize. Config tells you what could go wrong; metrics tell you what did."
Even with these tools and context, Claude doesn’t always get it right the first time. Human intuition and experience matter. Claude Tag allows the team to troubleshoot incidents in multi-player mode. Either of us can steer the investigation or add a hypothesis in real-time, together.
Resolution
If Claude can escalate and troubleshoot alerts, can it fix them too? The answer to this question will vary from team to team, but here’s how we do it.
Most deployments within our team happen behind a feature flag. I have created a separate agent in Claude Code, with my permissions, capable of progressive deployment behind each of these feature flags.
The first stage of our rollout process usually involves Claude managing canary traffic, monitoring for issues, and automatically ramping a given feature flag up or down. This could be an entirely separate article, so I won't go into more detail here.
Other resolution paths that Claude Tag helps my team with are:
- Letting us know if we need to drain or cordon off certain sections of our Kubernetes cluster;.
- Giving us instructions on how to scale up some of our infrastructure in responses to demand-surges (this is rare but it’s very helpful when Claude comes back with exactly what we can do for mitigation); and, most frequently,
- Fixes in the form of a PR that the on-call can review, merge, and then deploy for a swift resolution.
Verification, communication, and handoff
Claude uses many of the same MCP Connectors and tools that it did for its investigation to verify the fix worked as intended. As part of the standing instructions in oncall.md, it writes a post-mortem to lessons.md and for the handoff SITREP.
To communicate the full picture across multiple incidents, we created an agent called ci-weather. It compiles information from each incident Slack channel, build metrics, merge queue stats, and deploy lag. Then it posts a newsroom-style report to one public channel anyone in the company can read. Now, our engineers can reference that channel rather than pinging us when they are trying to determine if they should hold their merges or if they’re trying to answer “what’s wrong with CI?”.
One honest note: we needed to iterate the report format several times. Claude can one-shot a skill that generates a status report, but what makes it readable is team-specific taste. It's human communication, not plumbing.
Finally, while Claude keeps a journal for itself in lessons.md, we also want to produce handoff reports for humans as well every Monday. Claude produces daily and weekly summaries so one member of the team can pick up where the other left off.
From monitoring incidents to monitoring an incident response system
Our software engineers on average ship 8x as much code per quarter as they did from 2021 to 2025. And while we have kept the quality bar high (every PR has a named human owner, every change requires approval to merge, every change goes through the same set of CI gates), the only way to keep up with agentic coding is agentic CI.
Claude has absorbed the tedious parts of my job, the after-hours disruptions and the incident comms, while allowing me to focus on the medium and long term architectural changes that truly move the needle for system reliability.
The best part of what we have built is that it doesn’t feel scattered. Our on-call processes live in Slack, but now Claude has joined the channel.
How to get started:
- You’ll need a Claude Team or Claude Enterprise plan
- The organization owner needs to add Claude to the on call Slack channel via Claude Tag
- The org owner also needs to help connect Claude in the on-call Slack channel to the appropriate connectors, GitHub repo, and set up Claude Code Remote.
- Add Claude to your incident channel and instruct it to monitor for incidents and immediately triage
Set up your own Claude on-call with our setup kit .
This article was written by Sachin Malhotra, technical member of Anthropic staff with contributions from Michael Segner, Anthropic staff.