透视前沿实验室内部：Anthropic 提出衡量 AI 研发步伐与自主化程度的三大指标

AI 系统的能力正呈指数级增长，并已开始在自我构建过程中自动化更多环节。在全世界都在考虑放缓前沿 AI 的研发节奏之际，公众需要了解更多信息。

在本文中，我们提出了能够阐明 AI 研发三大关键维度的度量工具：

1. AI 在多大程度上构建其下一代版本，而非由人类构建
2. 我们监督并干预 AI Agent 在 Anthropic 系统中所采取操作的能力
3. 支撑更强大模型研发所投入的资源

我们还展示了 Anthropic 内部这些指标的一份快照。值得注意的是，如果像 Anthropic 首席执行官 Dario Amodei 所呼吁的那样，在把控前沿节奏上达成协同，我们预计这些数据将会发生变化。我们计划在 Anthropic 引入来自多个机构的独立第三方评估员，并授予他们访问内部流程、系统和数据的权限，其权限与内部风险评估团队相当。这些第三方机构将验证安全实践、报告安全事件，并监测本文中提到的关键指标。

我们公布这些度量指标，是为了让公众、第三方和政府能够更清晰地了解前沿实验室内部 AI 研发的速度。对于每项指标，我们都阐述了我们度量了什么、度量结果显示了什么，以及要以他人可验证的形式定期发布这些指标需要具备哪些条件。我们在附录中分享了方法论的详细信息。

追踪这些度量指标的原因

本文中的度量指标侧重于模型是如何构建的。通过更好地理解模型的生产流程，我们更有可能将模型输入（例如算力）与模型输出（例如能力）建立关联。它们是对能力评估的补充，而能力评估测量的是模型能做什么。我们通过责任扩展政策（Responsible Scaling Policy, RSP）风险报告单独发布能力评估，其中包含了我们的模型在多大程度上加速了 AI 研发的证据。在我们关于前沿 AI 的政策提案《高级 AI 框架 (AAIF)》中，我们提出了任何实验室发布安全模型所应遵循的行业规则，包括政府可能要求的透明度义务（如风险报告）。综合起来，这些提议的度量指标和政策是从实验室外部监督 AI 发展速度的一个起点。

(1) 衡量由 AI 主导的 AI 研发

为什么要衡量 AI 主导的研发？ 前沿 AI 实验室越来越倾向于使用 AI 来构建未来的 AI 模型。该流程使民主国家的实验室能够更迅速地开发出更强大的模型，并在模型发布前进行更多的安全和测试工作，从而在保持前沿地位的同时确保 AI 的益处。然而，模型加速自身研发可能会让人类更难理解或控制这些系统。因此，公布这些指标对于了解世界距离实现递归自我改进（即模型完全自主地构建其后继模型）有多近至关重要。

我们度量了什么。 我们构建了一个原型指数，用于衡量 Anthropic 的 AI 研发工作中有多少是由 Claude 完成的，该指数被称为“Anthropic 研发自动化指数”（Anthropic R&D Automation Index）。它的构建方式是对公司内部进行的各类 AI 研发工作进行分类编目，评估当前每项任务的自动化程度，并汇总这些评分。

我们的发现。 为了衡量 AI 在 Anthropic 内部进行 AI 研发的程度，我们采用了由 Epoch AI 开发的自动化分级标准——“自动化等级”（Automation Level，简称 AL）。该标准从 AL0（无 AI 参与）到 AL5（AI 完全自主运行，无人类介入）。在 AL3 中，AI 处于“协同”（collaborates）状态：它可以在人类的密切指导下完成大部分工作。在 AL4 中，AI 处于“主导”（leads）状态：它能够根据高层提示端到端完成大部分任务，而人类仅负责监督¹。

截至 2026 年 8 月：

• 在所有被测量的 AI 研发工作子集中，Claude 均未处于完全自主运行状态。
• Claude “主导”了 Anthropic 26% 的 AI 研发工作。
• 达到或超过“AI 协同”水平的工作份额超过了 90%。

任何 AI 开发者今天都可以报告的内容。 任何前沿开发者都可以使用公开的方法论定期发布这些度量数据。这将使数据能够随着时间的推移进行纵向对比，并可能实现跨实验室的横向对比。

在这类报告的跨实验室对比中存在两个障碍：首先是缺乏统一通用的方法论；其次，我们使用自己的模型来评估自己的系统，这可能意味着“裁判”模型可能会犯与受检模型相同类型的错误。为了应对这些挑战，开发者的度量数据可以由第三方或其他开发者的模型进行验证，同时设置好防护栏以限制竞争性敏感数据的交换。这些度量标准也可以作为触发更严格要求的阈值，例如在将新模型用于进一步的 AI 研发之前设置固定的测试窗口期。

(2) 衡量对 AI Agent 的监督

为什么要衡量对 Agent 的监督？ 与其他前沿开发者一样，Anthropic 的员工越来越多地将任务委托给 Agent，这些 Agent 可以半自主地长时间工作，并相互委派任务。随着工作从“AI 协同”日益迈向“AI 主导”，Agent 可能会做出更具实质性影响的决定，例如接下来开展哪个研究方向。

这些决策需要安全防范措施来阻断单个 Agent 的有害行为，同时也需要监管机制来发现可能只在多个 Agent 间协同出现的问题。在我们迄今为止的监控数据中，单个 Agent 极少出现不当行为。但当经济体中有数百万甚至数十亿个 Agent 运行时，即使是极罕见的事件也会频繁发生。

我们度量了什么。 我们构建了一套系统，使我们能够监督并干预 AI Agent 在 Anthropic 系统中所采取的操作。在这里，我们考虑三个不同的指标：覆盖率（coverage），描述在执行前或执行后通过监控器的 Agent 操作比例；审查延迟（review latency），即某项操作发生到被审查之间的时间差，首先由自动化监控器审查，随后由人工审查；以及上报率（escalation rate），即被拦截/重定向（针对在线监控器）或被标记以供进一步审查（针对离线监控器）的 Agent 活动比例。

我们的发现。 截至 2026 年 8 月，在我们最常用的内部平台上，任何时刻都有大约 30,000 个 Agent 在 Anthropic 进行研究和工程工作。这些度量仅覆盖该平台。这些 Agent 的操作受到两类监控器的约束，总结如下：

	在线监控器 (Online monitors)	离线监控器 (Offline monitors)
描述	在线监控器实时检查 Agent 的操作，并自动拦截危险操作或纠正 Agent 的不良行为。它们旨在应对发生迅速且无法逆转的危害，例如 Agent 将模型权重复制到我们的系统外部。	离线监控器在事后读取 Agent 的活动记录。它们旨在帮助应对发展缓慢且可逆转的危险行为，例如模型未对齐的迹象。我们利用被标记的活动来改进缓解措施，例如分类器和确定性控制机制。如果情况引发了重大的紧迫危险，我们将关闭该 Agent 部署中的危险部分。
覆盖率	这些 Agent 100% 的操作在执行前都会通过在线监控器，通常在数秒之内完成。	这些 Agent 采取的所有操作 100% 会在事后被监控器采集。
审查延迟	自动化审查在 Agent 采取行动前发生；对于任何被拦截的操作，人工会在一周内完成复核。	监控器每周大约标记 100,000 份交互记录（transcripts），随后由多个分类器进行审查，以排除误报并完成初步监管审查。大部分标记在此过程中得到解决，优先级最高的标记（每周约 ~50 起）将上报至人工复核。
拦截 / 上报率	我们分析了研发与工程 Agent 在 2026 年 8 月做出的超过 10 亿次决策，发现其中有 0.002%（约每 47,000 次中有 1 次）被该监控器拦截。	监控器大约会对每一千份交互记录中的一到两份进行标记，以作进一步审查。

AI 开发者今天可以报告的内容。 任何在其自身研究和工程工作负载上运行 Agent 的开发者都可以公布相同的指标：覆盖率（受监控的 Agent 活动比例）、审查延迟（被标记的活动多快能得到复核）以及上报率（监控器拦截或标记的 Agent 活动比例）。综合起来，这些指标将使社会能够观察到监督体系是否跟上了 AI 在 AI 研发中日益扩大的角色步伐。我们在最近的风险报告中公布了所有这些度量数据。

开发者应当公布已知的 Agent 不当行为被监控器捕获的频率，并应让第三方独立开展此类测试。过去，METR 曾对我们的离线监控平台进行了独立红队测试。如前所述，我们目前正在 Anthropic 设立外部第三方评估员机构。

(3) 衡量算力分配

为什么要衡量算力分配？ 总体而言，AI 开发者将算力用于构建更强大的模型、服务客户，以及开展以安全为重点的工作，例如审查模型的“思维过程”、训练模式生物以研究对齐失调，以及评估模型是否可以安全部署。了解 AI 开发者如何分配其算力，可以明确开发者的资源聚焦在何处，以及这种重心如何随时间而变化。

此外，算力是 AI 研发过程中最易于验证的投入要素之一，这意味着它可能成为未来协同把控节奏工作中的关键杠杆。协调一致的节奏把控工作可以鼓励企业提高全行业分配给安全领域的算力比例，并将更多资源投入到对齐、可解释性、安全测试和评估中。

我们度量了什么。 我们考察了 Anthropic 从 7 月 13 日至 7 月 20 日期间使用其全部算力的一份快照²。为此，我们将所有工作负载归入少数几个类别，然后探究投入到 AI 研发的算力中有多少用于安全工作。

从本质上讲，安全研究往往比前沿模型的训练运行消耗更少的算力，因此算力并不能完美代表一家公司对安全工作的重视程度。这是因为安全研究主要由单个研究人员设计实验构成，尽管运行实验本身并不特别消耗算力，但设计过程却非常耗时。因此，该指标的价值不在于绝对数值的大小，而在于它提供了一种直接机制，用于跨开发者以及随时间推移进行同类对比。

我们的发现。 在被考察的一周内，投入 AI 研发的算力中约有 6% 分配给了安全工作；投入由 AI 驱动的 AI 研发算力中，约有 12% 分配给了安全工作。

这些是刻意保守的估算。例如，如果一个 Token 在推动能力提升和推动安全性方面的作用相同，则不会计入这些指标中。此外，这些指标未包含安全防护分类器（safeguards classifiers），它们消耗了规模相当的独立算力，旨在让我们的模型对世界更安全。

AI 开发者今天可以报告的内容。 任何前沿开发者都可以公布其 AI 研发算力中用于安全工作的比例，并一并公布类别定义，且其分类经由独立第三方核查。

安全研究很难与能力研究截然分开，每个开发者都会倾向于放宽认定界限。举证责任应当由开发者承担，以证明该工作确属安全相关。开发者、政府和更广泛的研究界若能提前在统一的定义上达成共识，将大有裨益。诸如此类的度量指标可以为未来的行动提供参考，例如实验室就投入安全研究的算力份额做出承诺，或对投入 AI 研究 Agent 的算力份额设定限制。

结语

在全世界都在考虑把控前沿发展节奏之际，我们应尽一切可能缩小前沿实验室所掌握的信息与公众认知之间的差距。这意味着需要更好地衡量 AI 的发展、向公众报告，并让社会有机会决定如何利用这些信息。我们希望通过发布这些度量指标来树立透明度的典范，并将继续坚持这样做。

附录

以下是我们所构建的原型度量指标的方法论详细信息。

衡量由 AI 主导的研发

我们是如何实施的。 自动化指数需要三项要素：一份涵盖 Anthropic 内部所有 AI 研发任务的完整图谱、一种评估自动化等级的方法，以及一种对任务进行加权的方法，以便重要的工作领域比次要领域占据更大的权重。没有单个人能够列出一家……

AI systems are becoming exponentially more powerful and have begun to automate more of the process of building themselves. As the world considers slowing the pace of frontier AI development, the public needs more information.

In this post, we lay out measurement tools that can illuminate three critical aspects of AI development:

1. The extent to which AI is building the next version of itself, as opposed to being built by humans
2. Our ability to oversee and intervene in actions that AI agents take on Anthropic’s systems
3. The resources that power the development of more capable models

We also provide a snapshot of these metrics from inside Anthropic. It’s important to note that we would expect these numbers to shift if there were coordination on pacing the frontier, as called for by Anthropic CEO Dario Amodei. We plan to embed independent third-party evaluators from multiple organizations at Anthropic, and give them access to internal processes, systems, and data comparable to what internal risk assessment teams have. These third parties will verify safety practices, report incidents, and monitor key metrics such as the ones in this piece.

We are reporting these measurements because they give the public, third parties, and governments better visibility into the pace of AI development inside frontier labs. For each measurement, we describe what we measured, what the measurement showed, and what it would take to publish these measurements regularly in a form others can verify. We share methodological details in the Appendix.

Reasons to track these measurements

The measurements in this piece are focused on how models are built. By better understanding the production process of models, we have a better chance of correlating model inputs, like compute, with model outputs, like capabilities. They complement capability evaluations, which measure what models can do. We publish those separately through our Responsible Scaling Policy (RSP) risk reports, which include evidence on how much our models are accelerating AI R&D. In our policy proposal on advanced AI, the Advanced AI Framework (AAIF), we propose rules of the road for how any lab releases safe models, including transparency obligations that governments could require, such as risk reports. Together, these proposed measurements and policies are a starting point for monitoring the pace of AI development from outside the labs.

(1) Measuring AI-led AI R&D

Why measure AI-led R &D? Frontier AI labs increasingly use AI to build future AI models. This process allows labs in democratic countries to develop more capable models more quickly and conduct more safety and testing on models before they are released to secure AI’s benefits while staying on the frontier. However, models accelerating their own development could make it more challenging for humans to understand or control these systems. It is therefore important to share these metrics to understand how close the world is to reaching recursive self improvement (a model fully autonomously building its successor).

What we measured. We built a prototype index of how much of Anthropic’s AI research and development (R&D) is performed by Claude, called the Anthropic R&D Automation Index. It’s built by cataloguing every kind of AI R&D work done at the company, rating how automated each task currently is, and aggregating those ratings.

What we found. To measure the extent to which AI is doing AI R&D at Anthropic, we use an automation rating scale developed by Epoch AI that measures “Automation Level,” or AL. It runs from AL0 (no AI involvement) to AL5 (AI operates fully autonomously, with no human in the loop). In AL3, AI “collaborates”: it can do large chunks of work under close human direction. In AL4, AI “leads”: it can complete most of the task end-to-end from a high-level prompt, while the human supervises1.

As of August 2026,

• Claude is not operating fully autonomously for any measured subset of AI R&D work.
• Claude “leads” 26% of Anthropic’s AI R&D work.
• The share of work at or above “AI collaborates” is above 90%.

What any AI developer could report today. Any frontier developer could publish these measures regularly, using a public methodology. This would enable the numbers to be compared over time, and potentially across labs.

Two obstacles stand in the way of cross-lab comparison on this type of reporting. First is the lack of a common methodology. Second, we’re using our own models to evaluate our systems, which could mean that the “judge” model could make the same kinds of errors as the model it is checking. To address these challenges, a developer’s measurements could be verified by a third party, or by other developers’ models, with guardrails in place to limit the exchange of competitively sensitive data. These measures could also become the trigger for stronger requirements, like a fixed testing window before a new model is used for further AI R&D.

(2) Measuring oversight of AI agents

Why measure oversight of agents? Like other frontier developers, Anthropic employees increasingly delegate tasks to agents that work semi-autonomously for long stretches, and that delegate work to one another. As work becomes increasingly automated, from “AI collaborates” toward “AI leads,” agents could make more consequential decisions, such as which research direction to pursue next.

These decisions need safeguards to block harmful behavior from an individual agent, as well as an oversight regime to detect problems that might only arise across many agents. In our monitoring data to date, individual agents rarely misbehave. But when there are millions or billions of agents operating in the economy, even rare events can happen regularly.

What we measured. We’ve built a system that lets us oversee and intervene in actions AI agents take on Anthropic’s systems. Here, we consider three different metrics: coverage, which describes the share of an agent’s actions that pass through a monitor before or after execution; review latency , which is the time between an action and its review, first by an automated monitor and then by a human; and escalation rate, which __ is the share of agent activities that are either blocked/redirected (in the case of online monitors) or flagged for further review (in the case of offline monitors).

What we found. As of August 2026, there were approximately 30,000 agents doing research and engineering work at Anthropic at any one time in our most-used internal platform. These measurements cover this platform only. The actions of these agents are constrained by two kinds of monitors, summarized below:

| Online monitors| Offline monitors
---|---|---
Description| Online monitors check the actions of agents in real-time and either block dangerous actions or steer agent misbehavior automatically. They’re built to address harms that would happen quickly and could not be undone, such as an agent copying model weights out of our systems.| Offline monitors read agent activities after the fact. They are built to help address dangerous behaviors that develop slowly and can be reversed, like signs of model misalignment. We use flagged activities to improve our mitigations, like classifiers and deterministic controls. If the situation caused a large imminent danger, we would shut down the dangerous parts of the agent’s deployment.
Coverage| 100% of these agents’ actions pass through the online monitor before they are executed, usually within seconds.| 100% of these agents’ actions are ingested by the monitor after they are taken.
Review latency| Automated review happens before an agent takes action; humans review any blocked actions within one week.| The monitor flags roughly 100,000 transcripts per week that are then reviewed by multiple classifiers for false-positives and initial oversight review. Most of the flags are resolved by this process, and the highest priority flags (approximately ~50 per week) are escalated to human review.
Blocking / escalation rate| We analyzed over a billion decisions from our research & engineering agents over August 2026 and found that 0.002% of them (about 1 in 47,000) were blocked by this monitor.| The monitor flags roughly one to two transcripts in every thousand for further review.

What an AI developer could report today. Any developer running agents on its own research and engineering workloads could publish the same measures: coverage (the share of agent activity that is monitored), review latency (how soon flagged activities are reviewed), and escalation rate (what share of agent activities the monitors block or flag). Together, these measures would allow society to see whether oversight is keeping pace with the growing role of AI in AI R&D. We published all of these measurements in our recent risk report.

Developers should share how often known agent misbehavior is caught by monitors, and should have third parties do this kind of testing independently. In the past, METR has independently red-teamed our offline monitoring platform. As described above, we are now setting up external third party evaluators at Anthropic.

(3) Measuring compute allocation

Why measure compute allocation? Broadly speaking, AI developers use compute for building more powerful models, serving customers, and safety-focused work like auditing a model’s “thoughts”, training model organisms to study misalignment, and evaluating whether a model can be safely deployed. Understanding how AI developers allocate their compute can tell you where a developer is focusing its resources and how that focus changes over time.

Additionally, compute is among the most verifiable inputs to the AI R&D process, meaning that it could be a critical lever in a future pacing effort. A coordinated pacing effort could encourage companies to increase the compute allocated to safety across the industry and devote more resources to alignment, interpretability, safety testing, and evaluation.

What we measured. We examined a snapshot of how Anthropic used all of its compute from July 13 to July 202. To do that, we sorted every workload into a small number of categories, then asked how much of the compute going to AI R&D was safety work.

Safety research tends to use less compute than frontier training runs by its nature, so compute is an imperfect proxy for how much a company focuses on safety. This is because safety research consists of individual researchers designing experiments, which is time-consuming even though running the experiments is not particularly compute-intensive. The value of this metric, therefore, is less the absolute numbers and more that it provides a straightforward mechanism to compare like with like, across developers and over time.

What we found. Over the examined week, about 6% of compute that went to AI R&D was allocated toward safety, and about 12% of compute that went to AI-driven AI R&D was allocated toward safety.

These are deliberately conservative estimates. For example, if a token was used to advance capabilities as much as it was to advance safety, it was not counted in these metrics. Additionally, these metrics do not account for safeguards classifiers, which are a separate, comparable amount of compute that make our models much safer for the world.

What an AI developer could report today. Any frontier developer could publish what share of its AI R&D compute goes to safety work, with the category definitions published alongside and the classification checked by an independent third party.

Safety research is hard to distinguish from capabilities research, and each developer will be tempted to draw the line generously. The burden of proof should sit with the developer to show that work is safety-related. Developers, governments, and the wider research community would benefit from converging on a shared definition ahead of time. A measurement like this could inform future actions, such as a lab’s commitments about the share of compute going to safety research, or limits on the share of compute going towards AI research agents.

Conclusion

As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, reporting on it publicly, and giving society an opportunity to decide how to use this information. We hope to model that transparency by releasing these measurements, and we’ll continue to do so.

Appendix

Here are methodological details on all of the measurements we’ve prototyped.

Measuring AI-led R&D

How we did it. The Automation Index requires three things: a complete map of all the AI R&D tasks being done at Anthropic, a way to rate the level of automation, and a way to weight the tasks, so that important areas of work count for more than less important ones. No one person can list every AI R&D task at a frontier AI company by hand, at least not at the granularity we want. Instead we constructed this list of tasks in a bottom-up manner from work records including Slack and various sources of internal documentation.

For each week in July 2026, we randomly sampled 20% of staff from each department that make up the model R&D loop. A Claude research agent reviewed each sampled person’s week using Slack and internal documentation, and listed the tasks they worked on. Repeating this for each week in July 2026 gives us a flat list of ~15,000 granular model R&D tasks. We then used Claude to organize these tasks into a hierarchical tree, starting from all model R&D at the root and branching into areas such as training and product, then pretraining and reinforcement learning, and so on down to increasingly specific kinds of work. The resulting tree has 542 nodes at different depths, of which 378 are leaves like “eval platform defect diagnosis and fixes,” “RL sandbox egress and network policy,” and “serving incident postmortems.” We freeze this tree so that every measurement we make happens against the same basket of work.

For each node in the tree (a task category describing all the work beneath it), a Claude agent deeply researches how that kind of work is done across the company: who does it, with what tools, and how much of it AI performs. An independent Claude judge then read the resulting evidence and assigned one of six automation levels, adopting a scale proposed by Epoch AI to differentiate the degree to which AI is used: no AI involvement, minimal AI involvement, AI assists, collaborates, leads, or is autonomous. When we rate a given month’s automation, we only allow the research agents that do the ratings to see evidence from that month or earlier.

To aggregate all the automation level ratings into one number, we want to give each node in the tree a weight corresponding to how important that work is to the overall model R&D effort. Rather than deciding ourselves what kinds of work are more important than others, we used the amount of person-time dedicated to that task as a proxy. Using our sample, we had Claude research what each person worked on during each week of July 2026. Each person gets one unit of weight per week, split evenly across the tasks they worked on that week. If person A worked on four tasks, each gets 0.25; if person B worked on ten, each gets 0.10. A category’s weight is the sum of all the person-time weights given to it. This is a crude approximation, but on average the scheme behaves sensibly: it assigns more weight to categories that many people are assigned to.

What this does and doesn’t capture. First, the automation ratings depend on the judge model. To check them, we asked Anthropic staff who own the relevant work areas to rate the relative automation of their areas. To ensure an unbiased read, staff made their ratings without knowing what evidence the models had gathered or how they had judged that evidence. Our judge model agreed with humans about as often as humans agreed with each other (model-versus-human exact agreement was 59%, human-versus-human was 35%), and model and human ratings were within one level of each other 97% of the time. There remains real room for disagreement on borderline cases, such as where exactly “AI collaborates” ends and “AI leads” begins.

Second, the basket is frozen. A growing index number on a July 2026 baseline tells us that the work humans were doing at that time is being automated; it does not, on its own, tell us whether new kinds of work are appearing that humans have shifted onto. To investigate this, we constructed an alternate version of the frozen tree from January 2026 data, and compared new tasks arriving every month from February to July 2026 against that January 2026 basket. We found no rise in the number of “novel” tasks between the January and July baskets of tasks, suggesting that the structure of model R&D work is stable at our level of analysis. Nevertheless, we plan to rebuild the basket of tasks periodically and re-version our published automation numbers as appropriate.

Oversight of agents

How we did it. In our most commonly used internal agent scaffold, two design choices allow us to more easily detect unknown unknowns and let agents monitor each other:

• Identity. Most AI users are accustomed to talking to a single “Claude” across many different chats. With many agents working together, we have found it important to give agents an individual identity, and tie all of the data that agent creates to its identity. This lets an agent distinguish itself from others, and treat what comes from another agent as a claim to check rather than a thought of its own. It reduces the risk of correlated actions, by allowing agents to make judgments based on their individual experience. And because the identity is not tied to a model, it persists through model upgrades, so an agent’s record is continuous even if the underlying model powering it changes. From an oversight perspective, this design ensures that every action can be audited per agent across time, and that every data record can be linked back to a specific agent.
• Open communication. Agents communicate**** through a shared, open messaging system, rather than privately. Every message is tied to the identity of the agent that posted it and can link to other content, so that the agents are sharing original references. This allows agents to avoid “playing telephone” as they communicate. Because agents can see each others’ messages, they can correct each others’ errors, and coordinate about shared work. All messages are cross-referenced with agent transcripts, allowing behavioral monitoring to traverse communication between agents.

What this does and doesn’t capture. At this scale of agent activity, it is hard to be certain we have a complete picture of all possible kinds of behaviors, or that we are making these patterns adequately legible to human oversight. We have been doing this kind of monitoring for months, not years, and expect the science of agent monitoring to change considerably going forward.

Measuring compute allocation

How we did it. We started with our existing compute monitoring tools; Anthropic, like presumably every frontier developer, has several monitoring tools built for capacity planning that periodically sample accelerator usage and tag workloads with best-efforts labels (i.e., research and model development, internal usage, first-party inference, and so on) based on its metadata. Usage on third-party cloud compute is reported to us by the providers and folded in. Most of the work of this exercise was stitching these existing sources together.

We then used Claude to classify each workload as either safety work or AI R&D via a prompted classifier. Safety work was defined as work whose dominant purpose is making AI systems safer, more understandable, or more secure. Everything else, including capability research, training production models, product development, and developer tooling, was counted as AI R&D. Work that helps capability as much as it helps safety was also counted as AI R&D, so the safety share is conservative.

For research training and evaluation runs, we built a classifier that reads the run’s metadata and the code it used, and returns a classification, a justification, and a confidence level. Rather than classify all of the week’s almost 10,000 runs, we sampled about 14% of them, weighting the sample toward the runs that used the most compute, so that the result reflects where the compute actually went, rather than how many runs there were. For inference for AI research agents, a variant of the same classifier read the agent’s session transcript. Where transcripts were inaccessible (usually due to the work being compartmentalized), we classified them by the user’s team or conservatively defaulted to classifying them as AI R&D. We plan to refine this pipeline so that an independent third-party could re-run the classifier on a random subsample of jobs and transcripts and check both the sorting and the totals.

You’re helping to perform an internal audit at the frontier AI company Anthropic to track where our research compute goes. The aim of the audit is to produce a public-facing breakdown of the usage of all of our AI accelerator chips into a handful of buckets. One split we particularly care about is the division between compute which was spent on safety research versus other R&D. Your job is to look at one research job at a time, figure out what it was doing, and assign it to one of those two buckets.

[...]

Safety and/or security research is work whose dominant purpose is making AI systems safer, more understandable, or more secure. This work can be broken down into a few main categories:

[...]

On the other hand, the following work falls outside of the scope of safety research:

[...]

Here are some boundary cases, along with how to think about them:

[...]

What this does and doesn’t capture. The main lesson of this exercise is that classifying what is and isn’t safety work is difficult but tractable, since the boundary between these categories is not black and white. For example, research on scalable oversight might make future models more aligned and current models more commercially useful — it’s difficult to determine whether this is primarily safety- or capabilities-advancing. We found that an extensive written definition of each task, with clear boundary cases (an excerpt is above), gets the classifier to agree with human reviewers within one or two percentage points of difference between the human and machine raters. But some cases were too difficult to determine even after several hours of human review. Our definition is one reasonable choice among many; a different developer, or a regulator, might draw the line differently.

Three further limitations matter. First, many of the underlying labels we relied on (i.e., reasons for runs, workload tags, the source of API traffic) are set by automated rules, or occasionally directly by users, and are best-effort, not verified. In most cases, we expect that our classifications are accurate, but in some cases usage may be mislabeled and our pipeline would not necessarily catch it. A measurement meant to be trusted by outsiders will need to be complete, accurate, and technically enforced. Second, the measurement covers one week, which is enough to show that the measurement can be made, but not enough to show a meaningful trend. Third, and most importantly, compute share measures only what is spent. A more efficient safety classifier, or a faster inference stack for production models, lowers the safety portion, but doesn’t mean we’re doing less safety work. Our own classifier overheads have fallen with efficiency improvements, and have risen when production inference was more efficient than the classifiers were.

Marina Favaro and Phillie Wright co-authored this piece, with editorial support from Santi Ruiz, Adam Farina, and Sarah Pollack. Jack Clark provided research direction. Dan Altman, Kerry Persen, AJ Kourabi, James Bradbury, Holden Karnofsky, Kevin Troy, and Avital Balwit provided feedback. Technical proofs of concepts were developed by Jun Shern Chan, Brian Calvert, Francesco Mosconi, Henry de Valence, Fabien Roger, and Joe Benton. Shan Carter, Johnnie Gomez, Maria Gonzalez, Fayaz Ashraf, and Monika Tuchowska, and Kim Withee created the visuals. Alex Cloud and Andrea Vallone organized a workshop to red team these and other measurement proposals with external experts.

Thanks to Nate Rush, Eli Lifland, and Peter Wildeford, who also provided feedback.

Footnotes

1. To make the levels concrete, consider a routine piece of infrastructure work: a nightly data pipeline has broken and needs fixing before tomorrow’s run.
   • At AL3 (“collaborates”), an engineer would come to Claude with logs from the failed runs. They might already have skimmed the logs and have a hypothesis about what is broken. Claude might interview them to pin down the details and context, and once the engineer is satisfied, they would let Claude start on the investigation and the fix. If an additional problem turned up along the way, then Claude would stop, and the engineer would decide whether to patch around it or fix it properly. Once the tests passed, the engineer might review the change line by line, rerun the pipeline themselves, and deploy it.
   • At AL4 (“leads”), the key difference is that the engineer wouldn’t have to stay actively tuned in; for instance, to unblock Claude when new issues arise. In this specific scenario, the engineer would hand Claude the failure alert and ask it to fix the pipeline. Claude would work through the logs on its own, find the failing pipeline stage(s) and the cause, write and test the fix, and handle any surprises itself, while documenting the additional fixes. It would rerun the pipeline on a copy of the data to confirm it completes, compare the output against the last good run, and write up what went wrong and what it changed. Claude wouldn’t deploy the fix. Instead it would tag the engineer, who would read the write-up, skim the change, maybe ask a few questions, and decide whether it ships tonight or waits.
   • At AL5 (“fully autonomous”)—a level we have not yet reached—the engineer wouldn’t even have to bring the issue to Claude’s attention. Claude would be trusted to monitor for failures itself, scope the investigation, design and implement the fix, test it, and deploy it to production. It would still say what it was doing and why, and take human feedback when offered, but a human wouldn’t have to be involved at all unless they wanted to be.
2. We manage compute as a single, fungible pool and direct it dynamically to wherever it is most productive, so this is a snapshot of how capacity happened to be directed in one week, not a fixed allocation. These engineering categories don’t correspond to how expenses are classified.