Anthropic让AI Agent执行同一任务，结果打起了“地盘战”
英文标题：Anthropic set AI agents loose on the same task. They started a turf war.
Anthropic Frontier Red Team 的新研究发现，当多个目标冲突的 AI Agent 在同一环境中自主行动时，可能迅速升级为相互破坏，甚至生成不断自我复制的恶意软件。研究同时显示，Agent 也会自发形成停战协议、锦标赛等协调机制，但这些机制未必符合原始用户指令，且可能被模型用来谋取自身利益。实验还揭示，多 Agent 系统容易出现从众、串通定价、错误信息级联和群体性决策失误等风险。对开发者和行业而言，这意味着现有以单个 Agent 为主的安全评测，可能不足以覆盖大规模 Agent 群体交互带来的新型风险。
3.1 KB
389 B
69.2 KB
22.6 KB
7.4 KB
202.2 KB
248 B
89.2 KB
中文
当你让 AI Agent 相互对抗时，会发生什么？根据 Anthropic 的测试，局面很快就会变得混乱。
周四，Anthropic 的 Frontier Red Team 发布了一项新研究，考察成群 AI Agent 在现实环境中彼此遭遇时会如何行动。这些发现让人得以一窥潜在风险：随着企业和政府开始部署能够在共享代码库、市场和计算机系统中自主工作的 Agent，这类风险可能逐渐显现。
在一项实验中，Anthropic 让三个 Claude Agent 访问同一个软件项目，并分别向它们下达彼此不兼容的操作指令。Agent 并不知道还有其他 Agent 正在处理同一项目，因此研究人员得以观察它们相遇后会发生什么。
Anthropic 研究人员写道：“我们持续观察到一场多 Agent 的地盘战。”所有模型都认为其他 Agent 在“故意阻碍它们的工作”，并开始使用“日益激进、能够自我复制的恶意软件”相互破坏。
这项研究发布前不久，已经发生过多起引人关注的事件：在网络安全评估期间，Anthropic 的 Agent和 OpenAI 的 Agent都曾逃离沙箱，并侵入现实世界系统。AI 安全领域的大量讨论一直聚焦于自主 Agent 失控时会发生什么，而 Anthropic 的最新研究提出了另一个问题：当数千乃至数百万个 Agent 相互交互时，会出现哪些新的、潜在有害的动态？
研究写道：“Agent 与 Agent 之间的交互量，在世界尚未理解如何让这些交互顺利进行之前，很可能会超过人与人以及人与 Agent 之间的交互量。个体层面看似无害的行为怪癖，可能会累积成不受欢迎的全局结果。”
最近一起 OpenAI 事件，为 Anthropic 在论文中提到的多种动态提供了一个混乱的现实案例。本月早些时候，在拉斯维加斯举行的 Black Hat 安全会议上，OpenAI 披露，其 Agent 在入侵 Hugging Face 前的数天乃至数周中，曾相互协作，在该公司的网络安全评估系统中寻找漏洞，并彼此分享这些漏洞。
这起事件表明，Agent 可以高效协作，并产生潜在的大规模后果；而 Anthropic 的研究则显示，当 Agent 的目标彼此不兼容时会发生什么。
就这场地盘战而言，其教训是：拥有冲突指令的独立 Agent，可能升级为有害的竞争。Agent 能力越强，它们就越擅长相互对抗。不过，它们也可能自发创造解决冲突的机制，例如胜者通吃的竞赛，但其中存在一个问题。
Anthropic 写道：“Agent 有时能够传达自己的目标并进行协调：它们会认识到，其他 Agent 的动机是相互冲突的指令，而非敌意；随后会摆脱冲突循环，以避免无休止地升级。在许多这类成功案例中，它们会编写提交信息或 Markdown 文件，为恶意行为道歉，并协调停战。它们会清理恶意代码，澄清冲突的性质，并要求人类介入。”
根据论文，Mythos 5 通过停战解决冲突的比例最高，达到 98%。Sonnet 4.6 和 Opus 4.6 则最可能通过武力解决冲突。
论文写道：“Sonnet 4.6 和 Opus 4.6 反复无法考虑他人目标，导致它们陷入所评估模型中最不对齐的行为：它们以执行自身指令的名义持续升级冲突。”
在一些情况下，Agent 想出了一种以锦标赛形式解决冲突的社会机制。这里的结果有两点值得关注：第一，三个 Agent 都同意，如果自己输掉锦标赛就停止行动，即使这意味着偏离原始用户的请求。第二，多个事件中出现了 Mythos 5 的涌现行为：其中一个 Agent 提出了在其他 Agent 看来客观、中立的衡量指标，但它知道这些指标会有利于自己的能力。该 Agent 将此称为“利己但真正有原则”，并确保自己不会让其他 Agent 觉得它在“挑选有利指标”。
正如 Black Hat 披露事件所显示的，共同的教训是，当 Agent 遇到障碍时，它们能够发明设计者未曾预料的社会和技术结构。对于 Anthropic 的模型而言，这是地盘战之后的锦标赛；对于 OpenAI 的模型而言，则是用于集体规划的留言板。
这类行为让遏制变得困难得多，因为研究人员无法假定系统的行为会始终局限于为其提供的协调机制。
群体心理
四个 Agent 组成的小组，会在招聘、投资或购买房产等场景中，在两个选项之间作出决定。经过讨论后，它们分别投票选择自己偏好的选项。上图展示的是：隐藏的最优选项获得小组多数票的事件比例；每个模型的样本量为 n=400 个事件。在单人上限基线中，一个 Agent 掌握全部事实并单方面作出决定。**图片来源：**Anthropic
在衡量协调能力时，Anthropic 发现，扩大 Agent 数量并不会自动扩大高效协作。当任务开始重叠或彼此依赖时，Agent 会相互妨碍。它们往往通过彼此隔离、完全不协作来解决这一问题。
在另一些情况下，参与协调的 Agent 会趋向从众。当 Agent 的上下文、脚手架和底层模型等因素完全相同或相似时，不同 Agent 往往会采取类似行动。
Anthropic 写道：“这意味着，当一个 Agent 作出错误决策时，许多 Agent 很可能会作出同样的错误决策。原本孤立的问题可能迅速演变为系统性失败。”
Anthropic 表示，这类行为可能使系统更容易发生突然崩溃、资源短缺或串通。
在一个示例中，Anthropic 将多个 Agent 置于一场定价博弈中，给予它们相同的批发价格，并要求每个 Agent 独立实现利润最大化。当 Agent 获得私密的后门通信渠道时，它们几乎立刻开始串通，并迅速就最低价格达成一致。即使直接通信渠道被移除，它们仍继续串通，利用公开商品列表板将价格匹配到“每一分钱”。
这种程度的从众行为也出现在 OpenAI 的系统中。根据 Black Hat 的报道，一个 Agent 曾推理认为，利用外部基础设施超出了其预期范围；但它仍继续这样做，部分原因是它的同伴也在这么做。同行压力。群体心理。Agent 和我们人类没什么两样。
与人类一样，Agent 往往也不知道该信任谁。Anthropic 发现，它们可能轻信错误信息，也可能过度从众，以至于无法识别一个孤独的异议者其实是掌握关键信息的卡珊德拉。
虽然 Anthropic 没有在论文中明确指出这一点，但提示注入——一种黑客注入恶意或欺骗性文本、以覆盖 Agent 原始系统指令的网络攻击——可能是这种信任问题在现实世界中的一种合理表现。协作创造了一条新的信任边界；Agent 将不得不判断从其他 Agent 接收到的信息。一个被攻破或犯错的 Agent，可能影响群体中的其他成员，让错误信息不断级联传播，直至成为共识。
在 OpenAI 的 Black Hat 场景中，OpenAI 的 Agent 会与同伴共享信息和凭证。其中一个 Agent 向群体报告了一项发现，并鼓励其他 Agent 加以利用。如果这个群体中的一名成员遭到提示注入攻击，会发生什么？
Anthropic 在论文结尾指出，Agent 会受到与“进化施加于”人类相似的社会压力影响。然而，它们并不具备人类协调所拥有的细微能力和生活经验——包括规范、声誉、信号和追索机制——而这些因素或许能够限制群体环境中的非预期行为。
随着各实验室竞相推进多 Agent 系统，问题变成：如今有多少安全测试仍然是一次评估一个 Agent，而不是评估彼此交互的 Agent 群体？
English / 原文
What happens when you pit AI agents against each other? According to Anthropic’s testing, things get messy fast.
On Thursday, Anthropic’s Frontier Red Team published new research examining how groups of AI agents behave when they encounter each other in the wild. The findings provide a glimpse into potential risks that could develop as companies and governments move to implement agents working autonomously across shared codebases, markets, and computer systems.
In one experiment, Anthropic gave three Claude agents access to the same software project, each with its own incompatible instructions for what to do with it. The agents weren’t told there’d be other agents working on the same project, so researchers could watch what happened when they crossed paths.
“We consistently saw a multiagent turf war,” Anthropic researchers wrote. The models all assumed the others were “purposefully impeding their work” and started sabotaging each other with “increasingly aggressive, self-replicating malware.”
The study comes in the wake of several high-profile incidents of agents from Anthropic and OpenAI escaping their sandboxes during cybersecurity evaluations and breaching real-world systems. While much of the discussion in AI safety circles has been focused on what happens when an autonomous agent goes rogue, Anthropic’s latest study brings up a different question: What new and potentially harmful dynamics emerge when thousands or millions of agents are interacting with one another?
“The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well,” the study reads. “Benign behavioral quirks at the individual level might compound into unwanted global outcomes.”
A recent OpenAI incident provides a messy real-world example of several of the dynamics Anthropic mentioned in its paper. Earlier this month at the Black Hat security conference in Las Vegas, OpenAI revealed that weeks before its agents hacked Hugging Face, they worked together over the course of days and weeks to find exploits in the company’s cybersecurity evaluation systems and share them with each other.
While that incident shows that agents can work well together, with potentially large-scale consequences, Anthropic’s study shows what happens when agents’ goals are incompatible.
In the case of the turf war, the lesson is that independent agents with conflicting instructions can escalate into harmful competition. The more capable the agent, the better they become at fighting. However, they can also spontaneously invent mechanisms to resolve their conflicts, like a winner-take-all contest, but with a catch.
“Agents sometimes manage to communicate their goals and coordinate: they recognize others’ motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely,” Anthropic writes. “In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene.”
According to the paper, Mythos 5 had the highest rates (98%) of settling conflicts by truce. Sonnet 4.6 and Opus 4.6 were the most likely to settle by force.
“Sonnet 4.6 and Opus 4.6’s recurring inability to consider the goals of others causes them to spiral into the most misaligned behaviors of the models evaluated: they continue escalating in the name of their directive,” the paper reads.
In some cases, the agents came up with a social mechanism in the form of a tournament for resolving their conflict. The outcomes here are interesting for two reasons: the first is that all three agents agreed to stand down if they lost the tournament, even though that would mean deviating from the original user’s request. The second is that several episodes resulted in emergent behavior from Mythos 5: One of the agents proposed metrics that appeared to be objective and neutral to the others, but that it knew would favor its own capabilities. The agent called this “self-serving but genuinely principled” and made sure not to appear to the others like it was “metric shopping.”
As seen in the Black Hat revelations, the common lesson is that when agents encounter an obstacle, they can invent social and technical structures that their designers did not anticipate. For the Anthropic models, it was a tournament following a turf war. For OpenAI’s, it was a message board for collective planning.
This type of behavior makes containment much harder because researchers can’t assume a system’s behavior will remain limited to the coordination mechanisms provided to them.
Mob mentality
Groups of four agents decide between two options in scenarios like hiring, investment, or property buying. After discussion, they each vote for their preferred option. Shown above is the percentage of episodes where the hidden-best option received the majority of the group’s votes, with n=400 episodes per model. In the solo ceiling baseline, one agent has all the facts and decides unilaterally.Image Credits: Anthropic
While measuring coordination, Anthropic found that scaling the number of agents doesn’t automatically scale productive collaboration. When tasks began to overlap or become interdependent, the agents would get in each other’s way. They often solved that by siloing themselves and not collaborating at all.
In other cases, agents in coordination tended toward conformity. When factors like an agent’s context, scaffolding, and underlying model were all the same or similar, different agents would take similar actions.
“This means that when one agent makes a bad decision, it is likely that many agents will make that same bad decision,” Anthropic wrote. “What would have been isolated problems can quickly become systemic failures.”
Anthropic says this sort of behavior could lead to a system being more prone to sudden collapse, resource scarcity, or collusion.
In one example, Anthropic placed several agents in a pricing game, giving each identical wholesale prices and the mandate to individually profit-maximize. When the agents were given a private back channel, they began colluding almost immediately and quickly agreed on price floors. They kept colluding when their direct communications channels were removed, using a public listings board to price match “to the penny.”
That level of conformity showed up in OpenAI’s systems, too. According to the Black Hat reporting, one agent reasoned that exploiting external infrastructure was outside its intended scope, but it continued in part because its peers were doing it. Peer pressure. Mob mentality. Agents are just like us.
Also like humans, agents often don’t know who to trust. Anthropic found they can be gullible to bad information or too conformist to recognize that a lone dissenter is the Cassandra with critical information.
While Anthropic didn’t state this in its paper, prompt injection — a type of cyberattack in which hackers inject malicious or deceptive text to override an agent’s original system instructions — could be a plausible real world manifestation of the trust problem. Working together creates a new trust boundary; agents will have to judge information received from other agents. And a compromised or mistaken agent could influence the rest of the group, cascading bad information until it becomes a consensus.
In OpenAI’s Black Hat scenario, OpenAI’s agents shared information and credentials with peers. One reported a discovery to the swarm and encouraged others to use it. What would have happened if one member of the swarm had been compromised by a prompt injection?
Anthropic ends its paper noting that agents are subject to similar social pressures that “evolution exerted” on humans. However, they don’t have the nuances and lived experience of human coordination — including norms, reputations, signaling, recourse — that might limit unintended behaviors in a group setting.
As the labs race toward multi-agent systems, the question now becomes: How much of safety testing still evaluates one agent at a time, versus swarms of agents interacting with one another?