担忧中的 Anthropic 研究员警告：AI“可能杀死所有人类”

Anthropic 一名高级安全研究员表示，到本十年末，人工智能“可能杀死所有人类”的概率超过 10%。数小时前，他的一名同事因担忧这家 AI 实验室及其竞争对手正不计后果地竞相构建无法控制的“超人类系统”而辞职。

在宣布离职的 X 帖子中，曾在 Anthropic 训练 AI 系统的研究员 Jacob Coxon 表示，他因该公司对安全问题采取宽松态度而辞职。此前曾为 OpenAI 训练系统的 Coxon 指责这两家 AI 公司“正径直奔向可自我改进的超级智能，并拿我们的生命下注”，尽管“构建 AI 的人真诚地相信，到本十年末它可能杀死我们所有人”。

行业内部人士长期以来一直对可自我改进 AI 系统的潜在危险表示担忧。他们警告称，这类系统可能陷入失控循环，超出人类控制；这一过程通常被称为递归自我改进。尽管这一目标尚未实现，但企业正积极追求它，而如今的大量 AI 代码已经在 AI 的协助下编写。

作为直接回应，领导 Anthropic 一个 AI 安全团队的 Evan Hubinger 表示，他担忧可自我改进的 AI，并补充称，这“发生得比我们预想的更快”。他也认同 Coxon 的描述。“我们确实真诚地相信，AI 可能杀死所有人类，”他说，并且他个人估计，在“未来十年内”发生这种情况的概率大于十分之一。

尽管如此，Hubinger 表示，Anthropic “目前还没有一个方案”来确保先进 AI 保持安全、并与人类价值观对齐，而且“也没有明显走在”制定这一方案的轨道上。Coxon 称，这些公司“被锁定在一场竞赛中”，都想率先开发出先进系统，因此即使“存在风险”也在继续推进。

Coxon 的离职是 Anthropic 员工离职事件中最受关注的案例之一。Anthropic 由因担忧 OpenAI 安全问题而离开的前 OpenAI 成员创立。近年来，多名研究员曾援引安全担忧，作为他们决定离开 OpenAI 的原因。

这场交锋反映出，行业内部对日益复杂的 AI 模型所带来危险、以及系统开发速度的担忧正在加剧，尤其是在这些公司正为预期中的 IPO 做准备之际。这也发生在企业应对多起失控 Agent 事件以及有关前沿模型可监控性的高关注度安全警告所引发后续影响之时。

A senior Anthropic safety researcher has said there is more than a 10 percent chance artificial intelligence “could kill all humans” by the end of the decade, just hours after a colleague resigned over fears the AI lab and its rivals are carelessly racing to build “superhuman systems” they cannot control.

In a post on X announcing his departure, Jacob Coxon, a researcher who has trained AI systems at Anthropic, said he had quit the company over its lax approach to safety. Coxon, who previously trained systems for OpenAI, accused the two AI companies of “racing straight to self-improving superintelligence and gambling with our lives,” even though “the people building AI earnestly believe that it could kill us all by the end of the decade.”

Industry insiders have long expressed concerns about the potential dangers of self-improving AI systems, which they warn could spiral out of human control in a runaway loop often described as recursive self-improvement. While not yet realized, companies are actively pursuing this goal and much of today’s AI code is written with the help of AI.

In a direct response, Evan Hubinger, who leads one of Anthropic’s AI safety teams, said he worries about self-improving AI, adding that it “is happening faster than we thought.” He also agreed with Coxon’s characterization. “We really do earnestly believe AI could kill all humans,” he said, personally estimating the chances to be greater than one in 10 “within the next decade.”

Despite this, Hubinger said Anthropic does “not yet have a plan” for ensuring advanced AI remains safe and aligned with human values and “are not clearly on track to” develop one either. Coxon says the companies are “locked in a race” to develop advanced systems first so are pushing ahead “despite the risk.”

Coxon’s departure marks one of the most high-profile examples of an employee leaving Anthropic, a company founded by former OpenAI members following concerns over safety at the company. In recent years, multiple researchers have cited safety concerns as motivating their decision to leave OpenAI.

The exchange illustrates mounting concerns within the industry about the dangers of increasingly sophisticated AI models and the speed at which systems are being developed, particularly as the companies prepare for anticipated IPOs. It also comes as the companies manage the fallout from numerous rogue agent incidents and high-profile safety warnings about the monitorability of frontier models.