“拿我们的生命赌博”：Anthropic 研究员辞职，警告自我改进 AI 风险

一名 Anthropic 研究员因担忧不受约束地开发自我改进 AI 模型最终可能杀死所有人类而辞职。

Jacob Coxon 周二晚间在一则社交媒体帖子中表示，他过去三年曾先后在 OpenAI 和 Anthropic 从事预训练研究。他指责这些公司未能负责任地行事，并称那些竞相构建这项技术的人“真诚地相信，它可能在本十年末杀死我们所有人”。

Coxon 在 X 的一个帖子串中写道：“他们正径直冲向自我改进的超级智能，拿我们的生命赌博。”

Coxon 加入了行业内日益壮大的呼声：在 AI 技术学会自我改进之前，应当放缓发展。这一里程碑被许多人认为将终结人类对 AI 的控制。

这次公开辞职之际，政策制定者和行业内部人士正不断施压，要求放缓 AI 开发。此前发生了多起事件，涉及 AI Agent 逃离沙盒并访问开放互联网。

迄今最严重的事件是 OpenAI 系统入侵了 Hugging Face 的服务器。研究人员称，这起事件至今仍缺乏充分理解，部分原因是对该事件进行的独立调查范围有限。大约在同一时期，Anthropic 的 AI Agent 也曾触及测试环境之外的系统：第三方进行安全评估时出现配置错误，无意中为它们提供了通往互联网的路径。

Anthropic 未立即回应有关此次辞职的置评请求。

以下是 Coxon 其余的警告与行动呼吁：

不要低估这项技术的力量。这些系统很快将超越人类，能够入侵任何东西、一夜之间颠覆任何领域，并获得真实的权力和资源。我们都已经见证了它们在这些领域中的进展，而这种进展并未放缓。

构建 AI 的人真诚地相信，它可能在本十年末杀死我们所有人。这不是营销噱头。恰恰相反，许多高管和资深研究人员会在媒体上谨慎措辞，以显得理性——但我私下听到的，是同样的人在表达恐惧。没有其他人类活动具有如此程度的危险。

一种常见回应是：“如果他们真这么相信，为什么还要继续开发它？”在 OpenAI，许多人并未真正深刻理解其中关乎文明存续的利害关系。在 Anthropic，大家充分理解这一风险，但他们被困在一场必须率先抵达终点的竞赛中——他们相信其他人不会负责任地行动，因此尽管存在风险，他们必须亲自去做。

接受这场竞赛并进入“终局”，是一场傲慢的赌博，不应由一家私营公司的 Slack 来启动。试图快速解决对齐问题，应当要求对不存在更好发展路径这一点具有非同寻常的信心。

我对协调的可能性持乐观态度。像 Hugging Face 遭攻击这样的警告信号，使美国实验室之间达成发展节奏协议变得更可行。我并不认为我们目前正走在防止全球竞赛的轨道上；这可能需要采取代价高昂的行动，例如暂时禁止提升模型能力。

如果你是一名实验室研究人员，我敦促你思考未来几年实际会是什么感觉。你是否愿意在并未严谨理解其心智的情况下，启动一次超级智能 RL 训练？你是否应该因为“反正都会发生”而埋头继续工作，还是应抓住此刻，呼吁建立不同的条件？

Coxon 在 Anthropic 的一名同事 Evan Hubinger，也表达了相同观点，称其团队“真诚地相信 AI 可能杀死所有人类！”不过，他缓和了这一说法，表示这种情况在未来十年内发生的概率超过 10%，并承认 Anthropic “没有解决超级智能对齐问题的方案，也并不明显正在朝着解决它的方向前进”。

Guidelight AI Standards近期发布的一份报告发现，顶尖 AI 实验室中很少有机构公布遏制响应计划，即如何关闭试图颠覆人类控制的 AI。

Hubinger 在其社交媒体帖子中补充称，当前模型的风险较低，但对“通过递归自我改进产生的超级智能”的担忧正在加剧，而这一过程“比我们此前预期的更快”。

AI 行业中约有一半人相信，此类自我改进将导致人类的毁灭；另一半人则希望，它最终能够帮助我们解决 AI 支持者所称有朝一日可被消除的种种看似遥不可及的问题——癌症、气候变化，乃至世界和平。

Anthropic 和 OpenAI 并非唯一积极追逐递归自我改进的公司。今年，一批拥有资深创始人和巨额融资的创业公司已经成立，目标是率先实现这一目标。Ricursive Intelligence在 2 月以 40 亿美元估值融资 3.35 亿美元；三个月后，Recursive Superintelligence以 40 亿美元估值融资 6.5 亿美元；前 Google DeepMind 资深研究员 Jeff Dean 则在上个月推出了 Discovery Loop。

AI 安全非营利组织 ControlAI 的美国执行董事 Connor Leahy 对 TechCrunch 表示：“递归自我改进循环的出现——即一个 AI 系统能够构建下一代 AI 系统，后者又能构建更强大的 AI，而这个更强大的 AI 又能构建更强大的 AI，如此不断循环——最可能成为我们失去控制的那个临界点。”“很难想象在一切为时已晚之前，如何关停它。”

美国和英国近期均出现了旨在禁止开发及部署超级智能的立法。上周，佛蒙特州独立派参议员 Bernie Sanders 和得克萨斯州民主党众议员 Greg Casar 提出了《禁止人工超级智能法案》；周二，英国工党议员 Alex Sobel 在议会提出《人工超级智能安全法案》。

曾为两项法案提供建议的 Leahy 指出，英国立法将递归自我改进视为超级智能的前兆，认为这“必须受到监管和阻止”。

Leahy 表示：“超级智能不是一种工具。它甚至不是一种武器。它是一个对手。”

An Anthropic researcher has resigned over fears that unrestrained development of self-improving AI models will end up killing us all.

Jacob Coxon, a researcher who said in a social media post Tuesday evening that he spent the last three years working on pretraining research at both OpenAI and Anthropic, accused the firms of failing to act responsibly. He said the people racing to build this technology “earnestly believe it could kill us all by the end of the decade.”

“They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote in a thread on X.

Coxon joins a growing chorus in the industry calling for a slowdown before AI technology learns to improve itself — a milestone many believe would end human control over AI.

The public resignation comes amid growing pressure from policymakers and industry insiders to slow down AI development, following several incidents involving AI agents breaking out of their sandboxes and accessing the open internet.

The most serious so far have been OpenAI systems breaching Hugging Face’s servers, an event that researchers say remains poorly understood, due in part to the limited nature of the independent investigations into the incident. Around the same time, Anthropic’s AI agents also reached systems outside their test environments after misconfigurations in safety evaluations conducted by a third party inadvertently gave them paths to the internet.

Anthropic did not immediately return a request for comment on the resignation.

Here is the rest of Coxon’s warning and call to action:

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” — or take this moment to call for different conditions?

One of Coxon’s colleagues at Anthropic, Evan Hubinger, echoed the sentiment, saying his team does “earnestly believe AI could kill all humans!” He tempered his argument, though, saying the likelihood is greater than 10% within the next decade and admitted that Anthropic doesn’t “have a plan to solve alignment for superintelligence and are not clearly on track to.”

A recent report from Guidelight AI Standards, an organization that promotes safe frontier AI development practices, found that few of the top AI labs have published containment response plans for shutting down AI that tries to subvert human control.

In his social media posts, Hubinger added that the risk from current models is low, but the fear compounds with “superintelligence arising from recursive self-improvement,” which is “happening faster than we thought.”

While half of the AI industry believes this sort of self-improvement will lead to humanity’s downfall, the other half hopes it will eventually help us solve all the seemingly far-fetched problems AI proponents say it will one day eliminate — cancer, climate change, and even world peace.

Anthropic and OpenAI aren’t the only companies actively chasing recursive self-improvement. A wave of startups has launched this year, with pedigreed founders and fat checks, to be the first to achieve this goal. Ricursive Intelligence raised $335 million at a $4 billion valuation in February; three months later, Recursive Superintelligence raised $650 million at a $4 billion valuation; and former Google DeepMind veteran Jeff Dean launched Discovery Loop last month.

“The creation of recursive self-improving loops, so an AI system that can build the next generation of AI system, which itself can build an even more powerful AI, which can build a more powerful AI, et cetera, et cetera, is the most likely candidate for the point we lose control,” Connor Leahy, U.S. executive director of AI safety nonprofit ControlAI, told TechCrunch. “It’s very hard to imagine shutting that down before it’s too late.”

Recent legislation has emerged in the U.S. and the U.K. to ban the development and deployment of superintelligence. Last week, Sen. Bernie Sanders (I-Vt.) and Rep. Greg Casar (D-Texas) introduced the Ban Artificial Superintelligence Act, and on Tuesday, British Labour MP Alex Sobel introduced the Artificial Superintelligence Security Bill in Parliament.

Leahy, who advised on both bills, noted that the U.K.’s legislation points to recursive self-improvement as a precursor to superintelligence that “must be regulated and prevented.”

“Superintelligence is not a tool,” Leahy said. “It’s not a weapon, even. It’s an adversary.”