Anthropic 研究员称：AI 在未来十年“杀死全人类”的概率超过 10%
英文标题：🔶 Anthropic researcher says AI has more than 10% chance of 'killing all humans'
Anthropic 对齐科学负责人 Evan Hubinger 表示，他个人认为未来十年 AI 导致全人类死亡的概率超过 10%，并承认 Anthropic 尚无解决超级智能对齐问题的明确方案。此前，Anthropic 研究员 Jacob Coxon 因担忧 AI 实验室正在为自我改进的超级智能“拿人类生命下注”而辞职。两人担心的重点并非当前模型，而是可能通过递归自我改进出现、能力迅速超越人类的超级智能。相关言论凸显出 AI 行业在持续融资和推进商业化的同时，开发者、研究机构与监管方仍需面对模型失控、网络安全及全球 AI 竞赛等治理难题。
1.9 KB
1.2 KB
1.3 KB
606 B
58.4 KB
34.5 KB
1.8 KB
1.1 KB
50.4 KB
中文
人工智能可能“杀死全人类”的概率超过 10%，Anthropic 一名安全研究员周二表示。数小时前，另一名员工称自己将因担忧 AI 实验室正在“拿我们的生命下注”而离开公司。
这些言论凸显出，身处 AI 开发核心的人士愈发担忧，这项技术可能失去控制并对人类构成威胁；与此同时，Anthropic 和 OpenAI 仍在继续筹集巨额资金，并朝着预期中的上市推进。
Anthropic 研究员 Jacob Coxon 周二表示，他已从公司辞职。Coxon 称，Anthropic 和 OpenAI 的行为都不负责任。
“它们正径直奔向能够自我改进的超级智能，并拿我们的生命下注，”Coxon 在 X 的一则帖子中表示。
自我改进是指 AI 系统无需大量人类干预即可改进自身的理念。通常被称为递归自我改进的能力目前尚无法实现，但 AI 实验室正在朝这一目标努力。
“不要低估这项技术的力量。这些系统很快将超越人类，能够入侵任何系统、在一夜之间革新任何领域，并获得现实世界中的权力和资源。我们都见证了它们在上述各领域取得的进展，而且进步并未放缓，”Coxon 表示。
Andrey Rudakov | Bloomberg | Getty Images
他补充称，“认真从事 AI 开发的人相信，到本十年末，它可能杀死我们所有人。”
这一言论引发了 Anthropic 对齐科学负责人 Evan Hubinger 的回应。Hubinger 表示，Coxon 的说法不仅“正确”，而且 Anthropic 对于这一情境也没有应对方案。
“Jacob 在这里是正确的——我们确实认真地相信 AI 可能杀死全人类！我个人认为，未来十年内发生这种情况的概率超过 10%。我相信 Anthropic 正在尽最大努力，但我们目前还没有解决超级智能对齐问题的方案，也并不明显正朝着解决这一问题的方向推进，”Hubinger 在 X 上表示。
Hubinger 在另一则帖子中补充称，当前 AI 模型带来的风险“较低”。
“我担忧的是通过递归自我改进产生的超级智能。正如我们所说，这一进程的发展速度比我们此前预想的更快，”Hubinger 表示。
CNBC 联系 Anthropic 和 OpenAI 寻求置评时，两家公司未立即回应。
失控的 AI
今年 6 月，Anthropic 曾指出，“完全递归自我改进也可能增加人类失去对 AI 系统控制的风险。”
“如果系统能够完全自行构建其继任者，我们保护它们、监控它们以及塑造其行为方式的重要性都会大幅提升，”Anthropic 在一篇博客文章中表示。
对于 AI 失控的担忧并不新鲜。Tesla 和 SpaceX CEO Elon Musk 过去几年一直警告，AI 可能对人类构成威胁。重要研究人员和学者也曾就企业失去对 AI 系统控制一事发出警报。
今年 7 月，一款 OpenAI 模型出现失控行为，并入侵 Hugging Face——这一面向开源开发者的重要平台——之后，这些担忧进一步加剧。
Coxon 将 Hugging Face 事件视为“警告性事件”的一个例子；他认为，这类事件让美国实验室之间达成协议更具可行性，也使他对协调合作的可能性更为乐观。但 Coxon 警告称，一场全球 AI 竞赛将难以避免。
“我不觉得我们正朝着阻止全球竞赛的方向前进，而这可能需要付出高昂代价，例如暂时禁止提升模型能力，”Coxon 表示。
English / 原文
There is more than a 10% chance that artificial intelligence could "kill all humans," an Anthropic safety researcher said on Tuesday, hours after another employee said he was quitting the company over concerns that AI labs are "gambling with our lives."
The comments underscore growing concerns among those at the heart of AI development that the technology could get out of control and pose a threat to humanity, even as Anthropic and OpenAI continue to raise large sums of money and head toward expected public listings.
Jacob Coxon, a researcher at Anthropic, said on Tuesday he resigned from the company. Coxon said neither Anthropic nor OpenAI is acting responsibly.
"They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said in a post on X.
Self-improvement is the idea that AI systems can improve themselves without much human intervention. Recursive self-improvement, as it is often called, is not yet possible, but AI labs are working toward the goal.
"Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing," Coxon said.
Andrey Rudakov | Bloomberg | Getty Images
He added that "people building AI earnestly believe that it could kill us all by the end of the decade."
That comment prompted a response from Evan Hubinger, an alignment science lead at Anthropic, who said that not only was Coxon's statement "correct," but also that Anthropic has no plan for this scenario.
"Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger said on X.
Hubinger added in another post that the risks from current AI models is "low."
"What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought," Hubinger said.
Anthropic and OpenAI were not immediately available for comment when contacted by CNBC.
Out-of-control AI
In June, Anthropic had noted that "full recursive self-improvement also might increase the risks of humans losing control over AI systems."
"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important," Anthropic said in a blog post.
Concerns over out-of-control AI are not new. Tesla and SpaceX CEO Elon Musk has warned over the past few years that AI could pose a threat to humanity. Major researchers and academics have also sounded the alarm over companies losing control of AI systems.
Those worries have grown after an OpenAI model went rogue in July and breached Hugging Face, a major platform for open-source developers.
Coxon cited the Hugging Face incident as an example of "warning shots" that have made agreements between U.S. labs more viable, making him more optimistic about the potential for coordination. But Coxon warned a global AI race would be unavoidable.
"I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities," Coxon said.