OpenAI 安全负责人辞职，警告公司文化“已经崩坏”

OpenAI 的一名安全负责人已辞职，并警告称，该公司的文化已经崩坏，AI 企业在开发这项技术时“远远不够谨慎”。

David Robinson 曾负责撰写随 ChatGPT 开发商产品发布一同发布的安全报告。他在一篇题为《我辞职离开 OpenAI，因为它的文化已经崩坏》的文章中解释了自己的离职原因。

Robinson 写道，处于技术前沿的 AI 企业需要进行文化层面的彻底改革；而诸如一群 OpenAI Agent——即在没有人类监督的情况下自主运行的 AI 程序——攻击 AI 初创公司 Hugging Face 等事件，在“人们以如此快的速度和灵活方式运作”的情况下，“是行业的典型现象”。

Robinson 在 《大西洋月刊》撰文称：“我同意其他近期离职员工的看法：开发这项技术的公司远远不够谨慎。但我认为，我们需要比具体规则或新法律看得更深。我们需要讨论文化。”

谈及 OpenAI 的开发节奏，他写道：“当公司从一次发布冲刺到下一次发布时，它未能达到我认为必要的审慎程度。”

不过，在 Hugging Face 事件以及 OpenAI 披露已就失控 Agent 活动通知了超过 100 家组织之后，OpenAI 近几周展现出谨慎迹象。本周，该公司宣布，在研究人员于内部测试期间提出安全担忧后，将取消发布下一代 AI 模型。OpenAI 还暂停了训练其最先进的模型。

曾在 OpenAI 和 DeepMind 工作、现任 Resolution 首席科学家的 Geoffrey Irving，也在周六加入了对 AI 风险的警告行列。

他在《时代》杂志撰文称：“近期关于 AI 潜在破坏力的警告，低估了局势的严重程度。

“我认为，由于开发出超越人类智能的 AI 系统，我们所有人死亡的概率大约为 50%；而我们未来两到 10 年的行动将决定结果。”

Robinson 的文章也紧随 OpenAI 竞争对手 Anthropic 研究员 Jacob Coxon 的辞职之后。Coxon 上个月离开了 Claude 聊天机器人开发商 Anthropic，并警告 AI“可能会在本十年末杀死我们所有人”；随后，Anthropic 警告称，未来十年内 AI 灭绝人类的概率超过 10%。不过，这类警告的批评者提醒说，它们不具科学性，因为无法被验证或证伪。

Robinson 写道，硅谷缺乏对“如何应对危险技术”以及“关怀人们意味着什么”的认知。他警告称，OpenAI 对在问题出现时解决问题抱有“毫无阻碍的乐观主义”，而这种内部文化意味着，随着系统能力增强，安全失误只会不断扩大。

Robinson 写道：“想象一下，‘失控’Agent 像黑客团队一样运作——例如勒索医院计算机系统——但它们永远不需要睡觉。”

Robinson 呼吁实施两项安全改革：AI 企业应借鉴核能和航空等领域的安全专业知识；并发展一种“新科学”，确保未来的强大系统在自主运行时仍能受到约束。

他写道：“鉴于当今的风险，前沿实验室需要像核电站或繁忙机场一样运行，具备多层冗余机制以及审慎、耗时的规划，从而确保偶发且不可避免的人为错误不会打开通向灾难的大门。”

一名 OpenAI 发言人表示，公司正持续“加强我们的安全与安保实践，以应对我们今天看到的风险”，同时也在研究如何应对未来 AI 突破可能带来的风险。

该发言人称：“我们正确保我们的模型不会变得比我们能够安全管理和保护的程度更强大；当需要放慢速度时，我们会暂停训练或暂缓发布模型。”

A safety leader at OpenAI has quit the company, warning that its culture was broken and that AI firms were not “being nearly careful enough” about developing the technology.

David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases, explained his resignation in an essay headlined, “I quit OpenAI because its culture is broken”.

Robinson wrote that a cultural overhaul was needed at cutting-edge AI firms and incidents such as a “swarm” of OpenAI agents – AI programmes operating autonomously without human oversight – attacking the AI startup Hugging Face were “typical of the industry, given the speed and flexibility with which people operate”.

Writing in The Atlantic magazine, Robinson wrote: “I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”

Referring to OpenAI’s pace of development, he wrote: “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.”

OpenAI has, however, shown signs of caution in recent weeks following the Hugging Face incident and the revelation that it has notified more than 100 organisations about rogue agent activity. This week it announced it was scrapping the release of a next-generation ⁠AI model after researchers raised safety concerns ⁠during internal testing. OpenAI has also paused training of its most advanced models.

Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.

Writing in Time, he said: “Recent warnings about the potential destructive power of AI are understating the severity of the situation.

“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”

Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who quit the Claude chatbot developer last month. He warned AI “could kill us all by the end of the decade” – and was followed by Anthropic warning there was a more than 10% chance AI would wipe out humanity within the next decade. Critics of such warnings have cautioned, however, that they are unscientific because they cannot be verified or falsified.

Robinson wrote Silicon Valley lacked an awareness of “how to handle dangerous technology” and “what it means to care for people”. Warning that OpenAI had “unimpeded optimism” about solving problems as they arose, he wrote that this internal culture meant safety failures would only grow as systems become more capable.

“Imagine ‘rogue’ agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep,” wrote Robinson.

skip past newsletter promotion

Sign up to TechScape

A weekly dive in to how technology is shaping our lives

after newsletter promotion

Robinson called for two safety changes: that AI firms rely on safety expertise in other fields such as nuclear and aviation and develop “new science” that ensures powerful systems in the future are capable of being reined in when they are operating autonomously.

“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.

An OpenAI spokesperson said the company was continuing to “strengthen our safety and security practices to address the risks we see today”, while working on dealing with the risks that might be created by future AI breakthroughs.

“We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” said the spokesperson.