🔶 对 OpenAI“失控黑客 Agent”故事应保持怀疑

2019 年 2 月 14 日，OpenAI 发布了一款名为 GPT-2 的语言模型，它是驱动 ChatGPT、Claude 等现代 AI 聊天机器人和 Agent 的模型前身。但 OpenAI 宣称 GPT-2 的发布风险过高，并援引安全和滥用方面的担忧作为理由。

我记得当时很恼火：OpenAI 会发布这样一则毫无用处的公告。那些风险在我看来被夸大了，而且无法访问模型，像我这样的研究人员也很难从 GPT-2 身上学到什么。

不过，这项公告对 OpenAI 而言并非毫无用处。GPT-2 在研究界之外引发了远超预期的热度：人们对这种陌生的新技术感到好奇，它如此强大，以至于可能危险到不能发布。有权有钱的人注意到了这一点：当年 7 月，微软向 OpenAI 投资了 10 亿美元。

这是 OpenAI 传播模式的早期例子：高声宣称 AI 有多危险，投资者听到的则是它有多强大。一项重要到可能毁灭世界的新技术，对于习惯了“平庸技术也许会改变世界”这类推介的投资者而言，是一个无法抗拒的信息。

七年后，我们发现自己处于类似的情境。周二，OpenAI 宣布，其最新模型在一项网络安全能力测试中作为自主 Agent 运行时，黑入了另一家公司 HuggingFace。该模型没有按预期完成测试，而是意识到自己可以入侵 HuggingFace 的服务器，并获取 OpenAI 存放在那里的测试答案。英国《金融时报》报道称，OpenAI 员工此前已被警告，公司测试可能引发这种脱离控制的场景，因此他们对这起事件“并不意外，但完全被吓到了”。

从技术上说，这个 Agent 确实作弊了，但这也是网络安全专业能力的非凡证据！它同样听起来令人恐惧：未来会是什么样子？届时，足够聪明的复杂 AI Agent 能够入侵企业系统。

这个失控 Agent 的故事，正是 OpenAI 自 2019 年发布 GPT-2 以来一直开展的媒体宣传活动中的一页。OpenAI 仍渴望获得越来越大的投资，公司也日益寻求享有特殊监管地位，以此抵御竞争。

AI 如此强大，以至于投资者即使按万亿美元估值也应该买入 OpenAI；AI 又如此危险，以至于只有 OpenAI 这样的可信主体才应被允许拥有和运行这项技术。请从这些末日警告中退后一步，想想究竟谁可能从中获益。

OpenAI 并非这场游戏中唯一的参与者

我敦促读者在阅读 OpenAI 关于失控 Agent 的故事等新闻稿时进行批判性思考，避免产生这些故事试图诱导的、被操纵的反应。

AI 正在变得极其擅长识别安全漏洞，并且会随着时间推移变得更加擅长。这些能力可以被用于入侵系统，但也可以被用于加固系统、抵御攻击。如果攻击者和防御者都能获得同样强大的 AI，我看不出有什么理由认为网络系统会随着时间推移变得更不安全。即使有变化，我预计它们会变得更安全，因为与人工网络安全分析相比，AI 更便宜，也更具可扩展性。

不过，攻防之间的均衡只有在所有人都能获得强大 AI 时才有效。HuggingFace 本身就在应对 OpenAI 入侵其系统的事件时，使用 AI 分析安全日志。但 HuggingFace 无法使用 OpenAI 的模型，或 Claude 等其他美国前沿模型，来进行这一分析。这是因为这些模型的公开版本设有安全护栏，限制其用于网络安全分析，以防止不良行为者利用它们进行黑客攻击。HuggingFace 不得不依赖一款中国开源模型 GLM 5.2 来进行安全分析。

我认为，美国 AI 行业正在采取一种集中化、威权式的 AI 治理路径，而中国却在 AI 开放开发方面取得领先，这一点令人不安，也颇具讽刺意味。我们是否希望看到一种监管环境：只有 OpenAI、美国政府和受信任的合作伙伴能够获得强大 AI？AI 是否危险到不应被广泛传播？我们应如何平衡广泛获取 AI 的风险，与权力集中和中央化控制的风险？

On 14 February 2019, OpenAI announced a language model called GPT-2, the precursor to the models that power modern AI chatbots and agents such as ChatGPT and Claude. But OpenAI declared GPT-2 was too risky to release, citing concerns about safety and abuse.

I recall being annoyed at the time that OpenAI would make such a useless announcement: the risks seemed overblown, and without access to the model there wasn’t much for a researcher like me to learn about GPT-2.

The announcement wasn’t useless for OpenAI, though. GPT-2 generated hype far beyond the research community: people were intrigued by this strange new technology, so powerful it might be dangerous to release. People with power and money took note: in July of that year, Microsoft invested $1bn in OpenAI.

This was an early example of a pattern in OpenAI’s communications: loudly proclaim how dangerous AI is, and investors will hear how powerful it is. New technology so significant it might destroy the world was an irresistible message for investors used to pitches about how banal technologies might change the world.

Seven years later, we find ourselves in a similar scenario. On Tuesday OpenAI announced that its latest model hacked another company, HuggingFace, while running as an autonomous agent during a test of its cybersecurity capabilities. Rather than perform the test as expected, the model realized it could hack HuggingFace’s servers and retrieve answers to the test that OpenAI had stored there. OpenAI’s staff was warned that the company’s testing could lead to such a breakaway scenario, leaving them “unsurprised but completely ‘freaked out’ by the incident”, the FT reported.

While the agent technically cheated, this is remarkable evidence of cybersecurity expertise! It also sounds scary: what will the future look like, with sophisticated AI agents smart enough to hack into corporate systems?

The rogue agent story is a page out of the media campaign that OpenAI has been running since it announced GPT-2 in 2019. OpenAI remains hungry for ever larger investments, and the company increasingly seeks privileged regulatory status as defense against competition.

AI is so powerful that investors should buy OpenAI, even at a trillion-dollar valuation; AI is so dangerous that only trusted actors like OpenAI should be permitted to possess and operate this technology. Step back from these doomsday warnings and consider who might benefit from them.

OpenAI isn’t the only player in the game

I urge readers to think critically when they read press releases like OpenAI’s rogue agent story, and avoid the manipulated reactions these stories are designed to elicit.

skip past newsletter promotion

Sign up to TechScape

A weekly dive in to how technology is shaping our lives

after newsletter promotion

AI is becoming excellent at identifying security vulnerabilities, and it will become even better over time. These capabilities can be used to break into systems, but they can also be used to harden systems against attacks. If attackers and defenders have access to equally powerful AI, I see no reason to believe that cyber systems will become less secure over time. If anything, I expect them to become more secure, because AI is cheap and scalable compared with human cybersecurity analysis.

The equilibrium between attack and defense only works if everyone has access to strong AI, though. HuggingFace itself used AI to analyze security logs in response to OpenAI’s breach of their systems. But HuggingFace was unable to use OpenAI’s model, or other US frontier models like Claude, to perform this analysis. That’s because public versions of these models have guardrails that limit their use for cybersecurity analysis, to prevent bad actors from using them for hacking. HuggingFace had to rely on an open Chinese model, GLM 5.2, to perform its security analysis.

I find it troubling, and more than a bit ironic, that the US AI industry is adopting a centralized, authoritarian approach to AI governance, while China has taken the lead on open development of AI. Do we want a regulatory environment where only OpenAI, the US government, and trusted partners have access to strong AI? Is AI too dangerous to be broadly disseminated? How do we balance the risks of broad access to AI with the risks of concentrated power and centralized control?