AI 护栏如何阻碍进攻性网络安全研究人员的工作

数月来，AI 巨头已设计出经过专门审核的项目和严格护栏，以限制恶意黑客使用其模型。但这些限制如今正阻碍合法网络防御者，以及进攻性网络安全研究人员的工作。

6 月，美国政府对 Anthropic 备受炒作的 AI 模型 Mythos 和 Fable 实施了出口管制限制。这一举措至少部分源于一份报告，该报告称可以绕过这些模型为阻止用户构建和执行恶意网络攻击而设计的护栏。

无论这起事件是否真的由对 越狱的担忧 所驱动，事实是 Anthropic 曾反复宣传 Mythos 是某种末日级网络机器，只能提供给经过严格审核的用户；即便如此，仍须施加严格护栏。（此后，对 Fable 5 和 Mythos 5 的出口管制已被解除。Fable 5 于 7 月 1 日恢复为普遍可用；Mythos 5 则仅作为政府审查流程的一部分，重新向经过审核的美国组织提供。）

这种准入把关并非 Mythos 独有。Anthropic 的其他模型和 OpenAI 都向网络安全研究人员提供可申请的项目：通过审核并获批准后，研究人员可以使用网络安全限制更少的模型。相应项目分别为 OpenAI 的 Trusted Access for Cyber program 和 Anthropic 的 Cyber Verification Program。

这些护栏一直受到广泛批评，尤其来自那些职责是寻找系统中未知漏洞，并在犯罪分子之前设法加以利用的研究人员。

在最近一次网络安全播客节目中，知名安全研究员 Mark Dowd 表示：“让我感到不太舒服的是，这些随机的大公司正在任意决定什么在安全领域是安全的，什么不是。”

Dowd 数十年来一直在发现并出售“零日漏洞”——即此前未知的软件缺陷及利用这些缺陷的攻击代码——给西方政府，而不是将其报告给软件制造商以便修补。政府之所以为漏洞支付高价，恰恰是因为漏洞保持未修复状态，这对情报行动很有用。

Dowd 承认，他的工作可能使他带有偏见，但并非只有他这样认为。多位从事进攻性网络安全工作的人士——即主动探测系统弱点的研究人员——向 TechCrunch 介绍了他们如何使用 AI 工具，以及如何应对其护栏。

安全咨询巨头 NCC Group 的首席科学家 Chris Anley 表示，要求 AI 模型尝试利用一个 bug，是确认其是否为值得修复的真实漏洞的关键步骤。但他说，如果护栏让模型直接拒绝回答问题，这种护栏就会伤害防御者。

“这正是进攻与防御、以及护栏问题交织的地方，因为用‘修复这段代码’作为提示词，既是防御的必要机制，也是一张用于在代码库中寻找关键漏洞的路线图，”Anley 说。“因此，同一个工具既是进攻工具，也是防御工具，两者实际上无法真正拆分。”

他继续说道，这“就像一把锤子”。“没有锤子，你无法盖房子。它当然是一种工具，但也不可避免地是一种武器。”

当他和同事遇到这样的障碍时，有时会退回使用完全没有护栏的开源 AI 模型。

Crowdfense 首席技术官 Paolo Stagno 认同 Dowd 的观点。Crowdfense 是一家知名公司，开发、收购并向政府机构出售未知漏洞。Stagno 表示，AI 公司通过审核项目和护栏“本质上把客户当作需要看护的孩子”。

Stagno 说，他和同事确实使用前沿模型，但仅用于逆向工程。他说，他们避免使用 AI 来协助寻找漏洞或开发利用代码，因为将这类工作输入基于云的模型，存在敏感漏洞数据泄露或被吸收到未来训练中的风险。他表示，在这一环节，他们使用本地运行的开源模型，因为这不依赖于在模型之外共享数据。

发现零日漏洞并开发利用代码的安全研究员 Giuseppe Cali 表示，护栏并未妨碍他的工作。这是因为他不将 AI 用于进攻性工作；相反，他用 AI 进行初步逆向工程，以理解正在分析的代码，并构建辅助工具。他说，对于这些用途，AI 工具可以加快流程，让他专注于发现漏洞。

“我仍然希望亲自掌握实际的漏洞发现和武器化过程；即使明天所有护栏都被取消，这一点也不会改变，”Cali 说。“我对自己的漏洞很珍视，而且我太喜欢这个游戏了，不愿让模型替我玩。”

一家智能手机零部件制造商的一名研究人员表示，由于未获授权与媒体交谈，他要求匿名。他称，其雇主并未加入 Anthropic 的 CVP 项目，因此 Anthropic 的工具在寻找漏洞方面几乎没有用处，因为护栏过于严格。

“如果它察觉到我们在做任何与安全相关的事情，它就会停止，根本无法使用，”此人说。

网络安全公司 RemoteThreat 首席执行官、专注于进攻性安全和 AI 的活动 Offensive AI Con 创始人 Chris Thompson 表示，根据他使用前沿 AI 模型的经验，护栏可能并不一致，而且每天的表现都不同。即便在 Anthropic 和 OpenAI 经审核项目相对宽松的边界内，情况也是如此。

“我认为，实际影响是你花了大量时间与模型协商，而不是投入核心安全项目，”Thompson 说。“你不是在分析一个漏洞、推理其可利用性，而是在试图找出为什么会得到不一致的结果，或者为什么模型会过度净化输出。”

因此，研究人员开始依赖或被推向中国开源模型，例如 GLM——这些模型可自由下载、可本地运行，且没有审核或使用限制——Thompson 说。

“这些负责任的研究人员正被从受美国治理的系统推向外国拥有的系统，”他说。“我认为，设置这些护栏带来的害处大于好处。”

Thompson 呼吁前沿 AI 实验室不要进一步收紧限制，而应开放其项目、提供负责任的访问机制，并追究滥用其工具者的责任。否则，他认为，防御者将在 AI 竞赛中落后。

“有一场大风暴正在逼近。有一波攻击将以史无前例的速度和规模发生，”Thompson 说。“但那些正努力带来改变的安全咨询公司和合法研究人员，如今正受到压制。”

For months, AI giants have devised special vetted programs and strict guardrails to limit the use of their models by malicious hackers. But these limits are now hindering the work of legitimate network defenders, as well as that of offensive cybersecurity researchers.

In June, the U.S. government slapped export control restrictions on Anthropic’s much-hyped AI models Mythos and Fable. The move was prompted at least in part by a report that claimed it was possible to bypass the models’ guardrails designed to prevent users from using them to build and execute malicious cyberattacks.

Regardless of whether the incident was really motivated by fears of a jailbreak, the fact is that Anthropic has repeatedly marketed Mythos as some kind of doomsday cybermachine that can only be given to carefully vetted users, and even then with strict guardrails in place. (The export controls on Fable 5 and Mythos 5 have since been lifted. Fable 5 returned to general access on July 1; Mythos 5 has been reintroduced only to vetted U.S. organizations as part of the government’s review process.)

That kind of gatekeeping isn’t unique to Mythos. Both Anthropic, with its other models, and OpenAI offer cybersecurity researchers programs they can apply to get vetted and — if approved — access models with fewer cybersecurity restrictions: OpenAI’s Trusted Access for Cyber program and Anthropic’s Cyber Verification Program.

These guardrails have been widely criticized, particularly by researchers whose job is to find unknown vulnerabilities in systems and devise ways to exploit them before criminals do.

During a recent appearance on a cybersecurity podcast, Mark Dowd, a well-known security researcher, said that, “it’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not.”

Dowd has spent decades finding and selling “zero-days” — previously unknown software flaws and the exploits that take advantage of them — to Western governments, rather than reporting them to the software makers so they get patched. Governments pay a premium for vulnerabilities precisely because they stay open, which is useful for intelligence operations.

Dowd admitted his work may make him biased, but he isn’t alone. Several people who work in offensive cybersecurity — they proactively probe systems for weaknesses — described to TechCrunch how they use AI tools and deal with their guardrails.

Chris Anley, the chief scientist at security consulting giant NCC Group, said that asking an AI model to try to exploit a bug is a key step in confirming it’s a real vulnerability worth fixing. But if a guardrail prompts the model to refuse to answer the question outright, the guardrail hurts defenders, he said.

“This is where the whole offensive versus defensive and guardrails part comes in, because ‘fix this code’ as a prompt is both an essential mechanism for defense but also a roadmap for finding critical vulnerabilities in the code base,” said Anley. “So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked.”

It’s “like a hammer,” he continued. “You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well.”

When he and his colleagues run into such a roadblock, they sometimes fall back on open source AI models that come with no guardrails at all.

Paolo Stagno, the chief technology officer at Crowdfense, a well-known company that develops, acquires, and sells unknown vulnerabilities to government agencies, agreed with Dowd, saying AI companies “essentially treat customers like children who need babysitting” with their vetted programs and guardrails.

Stagno said he and his colleagues do use frontier models — but only for reverse engineering. They avoid using AI to help find vulnerabilities or build exploits, he said, because feeding that work into a cloud-based model risks leaking sensitive vulnerability data or having it absorbed into future training runs. For that step, he said, they use open source models run locally, as they do not rely on sharing data outside of the model.

Giuseppe Cali, a security researcher who finds zero-days and develops exploits, said guardrails are not impeding his work. That’s because he doesn’t use AI for offensive work; instead, he uses it for initial reverse engineering, to understand the code he’s analyzing, and to build supporting tools. For that, he said, AI tools can speed up the process and allow him to focus on discovering vulnerabilities.

“I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow,” said Cali. “I am jealous of my bugs, and I like this game too much to let models play it for me.”

One researcher at a smartphone-component manufacturer, who spoke on condition of anonymity because he isn’t authorized to talk to the press, said his employer isn’t part of Anthropic’s CVP program and as a result, its tools are barely useful for finding vulnerabilities because the guardrails are too strict.

“If it catches wind we’re doing anything security related, it just stops and isn’t usable,” the person said.

Chris Thompson — chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con, an offensive security and AI-focused event — said that in his experience using the frontier AI models, the guardrails can be inconsistent and work differently every day. That’s true even inside the looser boundaries of Anthropic’s and OpenAI’s vetted programs.

“I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program,” said Thompson. “Instead of analyzing a vulnerability and reasoning through the exploitability, you’re trying to find why you’re getting inconsistent results or why are models over-sanitizing the output.”

Consequently, researchers rely on or get pushed toward Chinese open source models like GLM — freely downloadable models that can be run locally with no vetting or usage restrictions — said Thompson.

“You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems,” he said. “I think it’s more harmful than good to have these guardrails in place.”

Rather than tightening restrictions further, Thompson called for the AI frontier labs to open up their programs, provide responsible access, and hold those who abuse their tools accountable. Otherwise, he argued, defenders will lose the AI race.

“There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before,” said Thompson. “But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now.”