Anthropic 和 OpenAI 希望引入驻场安全评估员，但他们真的能保持独立吗？
英文标题：Anthropic and OpenAI want to embed safety evaluators. Will they really be independent?
Anthropic CEO Dario Amodei 提议让第三方安全评估机构入驻前沿 AI 公司，获得系统访问权、调查安全事件，并可不经公司编辑公开关键发现；OpenAI CEO Sam Altman 也表示将采纳这一做法。METR、Redwood Research、Apollo Research 和 FAR.AI 等研究者欢迎更深入的访问权限，但警告评估机构若受制于 NDA、时间限制和企业披露控制，可能仍只是供应商而非独立监督者。研究者认为，评估应覆盖训练过程中的中间检查点、日志和内部环境，而不能仅测试最终模型，以防模型学会识别评测并伪装安全行为。加州和欧盟已开始建立第三方验证、风险评估和事故报告机制，但业内人士认为，真正独立的监督最终仍需要透明标准和强制性监管。
3.1 KB
389 B
74.3 KB
5.4 KB
202.2 KB
248 B
中文
在周末发表的一篇长篇文章中，Anthropic CEO Dario Amodei 提出了一项即使在一年前也会被 AI 行业立即否决的建议：让第三方评估机构入驻所有前沿 AI 公司，赋予它们报告安全事件、评估 AI 模型是否真正对齐，并向公众分享未经修饰的调查结论的权力。
Amodei 表示，Anthropic 将承诺向 METR 和 Redwood Research 等独立评估机构开放前所未有的系统访问权限。OpenAI CEO Sam Altman 表示，OpenAI 也将承诺采用这一做法，这预示着该行业与外部研究机构合作的方式可能发生深刻变化。
接受 TechCrunch 采访的第三方评估机构总体欢迎这一提议，但表示仍需敲定细节——最好以立法作为后盾——才能确定它们究竟会成为真正独立的监督者，还是按照 AI 公司条件运作的供应商。
随着模型越来越擅长识别自己是否正在接受评估，这种更深入的访问权限正变得愈发重要。这提高了一种风险：模型可能在测试期间表现良好，却隐藏有问题的行为。研究人员表示，只测试最终完成的模型时，可能会遗漏这类行为的线索；而通过调查模型在整个训练过程中的表现，则可能发现这些线索。
“A I 公司应该能够回答一些关于其训练过程的非常基本的问题，例如：AI 在训练过程中，是否曾主动试图破坏其自身的对齐训练？”Apollo Research 研究负责人 Alexander Meinke 对 TechCrunch 表示。“对此的答案应该是明确的‘没有’，而目前我们完全依赖 AI 公司自行认真检查这一点，然后如实向公众报告。近期事件已经表明，默认情况下，它们两者都不会做。作为驻场评估员，我们实际上可以进行核查。”
从历史上看，AI 公司通常会在模型发布前不久邀请外部审查人员测试已经完成的模型。如今，接受 TechCrunch 采访的评估人员提议，应允许他们不仅访问最终模型，还能访问模型整个训练周期中的中间版本，即“检查点（checkpoints）”。FAR.AI CEO Adam Gleave 表示，评估人员可以比较这些检查点，以确定令人担忧的行为在何时出现；检查通过奖励特定行为来塑造模型的后训练环境；并查看评估记录和日志，以核实公司有关模型表现的说法。
Anthropic 和 OpenAI 是否以及何时计划提供这类访问权限，目前尚不清楚。尽管 TechCrunch 多次询问，两家公司均未披露将与哪些评估机构合作、何时让其入驻、计划引入多少家机构、这些机构究竟能访问哪些系统和信息，以及哪些内容可以向公众披露。
这种深入查看内部机制的能力之所以重要，是因为即使模型在安全测试中表现优异，如果它专门学会了如何通过该测试，也不一定真的安全。Palisade Research 战略负责人 John Steidley 举例称，“抗关机基准测试（shutdown resistance benchmark）”用于衡量 AI 在特定情形下是否会抗拒被关闭。
“如果 AI 曾被专门训练以在该基准测试中取得好成绩，这一点就极其重要，”Steidley 说。他将其比作大众汽车的“柴油门”丑闻：汽车被编程为能够识别排放测试，并在测试条件下做出不同表现。
Gleave 指出，有意义的访问权限可以超出模型本身的范围，例如允许评估人员访谈员工，以核查公司的文档和对外公开的安全实践描述是否与内部实际发生的情况一致。
Amodei 的确概述了一项相当全面的提案，或许能赋予评估人员他们认为必要的访问权限，其中包括有权“在不受 Anthropic 编辑控制的情况下，发布关于风险等级、事件、实践，以及其获得或未获得的访问权限的关键发现”。
但评估人员表示，只有在 AI 公司真正愿意放弃对流程的控制时，这一体系才能运作。此前针对独立评估的尝试表明，这种让步将很难争取：第三方常常会在访问权限、时间、保密要求以及可公开发言的范围等问题上遭遇冲突。
Gleave 表示，FAR.AI 曾不得不拒绝多家前沿开发商提供的合同，因为这些公司希望对评估流程施加过多控制，从而威胁到该机构的独立性。他说，默认情况下，评估人员被视为普通承包商：受严格保密协议（NDA）约束，同时协议赋予开发商对最终可公开内容的重大控制权。
时间限制
另一个问题是，审查人员是否能够获得足够的时间和访问权限来完成被要求进行的工作。在调查 Hugging Face 事件时，OpenAI 向 METR 提供了大约一周的现场调查时间，也向 Redwood提供了类似条件；两家机构后来都表示，部分由于调查范围和时间限制，它们无法得出有把握的结论。
GPT-6 Astra 的发布前测试也出现了类似问题。OpenAI 将其宣传为迄今对齐程度最高的模型。根据 Apollo Research 对模型卡的贡献内容，该机构仅获准用三天时间测试 Astra，因此难以得出确定结论。
“Apollo 认为，鉴于更高的评估感知率以及有限的评估窗口，在这里观察到的低不当行为发生率，并不能为该模型是否对齐或失准提供实质性证据，”该机构在评估中写道。
这一过往记录令评估人员面临一个基本问题：为什么这一次会有所不同？
“当然，Dario 和 Sam 可能只是改变了想法，并将对此非常开放，”Gleave 说。“但这些公司的知识产权对它们而言价值极高，我认为默认情况下，它们会对哪些内容能够共享非常谨慎。”
多名接受 TechCrunch 采访的研究者呼吁建立一个各方都公开认可的透明框架。Steidley 表示，该框架的一部分应当包括针对公司可依赖何种审计机构的标准，以免公司通过挑选不具备资质、或不愿评估最令人担忧风险的评估机构来规避问题。
Safer AI 执行董事 Henry Papadatos 表示，即使存在公开框架，问题仍在于自愿措施始终依赖于公司的善意。
“理想情况下，我们应当有良好的监管来强制要求这一点……因为这样一来，公司在遭遇重大公关危机时就不能第二天改变主意，”Papadatos 对 TechCrunch 表示。他指出，这也是推动所有公司而非仅最愿意配合的公司遵守规则的有效方式。
并非所有公司都已加入。截至目前，Meta、SpaceXAI 和 Google DeepMind 尚未承诺引入第三方驻场评估员；不过，DeepMind CEO Demis Hassabis 提出成立一个独立测试前沿模型的行业标准机构。Google、OpenAI 和 Anthropic 此前也已就 AI 安全计划进行过数周的私下讨论。
一些法律已开始围绕第三方评估机构这一理念形成。加州去年签署成法的 SB 53，要求大型前沿 AI 开发商公布安全框架并报告重大安全事件。本月签署的新法 SB 813，则建立了一个由州政府认可、具备评估 AI 风险专业能力的“独立验证组织（independent verification organizations）”框架。
在欧洲，《欧盟 AI 法案》要求前沿开发商开展并记录模型评估和对抗性测试，并报告严重事件。欧盟 AI 办公室也可以自行开展评估并任命独立专家。
目前，法律的覆盖范围仍不及 Amodei 所提出的方案广泛，前沿实验室在很大程度上仍可自行决定接受多少独立审查。Papadatos 表示，自愿自律总比什么都不做要好，但归根结底，公司不能一边要求自由决定自己的安全规则，一边又要求公众相信它们正在遵守这些规则。
“你不能两者兼得：既完全没有外部问责，然后又说，‘我只需要有自己灵活的规则。’”Papadatos 说。
English / 原文
In a lengthy essay published over the weekend, Anthropic CEO Dario Amodei made a proposal that the AI industry would have rejected instantly even a year ago: embed third-party evaluators inside all frontier AI companies, giving them the power to report safety incidents, assess whether AI models are truly aligned, and share their unvarnished findings with the world.
Amodei said Anthropic would commit to giving independent evaluators like METR and Redwood Research unprecedented access to the company’s systems. CEO Sam Altman said OpenAI also would commit to the practice, signaling a potentially profound change in how the industry works with outside research groups.
Third-party evaluators who spoke to TechCrunch broadly welcomed the proposal, but said details need to be ironed out — and ideally backed by legislation — if they’re to know whether they will function as truly independent watchdogs or vendors operating on the AI companies’ terms.
That deeper access is becoming more important as models get better at recognizing when they’re being evaluated, raising the risk that they’ll behave well during testing while concealing problematic behavior. Researchers say clues to that behavior can be missed when testing the finished model, but uncovered by investigating how it behaved throughout training.
“AI companies should be able to answer some very basic questions about their training process, such as: Did the AI ever actively try to undermine its own alignment training while it was going through the training?” Alexander Meinke, head of research at Apollo Research, told TechCrunch. “The answer to this should be an unequivocal no, and right now we are completely relying on AI companies to both carefully check this themselves and then truthfully report this to the public. And we’ve seen from recent incidents that, by default, they will do neither. As embedded evaluators, we could actually check.”
Historically, AI companies brought in outside reviewers to test finished models shortly before their release. Now, evaluators that TechCrunch spoke to propose giving them access not just to the final model, but to intermediate versions, or “checkpoints,” from its lifetime of training. Adam Gleave, CEO of FAR.AI, said evaluators could compare those checkpoints to determine when concerning behavior emerged, inspect the post-training environment that rewards models for certain behaviors, and check evaluation transcripts and logs to verify a company’s claims about how a model performed.
Whether and when Anthropic and OpenAI plan to provide that kind of access is unclear. Neither company has shared which evaluators they’ll work with, when they will be embedded, how many they’ll bring on, exactly what systems and information they will be able to access or what can be disclosed to the public, despite repeated questions from TechCrunch.
Looking under the hood like this matters because models that perform well on safety tests aren’t necessarily safe if they’ve learned specifically how to pass that test. John Steidley, head of strategy at Palisade Research, pointed to an example of a “shutdown resistance benchmark” that measures if the AI will resist being shut down in certain circumstances.
“It’s extremely relevant if the AI has been trained specifically to perform well on that benchmark,” Steidley said, comparing it to Volkswagen’s Dieselgate scandal, in which cars were programmed to recognize emissions tests and perform differently under testing conditions.
Gleave noted that meaningful access could extend beyond the models themselves, with evaluators being given access to interview employees to check whether a company’s documentation and public descriptions of its safety practices match what happened internally.
Amodei did outline a fairly comprehensive proposal that might give evaluators the kind of access they think is necessary, including the right to “publish key findings about risk levels, incidents, practices, and the access they received or didn’t receive — without editorial control by Anthropic.”
But evaluators say such a system will only work if AI companies are actually willing to surrender control over the process. Previous efforts at independent evaluations suggest that that surrender will be hard won, as third parties have often run up against tensions over access, time, confidentiality, and what they can say publicly.
Gleave said FAR.AI has had to turn down contracts with several frontier developers that wanted too much control over the evaluation process, threatening the firm’s independence. By default, he said evaluators are treated like ordinary contractors: bound by restrictive NDAs and agreements that give developers significant control over what can ultimately be published.
The time limit
There’s also the question of whether reviewers will get enough time and access to do the work they’re being asked to do. When investigating the Hugging Face incident, OpenAI gave METR and Redwood roughly a week on premises to investigate, and both later said they could not draw confident conclusions due, in part, to scope and timing limitations.
A similar issue occurred during the pre-release testing for GPT-6 Astra, which OpenAI has touted as its most aligned model yet. According to Apollo Research’s contribution to the model card, the firm was given only three days to test Astra, which made it difficult to draw firm conclusions.
“Apollo believes that, given the higher rates of eval awareness and limited evaluation window, low rates of misbehavior here do not provide substantial evidence about the model’s alignment or misalignment,” the firm wrote in its evaluation.
That track record leaves evaluators with a basic question: Why should this time be different?
“It’s certainly possible that Dario and Sam just had a change of heart, and they’re going to be very open about this,” Gleave said. “But the intellectual property of these companies is so incredibly valuable to them, and I think they’re going to, by default, be very careful about what can be shared.”
Several researchers who spoke to TechCrunch called for a transparent framework that they all agree to publicly. Part of the framework, says Steidley, should involve standards for what kinds of auditors companies can rely on, lest they try to sidestep the issue by shopping for evaluators that either aren’t qualified or aren’t interested in assessing the most concerning risk.
Henry Papadatos, executive director of Safer AI, says the problem, even with a public framework, is that voluntary measures are always dependent on a company’s goodwill.
“Ideally, we would have good regulation mandating this…because then companies cannot change their mind tomorrow if they have a big PR crisis,” Papadatos told TechCrunch, noting that it’s also a good means of pushing all companies to adhere to the rules, not only the most willing.
Not everyone has signed on. So far, Meta, SpaceXAI, and Google DeepMind have not committed to embedding third-party evaluators, though DeepMind CEO Demis Hassabis has proposed a separate industry standards body to independently test frontier models. Google, OpenAI, and Anthropic have also privately been discussing AI safety plans for weeks.
Some laws are already forming around the idea of third-party evaluators. California’s SB 53, signed into law last year, requires large frontier AI developers to publish safety frameworks and report critical safety incidents. A new law, SB 813, signed this month, creates a framework for state-recognized “independent verification organizations” with expertise assessing AI risks.
In Europe, the EU AI Act requires frontier developers to conduct and document model evaluations and adversarial testing and report serious incidents. The EU AI Office can also conduct its own evaluations and appoint independent experts.
For now the law remains less expansive than what Amodei is proposing, leaving frontier labs largely responsible for deciding how much independent scrutiny they will submit to. Papadatos said voluntary self-regulation is better than nothing, but ultimately, companies can’t demand the freedom to control their own safety rules while also asking the public to trust that they’re following them.
“You cannot have it both ways, having zero accountability externally, and then say, ‘I’ll just have my own flexible rules,” Papadatos said.