Anthropic 拟议的 AI 监督机构 METR 与有效利他主义关系深厚

担心 AI 会对世界造成什么影响？别担心——有一群“觉醒派精英”会保护我们。

这就是 Anthropic CEO Dario Amodei 的说法：他表示已找到一个外部团队，可以监管 AI，并确保它不会毁灭世界。

这是一个由“全球主义者”组成的团体，他们相信开放边界、无性别代词，并倡导“有效利他主义”（Effective Altruism）；该理念认为，世界上的所有问题都可以通过技术解决。

在一篇题为《我们必须为前沿发展定速》（“We Must Pace the Frontier”）的重磅文章中，Amodei 主张，领先 AI 公司应承诺允许来自非营利组织模型评估与威胁研究机构（Model Evaluation and Threat Research，METR）的“嵌入式第三方评估人员”介入。该机构将保持“员工级访问权限”，监督安全工作，并确保先进 AI 模型不会变成《终结者》那样、进而毁灭人类。

METR 中这些自诩为约翰·康纳式角色的人声称完全独立，但该组织的根源可直接追溯至有效利他主义的关键人物。有效利他主义在硅谷拥有众多拥趸，其中曾包括失势的加密货币大亨 Sam Bankman-Fried——他也是 Anthropic 最大的早期投资者之一。

Anthropic 联合创始人兼 CEO Dario Amodei 主张由非营利组织模型评估与威胁研究机构派出第三方 AI 审查人员。REUTERS

有效利他主义是一种在硅谷科技巨头中流行的理念，主张运用理性——以及科技界精英积累的财富——在长期内为人类实现最大的善。

过去，这一理念曾被用于为“尽可能多赚钱”辩护，理由是日后可以捐赠更多资金、从而做更多好事。不过，当该运动最知名的倡导者之一 Bankman-Fried 的加密货币帝国在欺诈指控中崩塌并导致其入狱后，这场运动的声誉也遭受重创。

与 EA 的深厚联系

批评者称，METR 与 Anthropic 之间的交集充满潜在利益冲突。

METR 从一家与有效利他主义理念一致的科技孵化机构——对齐研究中心（Alignment Research Center，ARC）——分拆而出。ARC 由 AI 研究员 Paul Christiano 领导。2010 年代，两人都在 Anthropic 最大竞争对手 OpenAI 工作时，Christiano 曾是 Amodei 的室友、同事和研究合作者。

Ajeya Cotra 是 METR 的技术员工。METR

Christiano 还曾担任 Anthropic 长期利益信托（Long-Term Benefit Trust）最初五名受托人之一。

METR 的一名员工 Ajeya Cotra 是 Christiano 的妻子，并且曾是 EA 领域的重要组织者。

该组织的创始人兼 CEO 是 Beth Barnes；她由 Christiano 雇用，并在大学期间参与过 EA 运动。

首席科学家 Hjalmar Wijk，以及员工 Megan Kinniment 和 Pip Arnott，都来自牛津大学同一个研究集群——该校是 EA 人才的重要输送地。

一名 METR 官员在接受电话采访时承认，该非营利组织与有效利他主义有大量重叠，但坚称其员工拥有多元的意识形态观点。他们必须披露利益冲突，例如员工持有所受委托调查公司的股权。

“我们不认为任何小团体都应有权决定 AI 的发展走向；我们的目标是将这些公司内部的信息带入公共领域——外部调查人员可以帮助揭示更多信息。”一名 METR 发言人在声明中表示。

Beth Barnes 是 METR 的 CEO。METR

“我们的资助者无权决定我们开展的项目，而且我们不接受任何 AI 公司或其员工的资助。”

曾承诺捐出个人财富 80% 的 Amodei 与 EA 有着深厚联系。其妹妹 Daniela 嫁给了两家主要 EA 非营利组织的创始人 Holden Karnofsky。

METR 的参与很可能不会被特朗普政府接受；过去一年，该政府与 Anthropic 的关系紧张，并且对所谓的 AI“末日论者”持强烈怀疑态度。这些人认为，该技术正处在可能引发灾难的边缘。

一名熟悉白宫想法的消息人士周二对《纽约邮报》表示：“METR 本质上是 Anthropic 的另一种形式。”

周二，在旧金山 Moscone Center 举行的 Salesforce Dreamforce 大会上，Amodei（左）在主题演讲期间与 Salesforce CEO Marc Benioff 交谈。Getty Images

数千万美元融资

有效利他主义的坚定拥护者相信，应将财富投入他们认为对人类最重要的目标，包括气候变化和严格的 AI 监管。

自成立以来，METR 及其关联机构一直获得 EA 重要人物的资助，其中包括民主党超级捐助者 Dustin Moskovitz 和 Skype 联合创始人 Jaan Tallinn——两人也都是 Anthropic 投资者。

8 月，METR 宣布，其在此前六个月内从外部投资者处筹集了 7100 万美元新资金。

根据公开记录，Tallinn 的慈善组织 Survival and Flourishing Fund 自 METR 成立以来已划拨最高 75.2 万美元用于支持该机构，其中包括 32.4 万美元的一般现金资助，以及 42.8 万美元的配捐承诺。

此外，Moskovitz 的 Coefficient Giving——在去年仓促更名之前称为 Open Philanthropy——于 2022 年向 METR 的孵化机构 ARC 提供了 151.5 万美元。METR 最初名为 ARC Evals，后于 2023 年分拆为独立非营利组织，并改为现名。

图为 METR 员工 Pip Arnott。METR

这名 METR 官员表示，METR 不会就其工作从 AI 实验室收取任何报酬，也不接受 AI 公司高管或员工的捐款。对于 Moskovitz 在 2022 年向 ARC 的捐赠，该官员称这笔资金已被隔离，未用于 METR 的运营。

这几乎未能让批评者感到安心。

总部位于华盛顿特区的 AI 政策组织 Alliance for the Future 主席 Perry Metzger 表示：“说你希望由 METR 那些‘独立’人士审计，完全是个笑话。资助他们的人就是你的朋友。这完全荒谬。

“如果你明天出去，假设你是一家大银行，而你让一个完全由自己前员工组成、并由同样投资者资助的组织来做审计，你知道联邦政府会怎么对付你吗？”他补充道，“那会是极其严厉的。”

Amodei 上周六上午发布这一出人意料的提议，而就在几天前，曾在 OpenAI 工作的 Anthropic 研究员 Jacob Coxon 突然辞职，并警告两家 AI 巨头都在“拿我们的生命赌博”，原因是它们在缺乏适当监督的情况下追求进步。

Amodei 警告称，如果不建立保障措施，互联网可能在 6 至 12 个月内被 AI 机器人占据，“可能造成数千亿美元损失”。

“别再假装了”

这种末日警告在科技界和政界的一些圈子里遭到强烈反驳。特朗普总统称其为“骗局”，而规模较小的 AI 初创公司则认为，Anthropic 和 OpenAI 等领军企业只是试图改写法律、使其对自己有利。

Hjalmar Wijk 是 METR 的首席科学家。根据其 LinkedIn 个人资料，他目前从事“努力降低未来不同 AI 模型最坏情形风险”的工作。METR

白宫前 AI 沙皇 David Sacks 是怀疑者之一。他呼吁 Amodei“不要再假装 METR 是独立的，尽管它与 Anthropic 的投资者和员工盘根错节”。

他补充道：“最重要的是，别再假装放缓发展的动机纯粹是利他的。如果你的产品促成了一次真正具有破坏性的网络攻击，你将面临巨大的产品责任风险。”

目前也不清楚 AI 行业推动自我监管的做法，是否能通过美国监管机构的审查。

联邦贸易委员会（FTC）主席 Andrew Ferguson 警告称，对于任何要求给予 AI 公司反垄断豁免、以协调安全工作的请求，“所有人都应深表怀疑”。

Ferguson 周二在乔治城大学的一场活动中表示：“不过，我总体上会说，如果企业一边来到华盛顿要求出台大量监管措施，一边又要求反垄断豁免，我所有的警报都会响起。”

Jasmine Dhaliwal 是 METR 的政策员工。METR

根据 Amodei 的提案，第三方评估人员——一些人将其比作核查员——将“核实对安全实践和承诺的遵守情况，报告事件，并帮助评估的不仅是已完成的 AI 模型，还包括训练流水线和流程的对齐情况”，其文章写道。

METR 技术员工 Megan Kinniment 此前曾在伦敦长期风险中心（Center on Long-Term Risk）担任一系列职位。根据她的 LinkedIn 资料，她在那里专注于研究“恶意行为者如何增加不良 AI 结果发生的可能性”。METR

Amodei 写道：“Anthropic 现在单方面承诺采取这一步骤。我们希望这成为一项更广泛行动的一部分，以加倍推进我们的安全和对齐工作。”

OpenAI 的 Sam Altman 表示，他同意 Amodei 关于“需要为前沿发展定速”的观点，并补充称，“承诺让拥有员工级访问权限的独立评估人员参与，是一个好主意，我们也会这样做。”他没有提及 METR。

Anthropic 拒绝置评。

Jared Downing 和 Georgia Worrell 补充报道

Worried what AI will do to the world? Don’t fret — we got a group of woke elitists who will protect us.

That’s the pitch from Anthropic’s CEO Dario Amodei, who says he has found an outside team that can regulate AI and make sure it doesn’t destroy the world.

It’s a group of globalists who believe in open borders, genderless pronoun and promote “Effective Altruism,” that believe all the world’s problems can be solved with technology.

In a bombshell essay titled “We Must Pace the Frontier,” Amodei argued that leading AI firms should commit to allowing “embedded third-party evaluators” from a nonprofit called Model Evaluation and Threat Research, or METR, that would maintain “employee-like access” to oversee safety efforts and ensure advanced AI models don’t turn Terminator and wipe out humanity.

The would-be John Connors at METR purport to be fully independent, but the group traces its roots directly back to key figures in Effective Altruism, whose Silicon Valley-based devotees once included disgraced crypto boss Sam Bankman-Fried, one of Anthropic’s biggest early investors.

Anthropic co-founder and CEO Dario Amodei argues for third-party AI reviewers from a nonprofit called Model Evaluation and Threat Research. REUTERS

Effective Altruism, a philosophy that’s become popular with Silicon Valley masters of the universe, advocates for using reason — and fortunes amassed by tech’s best and brightest — to do the most good for humanity over the long term.

In the past, it has been used to justify making as much money as possible with the idea that you can donate more later to do more good – though the movement suffered a big black eye when one of its most famous proponents, Bankman-Fried, saw his crypto empire collapse amid fraud charges that sent him to prison.

Deep EA ties

The overlap between METR and Anthropic is rife with potential conflicts of interest, according to critics.

METR spun off from an Effective Altruism-aligned tech incubator called Alignment Research Center (ARC), which is headed by AI researcher Paul Christiano. He was Amodei’s housemate, coworker and research collaborator in the 2010s when they were both at OpenAI — Anthropic’s biggest rival.

Ajeya Cotra is a member of technical staff at METR. METR

Christiano also served as one of the first five trustees of Anthropic’s Long-Term Benefit Trust.

One staffer on METR is Ajeya Cotra, who is married to Christiano and has been a major organizer in the EA world.

The founder and CEO of the group is Beth Barnes, who was hired by Christiano and was involved in the EA movement at college.

Chief scientist Hjalmar Wijk and staffers Megan Kinniment and Pip Arnott all came from the same research cluster at the University of Oxford — a major feeder for EA.

When reached by phone, a METR official acknowledged that the nonprofit has a lot of overlap with Effective Altruism, but insisted its employees have a diverse array of ideological viewpoints. They’re required to disclose conflicts of interest, such as employees holding equity in firms they are tapped to investigate.

“We don’t think any small group should have authority over what happens with AI, and our goal is to get information from inside these companies into the public domain – external investigators can help bring more information to light,” a METR spokesperson said in a statement.

Beth Barnes is the CEO of METR. METR

“Our funders have no say in the projects we work on and we do not accept any funding from AI companies or their employees.”

Amodei, who has pledged to give away 80% of his personal wealth, has deep ties to EA**.** His sister Daniela married Holden Karnofsky, the founder of two major EA nonprofits.

METR’s involvement is likely to be a nonstarter for the Trump administration, which has had a testy relationship with Anthropic over the past year and has also expressed heavy skepticism of so-called AI “doomers” who argue the technology is on the precipice of disaster.

“METR is essentially Anthropic in another form,” a source familiar with the White House’s thinking told The Post on Tuesday.

Amodei, left, speaks with Salesforce CEO Marc Benioff during the keynote address at Salesforce’s Dreamforce conference at the Moscone Center on Tuesday. Getty Images

Multimillion-dollar fundraising

Effective Altruism’s obsessed adherents believe in funneling their wealth toward the goals they view as most important to humanity, including climate change and strict AI regulation.

Since its founding, METR and its affiliates have received funding from EA kingpins like Democratic megadonor Dustin Moskovitz and Skype cofounder Jaan Tallinn — both of whom are Anthropic investors.

In August, METR announced it had raised $71 million in new funding from outside investors over the previous six months.

Tallinn’s Survival and Flourishing Fund, a philanthropy group, has earmarked up to $752,000 in funding to support METR since its founding, including $324,000 in general cash and a $428,000 matching pledge, according to public records.

Elsewhere, Moskovitz’s Coefficient Giving – formerly known as Open Philanthropy before a hasty rebranding last year – gave $1,515,000 to METR’s incubator, the ARC, in 2022. METR was originally known as ARC Evals before spinning off as an independent nonprofit and changing to its current name in 2023.

METR staffer Pip Arnott is pictured. METR

METR does not take any compensation from AI labs for its work, nor does it take donations from executives or employees of AI companies, the METR official said. In the case of Moskovitz’s donation to ARC in 2022, the official said the funding was firewalled and not used for METR’s operations.

That’s hardly reassuruing to critics.

“Saying you want to be audited by the ‘independent’ people at METR is a complete joke,” said Perry Metzger, chairman of Alliance for the Future, a Washington, DC-based AI policy group. “It’s people who are funded by your buddies. It’s completely ridiculous.

“If you went out tomorrow and you’re a big bank and you had your audits done by an organization consisting entirely of your own former employees, which was funded by the same investors that are investing in you, do you know what the feds would do with you?” he added. “It would be positively medieval.”

Amodei’s surprise proposal published last Saturday morning came just days after Jacob Coxon, an Anthropic researcher who previously worked at OpenAI, abruptly quit the firm while warning that both AI giants were “gambling with our lives” by pursuing progress without proper oversight.

Without safeguards in place, Amodei warned the internet could be overtaken by AI bots within six to 12 months, “potentially causing hundreds of billions of dollars in damage.”

‘Stop pretending’

The doomsday warnings received sharp pushback in some corners of the tech and political world, with President Trump asserting that they were a “hoax” and smaller AI startups arguing that leaders like Anthropic and OpenAI were just trying to rewrite the law to benefit themselves.

Hjalmar Wijk is chief scientist at METR. He currently works at “trying to reduce worst-case risks from future models” of different AI models, according to his LinkedIn profile. METR

Among the skeptics was former White House AI czar David Sacks, who called on Amodei to “stop pretending METR is independent when it is interviewed with Anthropic’s investors and staff.”

“Most of all, stop pretending the motivation to slow down is purely altruistic,” he added. “You face massive product-liability exposure if your products enable a truly damaging cyberattack.”

It’s also unclear if the AI industry’s push to police itself will pass muster with US regulators.

Federal Trade Commission Chairman Andrew Ferguson warned that “everyone should be deeply suspicious” of any request to grant AI companies an antitrust exemption to coordinate safety efforts.

“I will say generally, though, that if companies are simultaneously coming to Washington and asking for a host of regulations and an antitrust exemption, all of my alarm bells go off,” Ferguson said during a Tuesday event at Georgetown University.

Jasmine Dhaliwal is a policy staffer at METR. METR

Under Amodei’s proposals, the third-party evaluators – which some have likened to nuclear inspectors – would “verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes,” according to his essay.

Megan Kinniment Kinniment, a member of METR’s technical staff, previously held a series of positions at the Center on Long-Term Risk in London, where she focused on how “malevolent actors could increase the likelihood of poor AI outcomes,” according to her LinkedIn. METR

“Anthropic is unilaterally committing to this step now,” Amodei wrote. “We intend this to be part of a broader push to redouble efforts on our safety and alignment work.”

OpenAI’s Sam Altman said he agreed with Amodei about the “need to pace the frontier” – adding that “committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” He made no mention of METR.

Anthropic declined to comment.

Additional reporting by Jared Downing and Georgia Worrell