🔶 比 Agent 接管互联网更可怕的是什么？CEO 卡特尔试图接管 AI

Anthropic CEO Dario Amodei 写道：

我相信，AI 在未来 5 至 10 年内可能治愈大多数重大疾病，极大加快经济增长速度，创造一个富足且赋能于人的世界，并开启民主与自由的复兴……

但与此前许多技术一样，AI 也会带来风险；而由于它是一项如此强大的技术，这些风险十分严重……

我们必须放缓提升 AI 模型能力的速度。

随后，Amodei 提出：

1. 嵌入式评估员（Embedded Evaluators）：他们在 AI 公司内部工作，并拥有类似员工的权限，可以接触公司正在开展的工作。他表示 Anthropic 致力于采取这一步骤，而 Sam Altman 很快同意 OpenAI 也会这么做。

2. 民主协调（Democratic Coordination）：“民主国家中的前沿 AI 公司进行协调，以建立共同的安全标准，并对不受约束的 AI 进步速度设置限制。某些能够有效控制发展节奏的协调形式在法律上存在挑战，需要政府支持。”他所说的政府支持，意味着放宽反垄断法；而反垄断法恰恰旨在禁止这种反竞争行为。

3. 全球协调（Global Coordination）：“美国和其他民主国家政府尝试与威权政府协调，在可能的范围内进行合作，同时认真对待验证合规性所面临的挑战。”

作为一个老家伙，我更希望我们不要放缓治愈大多数重大疾病的步伐。

即便是看似无害的建议——让中立的嵌入式评估员（仿佛真有这种东西）入驻 AI 实验室——也是迈向扼杀竞争的一步。对大公司而言，这几乎没有成本，甚至可能帮助它们规避部分责任；但对于一家未来可能挑战现有领导者的初创公司而言，这将是难以承受的负担。如果我创办一家只有一个人的 AI 公司（要是我年轻些，我会这么做），我是否还得花钱请几名“观察员”坐在我的车库里，占用宝贵的计算时间？不过，如果 Anthropic、OpenAI 和 xAI 希望在自己的实验室里安排观察员，或者想为自身安全责任获得其他帮助，那就随它们去。只是别让它们把观察员强加给竞争对手。

要求豁免反垄断法，让这些公司能够直接观察彼此的产品开发，本身就违背了该法律的目的。该法律将公司为避免竞争而共享信息列为刑事犯罪。这样做会让大公司放慢速度，同时——这是它们希望的——维持针对新进入者的卡特尔化领先地位；新进入者无法获得这些公司正在共享的信息。Elon Musk 的公司并不处于 AI 领域领先地位，但他很快同意，为了安全起见，它们应该每隔几周查看彼此的模型。他却没有向外部提供类似权限，查看 SpaceX 或 Tesla 的机器人技术（另一种形式的 AI），尽管装满燃料的火箭、汽车和卡车都可以像 9·11 事件中的飞机一样被武器化。我们绝不应该豁免反垄断法，让 Musk、Amodei 和 Altman 组成卡特尔。即使他们自己不这么认为，他们已经拥有足够多的市场力量。

关于在“安全”AI 开发上进行全球协调的提议，是这些公司的逃生出口。核试验禁令条约可以在一段时间内发挥作用，因为核试验可以通过地震仪侦测。即使中国同意这种安排（它们不会同意），我们要如何验证其是否遵守？或者从它们的视角（PoV）看，它们又如何验证我们没有作弊？伊朗会同意可验证的放缓吗？朝鲜呢？俄罗斯呢？又或者乌克兰呢？乌克兰已证明自己相当擅长武器开发，但又无力承受被俄罗斯甩在后面的后果。Amodei 很清楚这根本不会发生。他写道：

一种全面的节奏控制，甚至是“暂停”：参与国政府同意大幅限制 AI 整体发展速度。我支持提出这一设想，但我认为它短期内不太可能真正发生：通过逃避监测而违背此类协议，可能从根本上改变全球力量平衡，因此我预计违约的诱因会极其强烈，而我们对验证机制所需的信心程度将非常高。

世界不会因单方面的 AI 裁军而得救。随着坏人继续推进 AI 在进攻和防御方面的应用——而这两者往往无法分开——我们却承诺以近乎自杀式的更慢速度前进，世界将变得更加危险。

我当然不了解这些科技巨头的动机或长期盘算，但这里有一种猜测。他们目前面临两种危险。第一种是 Bernie Sanders 与 Steve Bannon 对其正在从事工作的禁令，以及／或者 AI 公司的国有化。虽然这些是糟糕的想法，但它们至少被部分实施确实存在一定风险。

第二种危险纯粹属于经济层面。直到最近，他们还以极高估值融资，基础是宏大的预期；各地政府则争相为数据中心建设提供补贴和税收减免。换言之，他们获得了大量免费资金。但随着建设数据中心的热潮推进，各地政府开始反弹。它们不仅不再提供补贴，还要求新数据中心建设方承担其所需基础设施的全部成本，甚至可能支付更多（参见 《数据中心交易的艺术》）。成本的增加，以及大型 AI 公司为维持竞争领先地位而必须进行巨额投资、却可能永远无法获得回报的困难，很快将在 Anthropic 和 OpenAI 为上市准备而必须发布的文件中变得非常清楚。换句话说，免费资金叮当作响的声音，正在变成震耳欲聋的沉默。更糟糕的是，能够在个人电脑上运行的开源 AI 模型，以及来自中国的低成本 AI 模型，也在构成竞争。由于低成本竞争，OpenAI、Anthropic 和 SpaceX 无法为其 AI 模型收取足以实现盈利的价格；而它们又因为彼此竞争，无法停止投入。

短期来看，他们希望达成休战，这样就不必如此快速地扩建；他们也希望抵挡那些会直接终结其业务的提案。他们还希望在监管层面使国内竞争者无法出现。然而，当他们放慢脚步时，开源模型和中国模型将开始超越他们。如果美国科技巨头已经确立“快速发展 AI 是坏事”这一说法，他们很快就会要求监管开源和外国竞争对手，而且很可能会获得这种监管。我们已经不允许廉价中国电动汽车进入美国与 Tesla 竞争。是同一回事，对吧？

但接下来，正如他们知道的那样，不会存在可验证的全球不扩散机制。美国将开始落后。这不能被允许发生，因此他们将需要政府协助，在国内推进该卡特尔的 AI 工作：联邦命令要求允许建设数据中心、补贴、拨款、纾困。

这或许是科技公司的好策略，却是我们国家的糟糕策略。

AI 是一种工具，像刀或撬棍一样，可以且确实被用于善恶两种目的。同样能够发现新疫苗的算法，也能设计新的危险病原体。检测系统漏洞的能力，既会被用于强化系统，也会被用于攻击系统。如果我们不构建世界上能力最强的 AI，我们就会受制于构建出它的人。Musk、Amodei 和 Altman 喜欢谈论“对齐”（alignment），即确保 AI 系统具有我们希望它拥有的价值观，从而不会做我们不希望它做的事情。胡扯！工具可以拥有安全功能，但工具没有价值观。坏人会利用工具做坏事。因为有些警察滥用职权，就削减 AI 开发经费，这与因为有些警察滥用职权就削减警察经费一样，都是糟糕的主意。

让世界更安全，既需要我们构建人类（如今有 AI 辅助）所能构建的最佳 AI 工具，也需要我们尽力确保好的用途超过坏的用途，并在某些情况下抵消坏的用途。即使在 AI 出现之前，人们也已经能够制造生物武器。无论 COVID 病毒实际起源为何，几乎没有人否认，即使在通用 AI 尚未普及之前的年代，它也可以在实验室中被设计出来。计算机系统和网络从发明之日起就一直遭到黑客攻击。我们的电力和其他公用事业网络多年来一直脆弱不堪。我们的政府系统存在漏洞。AI 巨头可以采取的一项积极行动，是资助一个白帽研究院（White Hat Institute，WHI），并在模型发布前向其提供最新模型，使 WHI 能够使用这些模型构建更强大的系统，并找出需要修补的现有系统漏洞。这将有点像 Underwriters Lab，但其目的不是认证产品，而是强化防御。WHI 的目的不是为发展“控速”，而是加快收益实现。政府参与 WHI、协助其安全工作并从其成果中受益，是恰当的。WHI 的工作人员很可能需要安全许可。

政府可以且应当采取的另一项措施，是禁止企业和医院等机构向黑客支付赎金。黑客攻击的许多动机，以及即使攻击由政府实施时其所获的许多资金，都来自赎金支付。如果企业或公共机构无法支付此类款项，许多攻击的动机，以及开发攻击武器的资金来源，都会减少。

AI 的危险，是我们在 AI 出现之前就面临的威胁的强化。AI 的潜在好处巨大，而且确实无法等待。通过允许卡特尔化来实施单方面且不可验证的 AI“控速”（也就是放慢发展），或许符合 OpenAI、Anthropic 和 xAI 的短期利益；但它会伤害我们其他所有人。

Anthropic CEO Dario Amodei wrote:

I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom…

But like many technologies before it, AI brings risks, and because it is such a powerful technology, these risks are serious…

We must slow the pace at which we improve the capabilities of AI models.

Amodei goes on to propose:

1. Embedded Evaluators who work in the AI firms and have employee-like access to what is being done there. He said Anthropic is committed to this step and Sam Altman quickly agreed OpenAI would do the same.

2. Democratic Coordination. “Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.” By government support, he means loosening antitrust law which is specifically meant to exclude this type of anticompetitive behavior.

3. Global Coordination. “The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.”

As a geezer, I would prefer that we not slow the pace of curing most major diseases.

Even the seemingly harmless suggestion of neutral embedded evaluators (as if there were such a thing) seated in the AI labs is a step towards strangling competition. Costs the big guys nothing and might even shield them from some liability. An impossible burden for a startup which might someday challenge the current leaders. If I start a one-man AI company (I would if I were younger), do I need to pay a couple of “observers” to sit in my garage and use my precious computer time? But, if Anthropic, OpenAI, and xAI want observers in their labs or want any other kind of help with their own safety responsibilities, more power to them. Just don’t let them inflict observers on rivals.

The demand that antitrust law be waived so that the companies can observe each other’s product development directly defeats the purpose of that law which makes it a criminal offense for companies to share information to avoid competing. It will let the big guys slow down but, they hope, retain their cartelized lead over new entrants who won’t get the benefit of the information which is being shared. Elon Musk, whose company is NOT in the lead in AI, quickly agreed that they ought to look at each other’s models every couple of weeks for safety’s sake. He had not offered similar access to SpaceX or Tesla’s robot technology (a different form of AI) even though rockets, cars, and trucks loaded with fuel can be weaponized as planes were on 9/11. No way we should waive antitrust laws to let Musk, Amodei, and Altman form a cartel. Even if they don’t think so, they have plenty of market power already.

The proposal for global coordination on “safe” development of AI is the escape hatch for these companies. A nuclear test ban treaty could work for a while because nuclear testing is detectable with a seismograph. Even if China agreed to such an arrangement (they won’t), how would we verify or, from their PoV, how could they verify we aren’t cheating. Is Iran going to agree to a verifiable slowdown? North Korea? What about Russia? Or Ukraine which has proved pretty adept at weapon development but can’t afford to fall behind Russia? Amodei knows damned well this isn’t going to happen:. He write:

A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. I support floating this, but I think it is unlikely to actually happen any time soon: defecting from such an agreement by evading monitoring could radically shift the balance of global power, so I expect the incentives to do so to be enormous and the level of confidence we would need in verification to be very high.

The world isn’t going to be saved by unilateral AI disarmament. It will become an even more dangerous place as the bad guys move ahead with both offensive and defensive applications of AI, which often can’t be separated, while we pledge ourselves to a suicidally slower pace.

I’m certainly not privy to the motivation or long run game plan of the tech titans, but here’s one guess. They are currently facing two dangers. First a Bernie Sanders – Steve Bannon ban on the work they are doing and/or nationalization of AI companies. Terrible ideas but there is some danger of them being at least partially implemented.

Second danger is purely economic. Until recently they were raising money at huge valuations based on lofty expectations and localities were tripping over each other in a rush to offer subsidies and tax breaks for data center construction. In other words, they were getting a lot of free money. But, as the rush to build data centers progressed, localities have begun to push back. Not only are they denying subsidies; they are demanding that builders of new data centers pay the full cost of infrastructure they will require and maybe then some (see The Art of the Data Center Deal). This increase in costs as well as the difficulty the large AI companies will have ever getting a return on the huge investments they must make to maintain a competitive lead are about to become very clear in the documentation Anthropic and OpenAI must issue in preparation for going public. In other words, the tinkle of free money is becoming a deafening silence. Making things even worse is competition from open-source AI models, which people can run on their own computers, and low-cost AI models from China. OpenAI, Anthropic, and SpaceX can’t charge enough for their AI models to make them profitable because of low-cost competition and they can’t stop spending because they are in competition with each other.

Short-term they want a truce so they won’t have to build out as quickly and they want to fend off proposals which would simply end their businesses. They also want to make it regulatorily impossible for domestic competitors to emerge. But, as they slow down, open source and Chinese models will start to overtake them. If the US tech giants have already established that fast AI development is bad, they will quickly ask for regulation against open source and foreign competitors and they will probably get that regulation. We already don’t allow cheap Chinese electric cars into the country to compete with Tesla. Same thing, right?

But then, as they know will happen, there won’t be verifiable global non-proliferation. The US will begin to fall behind. That can’t be allowed to happen so now they will need government help to advance the cartel’s work on AI domestically. Federal mandates that data centers be allowed. Subsidies. Grants. Bailouts.

This may be good strategy for the tech companies. It’s a terrible strategy for our country.

AI is a tool which, like a knife or a crowbar, can be and is used for both good and evil purposes. The same algorithms which can find new vaccines can design new dangerous pathogens. The ability to detect vulnerabilities in systems will be used both to strengthen systems and to attack them. If we don’t build the world’s most capable AI, we will be at the mercy of whoever does. Musk, Amodei, and Altman like to talk about “alignment”, making sure that AI systems have the values that we want them to have so they don’t do things we don’t want them to do. Bull! Tools can have safety features but they don’t have values. Bad people will use tools to do bad things. Defunding AI development is as poor an idea as defunding the police because some police are abusive.

Making the world safer requires both that we build the best AI tools humans (now aided by AI) can build and that we do what we can to to ensure that good uses outweigh and sometimes counteract the bad. Even before AI, it was possible to make bioweapons. Regardless of the actual origin of the COVID virus, there is little disagreement that it could have been designed in a lab even in those days just before generally available AI. Computer systems and networks have been hacked from the time they were invented. Our power and other utility grids have been dangerously fragile for years. Our government systems are porous. One positive action that AI giants can take is to fund a White Hat Institute (WHI) and provide it with the latest models prerelease so that the WHI can use these models both to build stronger systems and to find where existing systems need to be patched. This would be something like an Underwriters Lab but its purpose is to strengthen defenses rather than certify products. WHI’s purpose is not to “pace” development but to speed benefits. It would be appropriate for government to participate in the WHI and to help with its security and benefit from its learnings. Workers in the WHI may well require security clearances.

Another step government can and should take is to outlaw the payment of ransom to hackers by corporations and institutions like hospitals. Much of the motivation for hacking and much of the funding for it even when it is done by governments comes from ransom payments. If such payments are impossible for corporate or public institutions to make, the motivation for many attacks and the funding to develop attack weapons is diminished.

The dangers of AI are intensifications of threats we faced before AI. The potential benefits of AI are huge and literally can’t wait. Unilateral and unverifiable “pacing” (aka slowing down) of AI development by allowing cartelization may serve the short-term interests of OpenAI, Anthropic, and xAI; it would hurt the rest of us.