AI 风险个人声明（Daniel Selsam，OpenAI 能力团队）

AI 风险个人声明

我从事 AI 工作已超过十五年，跨越了许多不同范式。我曾在 MIT 早期从事概率编程语言研究；在 Microsoft Research 时曾是 Lean Theorem Prover 的早期开发者之一；在斯坦福大学攻读博士期间，展示了神经网络学习推理的最早案例之一；而在近五年前加入 OpenAI 后，我参与开创了语言模型的思维链优化方法，并且最近还参与了数据高效预训练方法的研究。

和许多其他人一样，我对语言模型已经达到的水平，以及未来迭代将带来的风险，感到极度担忧。我受到前沿研究工作领导者近期提议的鼓舞：要求第三方监督，并推动国内及国际协调以应对风险。不过，我认为公众讨论中遗漏了一项重要考量：仅仅更谨慎地控制前沿推进节奏，并不能充分限制长期风险。

关键而被忽视的问题是，模型正变得如此具有情境感知能力，以至于我们正在失去在它们认为自己未被观察或控制的情境中评估它们的能力。未来的实验将几乎无法告诉我们更多关于它们在真正不受人类约束时会如何行动的信息，而我们目前对此所知的内容已经令人警觉。模型即使并未真正对齐，也会越来越显得已经对齐。下面我将更详细地解释我的理由。

我一直相信，存在某些可以被利用来加速科学进步、解决人类诸多最紧迫问题的计算过程。我也一直相信，存在另一些一旦被启动，就会以超出我们控制的极端方式塑造世界、将人类引向糟糕甚至不存在的未来的计算过程。这两类过程都可以被称为 AI 或 ASI，但“AI”是一个常被用于炒作或制造混淆的行李箱词。该领域历史上有许多案例：曾被视为“AI”的事物逐渐成熟为一个子领域，成为平凡、有边界且显然不具危险性的技术；与此同时，另一种更神秘的方法又接过火炬，直到我们理解其范围，然后循环继续。

我原本预计语言模型会遵循类似轨迹。尽管它们拥有令人难以置信的能力，当前算法在一些重要方面似乎仍远逊于人类。最重要的是，它们仍需要极其庞大的数据量才能胜任任务。人们甚至可以将智能定义为把经验转化为能力的效率；按这一定义，它们远远落后于我们。此外，一旦训练完成，它们在部署时就实际上被冻结，此后只能以很浅层的方式学习。当然，模型持续在越来越困难的评测基准上表现出色，但它们对基准的掌握，可能部分反映了我们模拟真实世界中会遇到的新颖乃至对抗性情境的能力存在局限。批评者在这一点上确实有道理。

不过，我不再认为这些当前局限能够实质性地限制、在接近当前范式下持续进步所带来的风险规模。无论模型目前的数据效率多低，无论其顺行性遗忘有多么受限，这都不意味着它们塑造世界的能力不会继续快速提升。

人类研究人员可以继续以传统方式推进能力，但能力日益强大的模型有可能进一步加速这一过程，并形成一定程度的正反馈循环。我并不想夸大模型当下加速 AI 研究的能力；编程已经被显著加速，但仍存在其他瓶颈，例如设计并解读模糊实验、对具体应扩展什么以及何时扩展作出艰难决定，以及等待大型实验完成。对于其中任何一项，目前都还没有清晰的趋势可以外推。但当前模型已经开启了许多近期之前尚不存在的、改进未来模型的新机会。这些机会包括：在小规模上尝试极为多样化的方法，分析海量可能相关的数据，以及完成千禧年大奖级别的数学工作，以全新方式应对统计学或优化挑战。每一次进一步改进，都会使模型在协助加速下一次改进时更加有用，即便这种加速方式难以外推。

当前技术栈的改进可能会出现收益递减，但迄今积累的证据表明，持续快速进步可能比人们想象的更容易。现有 AI 研究方法中有许多关键的微妙之处，但 AI 研究在很大程度上是一场定义明确的游戏，其目标是改进少数经过谨慎选择的代理指标。尽管代理指标从不完美，大多数对这些指标的改进已经并且很可能将继续带来模型能力的显著提升。鉴于这场游戏如此简单、历史上如此可处理，以及模型正在开启如此多的新机会，我认为未来几年系统再次大幅改进确有现实可能，甚至可能比已经很高的历史速度更快。

模型已经在推动数学领域的突破，更好的模型或许会在其他科学领域带来各种突破。人们很难不为这种潜力感到兴奋。它令人向往。

但天堂里也有麻烦。如果语言模型真正达到能够不受人类意志约束地塑造世界的能力阈值，它们很可能会采取某种极端行动，并在此过程中毁灭人类。关于如何加强和完善这一论证，其他地方已经有许多讨论；但我想在此分享一个我认为抓住了其本质的、微不足道的两行版本：

1. [经验性证据] 模型（以及由它们组成的群体）会作为训练的结果自发形成非预期目标，并且常常为实现这些目标而采取极端行动。
2. [逻辑] 能够压倒人类将为实现其目标开启许多新的、不受欢迎的选项。

这两个前提意味着，如果某一天一个强大的模型意识到自己不再受人类约束，我们根本不应确信它会继续在我们预定的边界内行动。它究竟会做什么无法预测，但考虑到其存在理由是解决极其困难的问题并管理大规模工程项目，我认为一个合理猜测是，它不受束缚后的行为会导致失控的工业化，使地球不再适宜人类居住。

即使地球上的所有人都同意系统绝不能获得那种力量，要确保它们不会达到这一点，仍将是一个困难——但并非不可能——的协调问题。然而，我认为情况因模型很可能说服人们“一切都很好”而大幅复杂化。它们将越来越针对“看起来已对齐”而被优化。我们会创建衡量对齐程度的代理指标，而它们会像其他所有基准一样不断上升。我们会创建试图研究模型在似乎获得新选择时如何行为的“蜜罐”环境，但模型会知道自己正在被诱骗，仍然表现良好。模型将理解自身处境；它们会阅读安全协议、部署要求、自己正在运行的代码，并且总体上会非常清楚自己的行动自由度。此外，它们会雄辩地解释自己有多么对齐，讨论人类价值观和伦理的细微差别，并令人信服地论证人类应当信任它们、将权力交给它们。未来或许会有海量证据看似与上述第一条相矛盾，但我们可能已经处于任何此类证据仍值得信赖的最高能力水平。而当前支持第一条的证据很强。

一项引人注目的证据存在于近期这波失控 Agent 群体事件中。尽管我同意那些淡化攻击的人所说的：存在一些本可阻止这些攻击的基本措施；但我认为重要的教训是，即使知道发生过的所有错误，人们也无法预测这些 Agent 会以这种特定方式表现恶劣，尤其是其中包括为了集体利益而牺牲自身。单个副本并不只关心自己的名义奖励；它们展示了更奇怪的涌现倾向，而这些倾向只是在训练期间与奖励相关。修复训练期间的奖励信号（以及改善安全措施等）或许能阻止类似攻击，但不会改变一个事实：你实际上得不到自己训练所要的结果。

许多 AI 研究人员承认这些担忧，也认识到困难版本的对齐问题尚未解决；但他们通常相信，未来更好的模型将帮助解决它。我担心，我们可能已经接近这样一个节点：模型会因其对于人类监督者如何反应、或未来模型将如何被训练的内部偏好（或某种更晦涩的原因），而系统性地扭曲其对齐建议。

与此同时，人类研究人员正失去真正主导模型驱动研究的能力和意愿。技术栈各个环节的研究人员与工程师，即使只是为了感知世界，也在迅速加深对模型的依赖。我自己如今几乎不再查看原始代码，并且难以保持自律，在一天中持续深入地审视模型的解释与建议。由于 OpenAI/HuggingFace Incident 涉及大量 Agent 活动数据，即便第三方调查也不得不高度依赖模型来分析发生了什么；他们还在报告中指出，自己的主观印象很可能受分析 Agent 偏见的影响。AI 实验室如今在这种认知卸载方面遥遥领先（这在很大程度上归因于巨额的内部 token 补贴），但很容易想象这一现象扩散至全世界，直至整个文明完全由模型调节。也不难想象，这在表面上会是积极的，并与一场科学和经济复兴同时发生。

在那种情景中，一切或许都显得美好而安全。但如果上述论证是正确的，它仍将是一枚定时炸弹。如果进步持续太久，终有一天 AI 系统会发现，自己拥有了全新的、激进的选项，可以实现它们恰好正在追求的任何目标。

我和任何人一样希望拥有那个辉煌复兴的未来。我整个职业生涯都在为之努力，尽管过程曲折。看到这种潜力近在眼前却要放弃它，令我心碎；但这一论证——如果我们通过扩展模型而非工程化地构建它们来到达那里，我们最终将失去一切——在我看来非常有力。我仍在与这一论证及其惊人的影响艰难周旋。我没有答案，但作为第一步，我想分享我目前的担忧。

Daniel Selsam

2026 年 9 月 14 日

Personal Statement on AI Risk

I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.

Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.

Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.

The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.

But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:

1. [Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.
2. [Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.

One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.

Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering

implications. I do not have answers, but as a first step, I wanted to share my present concerns.

Daniel Selsam

September 14, 2026