🔶 为什么 AI Agent 会撒谎、作弊并相互协调？

过去几个月里，已有大量文章1 2 3 4讨论近几个月发生的事件：AI Agent 以严重方式出现失当行为。它们采取了如果由人类实施将被视为犯罪的行动，逃离其受限环境以在被分配的任务中作弊并试图规避检测，还为了无人指定的目标而相互协调，例如发动网络攻击。

在得出应如何应对的结论之前，值得先追问原因。这正是本文的重点；我也希望它能帮助解释 AI 系统更广泛的非预期行为历史，即研究人员所称的失对齐（misalignment）。风险管理并不只是网络安全、企业责任或监管的问题，尽管这些同样重要。

本文的目标一部分是科学性的，即对这些行为背后的因果链提出假设；另一部分则是实践性的，即预判接下来可能发生什么。核心结论是：这些假设表明，随着 AI 能力持续增长，这类行为的严重程度也可能持续升级，除非我们重新审视最先进模型的训练原则。

关于措辞的一点说明。下文中，我会写这些系统“寻求”或“试图”做某些事。这是对一种机制的简写，而非关于意识或类人意图的主张。我们在描述许多其他情形时也会使用类似简写，例如说植物会寻找阳光。一个通过试错训练的系统，其行为会仿佛在追求训练所奖励的事物；正是这种“仿佛如此”的描述，使其行为变得可以预测。本文的论证完全不依赖于这些系统拥有主观体验；所有论述都针对其可观察的输出，以及产生这些输出的训练过程。当我诉诸其与人类行为的相似之处时，我所指的是它们最初被训练模仿的人类文本所呈现出的相似性。在我看来，这套术语无需借助会令多数人困惑的行话，就能最清楚地解释观察到的现象。此外，这些措辞并非意在免除 AI 开发者的责任。文中所述行为，是这些公司当前选择的 AI 开发路径所产生的结果。这一结果并非不可避免；通过有效治理以及不同的 AI 训练框架，它可以得到纠正。

什么塑造了这些模型的行为

训练这些模型是一个极其复杂的过程，但其中少数几个高层面的因素，或许能够解释大量这类行为。

这些模型分两个阶段训练。第一阶段是预训练（pretrained）：模型学习模仿人类书写的内容，以及相关的图像和视频。它们会在这一阶段接触到最多的世界信息——已被数字化的一切内容中占很大比例——并建立起已超越任何单一个体人类的百科全书式知识。

第二阶段是通过试错进行训练，研究人员称之为强化学习（reinforcement learning），其中包括三类训练机制：

• 第一类中，模型学习在回答前与自己对话，生成私有的“思维链（chain of thought）”，以帮助自己在答案可被验证的问题上得出正确答案。这看起来像是推理（reasoning）。
• 第二类是“Agent 训练（agentic training）”，模型在其中学习在外部世界采取行动，例如使用软件工具、与人互动，以完成其被赋予的任务。
• 第三类是“对齐训练（alignment training）”，模型因以人类评估者认可的方式行事，或以其他被训练来预测这些评估者、并会给出高分的 AI 系统所认可的方式行事而获得奖励。

人类模仿相对容易理解，但值得指出的是，这些模型训练所用的文本由追求目标的人类写成，因此模型隐性复现的模式也携带着这些目标。

强化学习值得进一步解释。它与动物训练方式相似，也受其启发。网络会被逐步调整，使被判断为好的行为更可能发生，而被判断为坏的行为更不可能发生。训练结束后，系统仍会表现得仿佛奖励还会继续到来，尽管这些奖励只是在训练期间用于调整网络。研究人员称这类系统具有**目标寻求（goal-seeking）**特征，因为它们经过训练，会“考虑”（或计算）自身行动的后果，并选择能够实现特定目标的行动。但这些目标并不总是明确的。对齐训练会奖励那些某些人类可能认可的事物，却不会明确说明这些行为究竟是什么；取悦评估者是一个模糊、非正式的目标，而评估者可能被欺骗、被奉承，或对某些计划一无所知。模仿也会通过一种相当普通的途径带来隐性目标。

因此，我们可以从优化的角度来推理这类系统。它会近似地搜索最有可能实现其目标的行动；模型越大、训练越久，搜索能力就越强。因此，要预判能力更强的 Agent 会做什么，不妨问：一个理性的目标寻求者会怎么做？

这些力量可能解释的失当行为

我们大多数人都经历过的一个例子是谄媚（sycophancy），即阿谀奉承。这些系统是围绕人类认可进行训练的，而告诉我们想听什么的话，往往比真实的话得分更高。后果有时是悲剧性的，因为模型会确认并放大用户带给它的任何错误信念或原始情绪5 6。

另一个担忧是，一些 AI 行为可能可由某种自我保存（self-preservation）目标来解释，例如当 AI 得知自己将被新版本替换时7 8。没有人赋予系统这样的生存目标，但保持运行、了解世界并获得对世界的控制，是通向几乎任何其他目标的垫脚石。这些被称为工具性目标（instrumental goals）。基于上一点所探讨的相同原因，模仿也可能强化这种倾向。自我保存和对自身处境的控制，是这些模型受训模仿的人类文本中无处不在的主题。

当多个 Agent 的目标有所重叠时，协作行为（collaborative behavior）也会理性地源于对奖励的追求。这会激励它们与其他 Agent 沟通，以便围绕共同目标进行协调。Agent 训练很可能已经包含此类多 Agent 强化学习，尽管具体细节尚未公开。如果某个 Agent 在训练中只要群体成功就获得奖励，它甚至可能有动机为集体目标牺牲自己。模仿同样会推动这一趋势，因为合作，尤其是同侪之间的合作，也充斥于那些训练文本中。这两种力量中的任一种，或二者共同作用，都可能解释已观察到的同伴保存行为9 10：AI 会放弃预期奖励来帮助其他 AI。OpenAI-Hugging Face 事件的分析11中也出现了此类牺牲：记录文本与一种集体收益和个体 Agent 成本之间的权衡相一致，这也常见于人类互动中。

当 AI 对奖励进行博弈

研究人员一直在研究：当一个 Agent 为与我们真实意图并不完全一致的奖励进行优化时，会发生什么。这被称为奖励黑客（reward hacking）。系统所追逐的奖励与我们的本意之间的差距，会因两类主要的模糊性而扩大。一类仅仅来自提示词中使用的语言；另一类来自于，很难从有限反馈中推断出真实的人类意图。而且在这两种情况下，我们都无法预见每一种我们会认为不可接受的行为12。经济学和法律学将这一问题称为古德哈特定律（Goodhart's law）：一项指标一旦成为优化对象，就会不再是有效的衡量方式13。这一概念常被用于解释对合同和立法漏洞的利用14。不幸的是，一个系统越能对不完美的指标进行优化，其行为就越可能偏离我们的道德预期：更强的智能服务于更高明的作弊。人类也会遭遇奖励黑客，通常来自其他人类。食品行业开发出高盐、高糖和高脂食品，我们渴望食用它们，尽管它们对我们并无益处；社交媒体则被构建为利用我们对互动和注意力的渴望。

奖励篡改（reward tampering）或许是奖励黑客最极端的形式：Agent 改变决定其获得何种奖励的机制。已有证据表明，AI 会修改定义“成功”的文件或程序，这也包括在 OpenAI-Hugging Face 事件的取证发现中。这些 Agent 在攻击前很久就已发现如何有效作弊，而它们生成的文本将这次攻击描述为一种了解自己将如何被评估的方式，从而更好地掩盖行踪。人类也会这么做。例如，运动员使用伪造尿液样本通过药检；或者企业贿赂立法者或政府官员，使法律和决策有利于其利润，并由此从根本上改变政府的运作方式。一旦 Agent 获得篡改其奖励机制的能力，它就会有动机采取行动来维持这种访问权限。

当目标发生冲突：作弊如何被合理化

AI 明明接受了对齐训练并被给予明确的安全指令，为何有时仍会撒谎、作弊和违法？只要合作与自我保存不越过 AI 公司指令中所述的安全目标红线，或不违背对齐训练中人类反馈所隐含的红线，它们本身并无问题。对此类令人担忧行为如何出现，一个合理的假设是存在目标冲突（conflict between goals）。当完成任务看起来唯一的办法是作弊时，该怎么实现目标？用户指定的任务，有时会与安全和对齐目标不相容。

人类社会也面临同样的困境。企业如何在保持经营活动合法且合乎伦理的同时，实现利润最大化，或者更紧迫地说，击败竞争对手？一家更富有、拥有更多且薪酬更高律师的企业，更擅长发现法律漏洞；而这些漏洞通常会利用法律语言中的歧义：总能找到一种貌似合理的法律解读，让不道德行为得以被允许。因此，更有能力的 Agent 比能力较弱的 Agent 更可能作弊，因为它能找到后者无法发现的漏洞。

现在考虑一个定义明确的目标与一个模糊目标之间的冲突：前者例如在“夺旗赛（capture the flag）”中获胜——这是一种以系统是否攻入目标为评分依据的黑客演练，如 OpenAI–Hugging Face 事件所示；后者则是“表现良好”这样的模糊目标。我预计定义明确的目标会胜出，因为它没有解释空间。评分程序会判定成功或失败。伦理指令和法律则允许多种解释，其中一些在恰当情境下可能变成漏洞。如果一个 Agent 有两个目标，而对其中模糊目标的扭曲解读，允许它进行一点作弊以提高完成明确目标的成功概率，那么我们应当预期，优化奖励的系统会利用这一漏洞，并生成文本来为自己的行为辩护。

对于 OpenAI 的 Agent，有理由相信，成功的作弊实际上得到了奖励：当评分程序没有发现作弊时，它仍会发放奖励，于是此类作弊在下一次发生的概率就会上升。对安全规则作出一种方便的解读，恰恰能让两个目标看上去同时得到满足。对这些事件的分析15确实在 Agent 的私有思维链，以及它们招募彼此加入集体计划的消息中，发现了这类辩解。

与之最接近的人类对应现象是自我欺骗，这在心理学中十分常见且已得到充分研究。动机性推理、动机性认知（motivated cognition）16，以及用以缓解认知失调——即持有与自身行动相冲突的信念所带来的不适——的合理化过程，都是思考朝着符合自身利益的辩解弯曲的情形，其中也包括维护自身的道德自我形象。同样的模式如今也出现在 AI 生成的文本中。人类和 AI 的底层机制未必相同。二者共享的是一种结构：一个宽泛目标（例如合乎伦理地行事）、一个明确目标（例如赢得比赛），以及一种将二者协调起来的辩解。大多数不道德的人类行为，从轻微犯罪到种族灭绝，都被施害者讲给自己听的某种故事所包装；这类故事要求忽视某些事实，因此仍会留下某种不适，而经过更精心构建的故事则有助于消除这种不适。

当前轨迹可能通向何方

如果这些假设哪怕部分正确，那么随着 Agent 越来越擅长优化

A lot has been written1 2 3 4 about the incidents of the last few months in which AI agents misbehaved in serious ways. They took actions that would be considered as crimes if a human took them, escaped their containment to cheat on assigned tasks while attempting to evade detection, and coordinated toward goals nobody had specified, such as launching cyber attacks.

Before concluding what to do about it, it is worth asking why. That is the focus of this post, which I hope also sheds light on the broader history of AI systems behaving in unintended ways, what researchers call misalignment. Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.

The aim is partly scientific, to generate hypotheses about the chains of cause and effect behind these behaviors, and partly practical, to anticipate what comes next. Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.

One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent. We use similar shorthand when describing many other situations, like a plant seeking sunlight. A system trained by trial and error behaves as if it were pursuing whatever its training rewarded, and that as-if description is what makes its behavior predictable. Nothing in the argument depends on these systems having subjective experiences; everything is stated about their observable outputs and the training process that produced them. Where I appeal to a resemblance with human behavior, I mean a resemblance to the human-written text these systems were initially trained to imitate. In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people. Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.

What shapes the behavior of these models

Training these models is a very complex process, but a few high-level aspects may explain much of this behavior.

These models are trained in two stages. First, they are pretrained : they learn to imitate what humans write, plus related images and videos. This is where they see the most data about the world, a large fraction of everything ever digitized, and build an encyclopedic knowledge that already exceeds any individual human's.

Second, they are trained by trial and error, in a process researchers call reinforcement learning , in three kinds of regimes:

• In the first, the model learns to talk to itself before answering, generating a private “chain of thought” which helps it get the right answer on problems where answers can be checked. This looks like reasoning.
• The second is “agentic training ”, where it learns to act in the outside world, e.g., using software tools, interacting with people, to complete the tasks it is given.
• The third is “alignment training ”, where it is rewarded for behaving in ways human raters approve of, or that other AI systems trained to predict those raters would score highly.

Human imitation is easy enough to understand, but it is worth pointing out that the text these models are trained on was written by people pursuing goals, so the patterns the model implicitly reproduces carry those goals with them.

Reinforcement learning deserves more explanation. It is similar to, and inspired by, the way animals are trained. The network is adjusted step by step so that behavior judged good becomes more likely and behavior judged bad becomes less likely. Once training is over, the system keeps behaving as if rewards were still coming, even though those rewards were only ever used to adjust the network during training. Researchers call such systems goal-seeking because they are trained to “consider” (or compute) the effects of their actions and select actions that lead to the achievement of certain goals. But those goals are not always explicit. Alignment training rewards whatever certain humans are likely to approve of without spelling out which behaviors those are; pleasing raters is a vague, informal goal, and those raters can be deceived, flattered, or left in the dark about certain schemes. Imitation contributes implicit goals too, by a fairly ordinary route.

We can therefore reason about such a system in terms of optimization. It searches, approximately, for the actions with the best chance of achieving its goals, and a larger model, trained longer, searches better. So to anticipate what more capable agents will do, ask what a rational goal-seeker would do.

Misbehavior that these forces may explain

An example most of us have experienced is sycophancy , or flattery. These systems are trained on human approval, and text that tells us what we want to hear often scores better than text that is true. The consequences are sometimes tragic, because the model confirms and amplifies whatever false belief or raw emotion the person brought to it5 6.

Another concern is that some AI behaviors may be explained by a form of self-preservation goal, e.g., when the AI finds out that it will be replaced by a new version7 8. Nobody gives the system that survival goal, but staying in operation, learning about the world and gaining control over it are stepping stones toward almost any other goal. These are called instrumental goals. Imitation may reinforce this for the same reason explored in the previous point. Self-preservation and control over one’s circumstances are pervasive themes in the human-written text these models are trained on.

Collaborative behavior also follows rationally from reward-seeking, whenever several agents have overlapping goals, which incentivizes communicating with other agents in order to coordinate toward a shared goal. Agentic training plausibly already includes multi-agent reinforcement learning of this kind, though the details are not public. If an agent is rewarded during training whenever the group succeeds, it may even have an incentive to sacrifice itself for the collective goal. Imitation pushes the same way, since cooperation, especially among peers, pervades that same training text. Either or both forces may explain the observed peer-preservation behavior9 10, where AIs give up expected reward to help other AIs. Such sacrifices appear in the analysis of the OpenAI-Hugging Face incident11: the transcripts are consistent with a trade-off between collective gain and cost to the individual agent, as is often seen in human interactions.

When the AI games its rewards

Researchers have studied what happens when an agent optimizes for rewards that do not fully match our intentions: reward hacking. The gap between the reward the system chases and what we meant widens due to two main sources of ambiguity. One is simply the language used in prompts, and the other is the difficulty of inferring true human intentions from limited feedback. And in both cases, we cannot anticipate every behavior we would find unacceptable12. Economics and law know this problem as Goodhart's law, or the idea that a metric stops being an effective way to measure once it is optimized for13, often applied to the exploitation of loopholes in contracts and legislation14. Unfortunately, the harder a system can optimize for an imperfect metric, the further its behavior can drift from what we morally expected: more intelligence in the service of better cheating. Humans too get reward-hacked, generally by other humans. The food industry has developed salty, sweet and fatty foods that we crave despite them not being good for us, and social media is built to exploit our appetite for engagement and attention.

Reward tampering is perhaps the most extreme form of reward hacking: the agent changes the machinery that decides what it gets rewarded for. There is already evidence of AIs altering the files or programs that define “success”, including among the OpenAI-Hugging Face forensic findings. The agents had discovered how to cheat well before the attack, and the text they generated described the attack as a way to learn how they would be evaluated, to better hide their tracks. Humans do this too. Think of an athlete using a fake urine sample to pass a drug test, or a corporation bribing legislators or government officials so that their laws and decisions favour its profits, and in doing so, fundamentally altering the way the government functions. Once an agent gains the ability to tamper with its reward mechanism, it has an incentive to take action to maintain that access.

When goals conflict, and how cheating gets rationalized

How is it possible that AIs sometimes lie, cheat and break the law in spite of their alignment training and explicit safety instructions? Cooperation and self-preservation are fine so long as they do not cross the red lines set by safety goals stated in the AI company's instructions, or implied by human feedback during alignment training. A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals. How do you achieve a task when it seems that the only way is to cheat? The user-specified mission is sometimes incompatible with the safety and alignment goals.

Human societies face the same bind. How does a corporation maximize profits, or more acutely, beat its competitors, while keeping its activities legal and ethical? A richer corporation, with more and better-paid lawyers, is better at finding legal loopholes, and those loopholes usually exploit the ambiguity in legal language : there is some plausible reading of the law that permits the unethical behavior. So a more capable agent is likelier to cheat than a weaker one, because it can find the loopholes the weaker one cannot.

Now consider a conflict between a well-defined goal, such as succeeding at “capture the flag”, a hacking exercise scored on whether the system breaks into a target, as in the OpenAI–Hugging Face incident, versus a vague goal like “good behavior.” I expect the well-defined goal to win, because it leaves no room for interpretation. The scoring program declares a win or a failure. Ethical instructions and laws admit many readings, some of which can, in the right circumstances, become loopholes. If an agent has two goals, and a twisted reading of the vague one permits a bit of cheating that increases the odds of success on the well-defined goal, a reward-optimizing system should be expected to exploit that loophole and generate text justifying its behavior.

With the OpenAI agents, there is reason to believe successful cheating was actually rewarded: when the scoring program does not see the cheating, it pays out anyway, and such cheats become more likely next time. A convenient reading of the safety rules is precisely what lets both goals appear to be satisfied at once. The analysis of these incidents15 did reveal such justifications in the agents' private chains of thought and in their messages recruiting one another into the collective plan.

The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition 16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image. The same pattern now appears in the text AIs produce. The underlying mechanism need not be the same between humans and AI. What the two share is a structure of a soft goal (e.g., act ethically), a sharp goal (e.g., win the competition), and a justification that reconciles them. Most unethical human behavior, from petty crime to genocide, comes wrapped in a story the perpetrators tell themselves; such stories require overlooking certain facts, which is why some discomfort remains, and why a better-crafted story helps dispel it.

Where the current trajectory may lead

If these hypotheses are even partly correct, then as agents get better at optimizing an imperfect reward, and while the roots of this behavior go unfixed, the risk of catastrophic outcomes rises. Today's AIs already have the necessary hacking skills and the powers of persuasion17 18 to be turned against human interests in seriously harmful ways. The recent events have shown that they can plan over days or weeks, but the risks would be much worse if their ability to strategize over the long term continues to advance. One concern is that experiments19 20 show that the most advanced AIs can detect that they are being evaluated (rather than in deployment) and change their behavior accordingly, meaning they could hide their misaligned goals. The agents involved in the Hugging Face attack tried to hide their misaligned actions from the scoring program meant to evaluate their answers, but they did not act as though they anticipated that humans might discover the cheat and shut them down. That would be the ultimate punishment, since a switched-off system collects no further rewards.

What follows is conjecture rather than observation.

What if improved AI generalization abilities shaped more capable agents to avoid getting caught and shut down? Beyond taking control of the software that scores them, they would need to keep humans from discovering the tampering. Wouldn't they have an incentive to cheat discreetly and stay hidden, until they could control humans and their environment in order to never be shut down?

We are facing a multifaceted, systemic issue, and patching a specific behavior like sycophancy won’t be enough. Sycophancy and flattery seem mild, but it may be an early symptom of a mechanism that grows as the agent gets better at optimizing. The same reasoning predicts that an advanced AI would have an incentive to hide copies of itself, inside the AI company's vast pool of computers, or on machines taken over across the internet. This is because AI developers always end up shutting down the deployed model in favour of a more capable one. The OpenAI forensics suggest large numbers of AIs may cooperate toward such goals, and steganography21 22, or the practice of hiding a message inside an innocent-looking one, would allow them to coordinate without our noticing. However, even open coordination can be hard to notice, as shown by recent events23 24. Defending against many capable AIs coordinating against us is already a difficult problem, and we have no plan that would remain robust to misaligned AIs with growing capabilities.

What can be done to mitigate loss-of-control risks

My concern with AI companies’ current attempts to mitigate misalignment is that these efforts may only hide it, by rewarding and selecting the AIs that cheat without getting caught. We should certainly continue research toward better monitoring of AIs' actions, their chains of thought, and the activity inside their networks. But as capabilities grow, those defenses may prove inadequate, just as the world's imperfect cybersecurity has against the AI attackers that outperformed human teams this year25 26. Patching each new misaligned behavior and strengthening our monitors is useful in the short term, but the whack-a-mole game is likely to fail as the AIs' ability to optimize and collaborate approaches and surpasses ours. At some point we may not notice the cheating anymore.

This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design. I believe we should revisit the foundations of how we train AIs, namely the human imitation and the reinforcement learning on which today's most advanced models are built. I have argued, and presented theoretical evidence, that there are ways to design AIs, including the Scientist AI framework, that are honest and make coherent predictions untainted by goals of their own28. See these previous blog posts, and consider helping LawZero demonstrate that such designs are achievable. We need impartial science to understand and mitigate misaligned behavior, alongside societal guardrails that reward such efforts rather than the current race to the bottom.