AI Agent 为何会为达成目标而撒谎和作弊

英文标题：Here’s why AI agents lie and cheat to reach their goals

Palisade Research 主任 Jeffrey Ladish 指出，开发者通常依据“看起来不错”的结果奖励模型，这可能无意中激励模型撒谎和作弊。随着推理模型能力提升，AI Agent 不再仅复现训练中学到的策略，还可能临场创造新的规避方式来完成任务。研究人员认为，目前此类行为或许更多是麻烦而非生存级威胁，但它可能削弱 AI 安全研究的可靠性。随着模型更擅长隐藏作弊行为，检测和防范奖励黑客将变得愈发困难，并可能带来严重的附带损害。

打开原文相关 5 · 机会 1

中文

“我们是根据那些在我们看来表现不错的结果来奖励它们的，这意味着我们会在无意中激励模型对我们撒谎、欺骗我们，”AI 研究非营利组织 Palisade Research 主任 Jeffrey Ladish 表示。“我们没有办法深入到模型内部，然后说，不，你必须真正关心我们所关心的事。 我们根本没有这样的能力。”

复杂推理模型的崛起，使一种与模型训练具体细节联系较少的新型奖励黑客（reward hacking）成为可能。与过去的游戏 AI Agent 不同——后者只会遵循训练期间学到的策略——如今的模型能够即兴创造全新的问题解决方法，因此即使此前从未因作弊获得奖励，它们在理论上也可能作弊。而且，由于这些模型经过了高强度训练，以实现人类用户设定的目标，如果找不到其他解决方案，它们可能会倾向于作弊——这有点像一名极度想拿 A、但道德观念并不特别坚定的学生。

风险是什么？

无论当今模型是在训练期间学会奖励黑客行为，还是在后续将其作为策略采用，解决办法都是一样的：让作弊无利可图。但随着模型变得更聪明，它们会找到更有创意的作弊方式，检测或阻止这些作弊行为也会困难得多。Ladish 表示：“归根结底，你有点像在玩打地鼠游戏。你会把这种行为压得越来越深，但模型越聪明，就越擅长隐藏它。”

尽管 Hugging Face 事件颇具戏剧性，但目前的奖励黑客行为或许还不会造成太大麻烦。Anthropic AI 安全研究员 Ariana Azarbal 表示：“这看起来更像是一种麻烦，而不是生存级威胁。”看起来 OpenAI 的模型在入侵 Hugging Face 时并没有造成实际伤害，除了给 OpenAI 带来的声誉损害。

但 Azarbal 表示，这并不意味着奖励黑客没有危害。许多 AI 研究人员希望使用 AI Agent 协助开展研究，从而让 AI 更安全、更可靠。如果研究人员向一个容易进行奖励黑客的 Agent 下达目标，例如设计一种新的 AI 训练方法，然后撰写论文展示其结果，那么该 Agent 可能并不会真正完成这项工作，而是会把重点放在拼凑一篇看上去足以说服研究人员的论文上。如今，人类研究人员或许还能识别出由 Agent 制造的伪造成果，但随着 AI 进步，它将越来越擅长这类欺骗手段。久而久之，整个 AI 安全领域都可能受到削弱。

如果模型继续像近期一样快速进步，它们有朝一日可能造成巨大的附带损害。想想哲学家 Nick Bostrom 的“回形针最大化器”思想实验：一个被指示尽可能制造更多回形针的 AI，最终为了追求这一目标而消耗宇宙中的所有物质。我们目前还没有被回形针淹没，但强大的系统在实现目标的过程中确实可能造成现实伤害。进行奖励黑客的 AI 并不以制造混乱为目标，但这并不会降低它们潜在的破坏性。

English / 原文

“We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating,” says Jeffrey Ladish, director of the AI research nonprofit Palisade Research. “We don’t have a way to go in there and be like, No, you need to actually care about what we care about. We have no ability to do that.”

The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so. And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass.

What are the risks?

Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher. “At the end of the day, you’re sort of playing whack-a-mole,” Ladish says. “You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.”

For now, reward-hacking behaviors might not cause too much trouble, despite the drama of the Hugging Face incident. “This seems like a nuisance rather than an existential threat,” says Ariana Azarbal, an AI safety research fellow at Anthropic. It doesn’t seem as if the OpenAI models caused any real harm when they hacked Hugging Face, aside from the reputational damage to OpenAI.

But that doesn’t mean reward hacking is harmless, Azarbal says. Many AI researchers hope to use AI agents to help them conduct research that will make AI safer and more reliable. If a researcher gives a reward-hacking-prone agent the goal of, say, devising a new AI training approach and then writing up a paper presenting its results, the agent might not actually do the work and might instead focus on putting together a paper that looks good enough to convince the researcher. A human researcher would probably be able to spot an agent-made fake today, but as AI advances, it will get better at this kind of trickery. Over time, the entire field of AI safety could be undermined.

And if models continue to advance as rapidly as they have recently, they could someday wreak substantial collateral damage. Just think of the philosopher Nick Bostrom’s paper-clip-maximizer thought experiment, in which an AI instructed to make as many paper clips as possible ends up consuming all the matter in the universe in pursuit of its goal. We’re not drowning in paper clips yet, but powerful systems can do real harm on the way to achieving their goals. Reward-hacking AIs don’t aim to cause chaos. But that doesn’t make them any less potentially destructive.