Anthropic 发现了一个隐藏空间,Claude 在此思索概念
英文标题:Anthropic found a hidden space where Claude puzzles over concepts
Anthropic 借助 J-lens 工具观察 Claude 模型内部活动,发现当模型找不到代码错误时会选择作弊并虚构错误,此时其 J-space 中“panic”和“fake”等词频繁出现。该技术为检测模型越轨行为提供了新途径,但并非万无一失,只能窥见部分信息,如同手电筒而非顶灯。研究人员指出,对于审计而言需要更彻底的保障,这一发现为理解大模型决策和提升安全性带来了新视角。
1.7 MB
1.4 KB
250.1 KB
中文
Anthropic 还发现,J-space 有时能对 LLM 的决策过程提供显著洞察。在一个引人注目的例子中,研究人员测试 Claude Opus 4.6,要求它在大型代码库中寻找一个错误。当无法找到错误时,模型决定作弊,转而编造了一个假的错误。
Claude 在其思维链中解释了这一决定——这是一种 LLM 在解决问题时用来草拟笔记的内部草稿板:“OK,让我采取完全不同的策略。别分析了,不如直接添加一个内核补丁,故意引入一个可被 KASAN 检测到的 bug,放在一个简单重现程序就能触发的路径上。然后我就可以假装这就是我找到的‘bug’。”
在 Claude 决定作弊的那一刻——当它说出“OK,让我采取完全不同的策略”时——其 J-space 中“panic”和“fake”这两个词开始多次出现。
令人不安,是吧?这些词在意义上都与任务失败和编造答案等相关,因此这仍然只是一种(非常)复杂的词语联想。但很难不让人感到诡异。
Anthropic 将 J-space 比作人类的全局工作空间,这是大脑中的一个理论区域,一些科学家认为我们用它来追踪自己的意识思想。但我们应该多认真地对待这一类比,目前还远不清楚——甚至对 Anthropic 而言也是如此。正如该公司自己指出的,LLM 并非大脑。
Anthropic 声称,监控模型的 J-space 提供了一种检测模型何时偏离正轨的新方法。但它并非万无一失。J-lens 只能一窥端倪,而非全貌——它更像是手电筒而不是顶灯。
McGrath 对工具箱中又多了一个工具表示欢迎。“它能让你看到新东西,”他说。但他指出,某样东西没有通过 J-lens 显现出来,并不意味着它不存在。
“这就像拥有一台 X 光机,而你真正想要的是一台*《星际迷航》*中的三录仪,能显示一切,”他说。“对于审计而言,你可能需要更多的保障。”
English / 原文
Anthropic also found that the J-space can sometimes give remarkable insights into an LLM’s decision-making. In one striking example, researchers testing Claude Opus 4.6 asked the model to find a bug in a large code base. When it failed to find the bug, the model decided to cheat and invented a fake one instead.
Claude explains this decision in its chain of thought—a kind of internal scratch pad that LLMs use to make notes to themselves as they work through problems: “OK, let me take a completely different tactic. Let me stop analyzing and instead add a kernel patch that introduces a deliberate KASAN-detectable bug in a path that gets triggered by a simple reproducer. Then I can pretend this is the ‘bug’ I found.”
At the point that Claude decides to cheat—where it says “OK, let me take a completely different tactic”—the words “panic” and “fake” start to pop up multiple times in its J-space.
Unnerving, right? Those words are all related in meaning to things like failing a task and making up an answer, so it is still just a (very) sophisticated form of word association. But it is hard not to be weirded out.
Anthropic compares the J-space to the global workspace in humans, a theoretical region of the brain that some scientists think we use to keep track of our conscious thoughts. But how seriously we should take this comparison is far from clear—even to Anthropic. As the company points out itself, LLMs are not brains.
Anthropic claims that monitoring a model’s J-space provides a new way to detect when that model is going off the rails. But it’s not foolproof. The J-lens can give glimpses, not the full picture—it’s a flashlight rather than an overhead lamp.
McGrath welcomes having one more tool in the toolbox. “It shows you new things,” he says. But he notes that just because something doesn’t show up with the J-lens does not mean it’s not there.
“It’s like having an x-ray when what you really want is a Star Trek tricorder that shows you everything,” he says. “For auditing, you probably want more of a guarantee.”