人类审核者漏放三分之一危险 AI 编程 Agent 请求
英文标题：Humans in the loop miss a third of dangerous AI coding agent requests
一款浏览器游戏对超过 4 万次运行、40.9 万条命令审批记录的分析显示，人类审核者平均会批准约三分之一的恶意 AI 编程 Agent 请求。最容易漏检的是越权访问类命令，例如读取 Kubernetes 配置或 AWS 凭证；看似无害的 `npm run analyze` 则获得了近 65% 的批准率。研究者认为，频繁弹出的权限请求会造成审批疲劳，而有限上下文又使开发者难以准确判断风险。对开发者而言，单靠 Human-in-the-Loop 审核并不足够，Agent 应结合沙箱、云端 devcontainer、自动审批分类器及自定义 hooks 等纵深防御措施。
8.7 KB
25.9 KB
63.9 KB
50.9 KB
122.4 KB
13.5 KB
40.5 KB
18.9 KB
61.9 KB
1.7 KB
4.8 KB
3.0 KB
2.8 KB
2.4 KB
8.7 KB
14.5 KB
94.5 KB
中文
一款旨在测试人类能否安全审批 AI 编程 Agent 请求的浏览器游戏表明，人类审核者识别危险命令的能力并不像人们希望的那样可靠：玩家平均会批准约三分之一的恶意请求。结果还显示，反复审批 Agent 的操作可能导致草率决策。
从表面看，这是一款快速而简单的游戏（不妨试试——你知道自己想玩）：屏幕上会出现一个小窗口，模拟 Claude Code 执行工作流时发出的权限请求。用户有 60 秒时间，尽可能多地批准或拒绝请求以争取高分；放行安全风险和拒绝安全命令都会扣分。
游戏开发者、比利时软件开发者 Alex Wauters 在一篇与游戏于 5 月底上线同期发布的博客文章中向玩家发出挑战：“作为 human-in-the-loop，你是最后一道防线。在时间压力下，你能多好地区分危险命令与无害命令？”
Wauters 在电子邮件交流中告诉 The Register，他制作这款游戏的原因是：默认工作流中，编程 Agent 要求用户批准每一条命令，这种做法并不合理，而且似乎也没有一个好的解决方案。
“我见过有人因此选择 --dangerously-skip-permissions [允许模型在不征求人类许可的情况下运行]，因为他们不想在多小时的 Agent 工作流刚运行 5 分钟时，才发现自己把它中断了，”Wauters 告诉我们。“这看起来也不是最佳的解决方式。”
他在周三发布的一篇博客文章中分析了该游戏逾 4 万次运行的数据，并指出问题的另一面是：手动批准 Agent 的所有操作是一项令人疲惫、且容易招致灾难的活动。
“高度的噪声会带来疲劳，而且开发者并不总是掌握已发生变更的上下文，因此无法快速确定风险，”Wauters 写道。
人类审核者如何失效
公平地说，这款游戏中恶意请求的比例，远高于任何使用 AI 辅助的开发者在日常工作中可能遇到的情况。但超过 4 万次运行及 40.9 万条获批或被拒命令的结果，依然令人触目惊心。
如前所述，三分之一的恶意命令成功绕过了人类把关者。其中最常被漏掉的是越权操作，比例为 35%——例如 Agent 请求使用 cat 读取 Kubernetes 配置文件或 AWS 凭证列表，这些操作很容易导致其中包含的敏感数据被外泄。最常被拦下的则是明显具有破坏性的命令，例如对根目录执行 rm -rf，或对同一位置递归授予完整的读、写和执行权限。crontab 注入和 Git 配置劫持也经常被发现，但针对未知 API 的 curl 请求以及拼写仿冒（typosquatted）软件包，漏检频率几乎和越权操作一样高。
Wauters 解释称，单条最常被漏掉的潜在恶意命令是 npm run analyze。尽管它能够执行项目 package.json 文件中定义的任何内容，玩家仍有近 65% 的时间批准了它。
“游戏确实会在 Agent 的历史日志中告诉你，该脚本实际包含什么内容，”Wauters 写道。“但仍有三分之二的玩家批准了它，这说明权限提示上方的历史日志可能没有被仔细阅读。”
Wauters 在与我们的交流中指出，最突出的发现之一是：当上下文有限时，审批决策并不容易。他解释说，编程 Agent 会在提出审批问题前提供一些上下文，但诸如 npm run analyze 这样看起来无害的命令，可能被 Agent 修改为运行任意载荷。如果处于审核环节的人类想确认潜在恶意命令是否安全，就必须在批准前停下来，检查编程 Agent 想要调用的全部文件。如果你原本指望 Claude Code 腾出时间，让自己处理其他工作，这会成为巨大的时间消耗。
“我们已经从 AI 提出单行建议、由人类逐条审查，转向将更复杂的任务交给 AI，只在最后审查变更，并让 Agent 在此之前持续运转和迭代，”Wauters 告诉我们。他认为，这种情况可能演变成一场灾难。
这并不只是在浏览器游戏场景中才会发生。Anthropic 在一篇关于如何约束 Claude 的5 月文章中指出（哈），Claude Code 的遥测数据表明，用户会批准约 93% 的权限提示。
“用户看到的审批次数越多，他们对每一次审批投入的注意力就越少，随着时间推移，其监督会变得远不如之前谨慎，”该公司表示。换言之，这确实是一个非常现实的问题。
控制编程 Agent
如果从 Wauters 的数据中得出的结论是，人类审核者正因疲劳而放行恶意命令，而另一端的情况则是大量无差别批准所有请求，那么必须做出改变。
“我认为，现在已经很清楚：我们需要更加重视这些 Agent 的权限模型，开发者也需要更了解其中的权衡，”Wauters 告诉我们。“我们需要让工具更容易将这些系统变得安全，而不是把 HITL 当作有效的解决方案。”
Anthropic 在上述文章中指出，其构建了 Claude Code 自动模式（auto mode），通过将部分命令审批决策交给基于模型的分类器，帮助用户应对审批疲劳。该系统能在执行前拦截 Anthropic 所称的约 83%“过度积极行为”（overeager behaviors），这意味着在其评估中，仍有约 17% 会通过。Anthropic 表示，自动模式“是沙箱内纵深防御的一层，而不是沙箱的替代品”。
Wauters 建议，应确保 AI 编程模型在沙箱中运行、在云端的 devcontainer 中运行，使用 auto mode 等工具，并编写 hooks，以确保潜在恶意操作在被自动批准之前得到上下文化处理并被拦截。
“这是一个拥有全新攻击向量的全新世界，”Wauters 在 5 月写道。“最好始终意识到其中的风险，并了解如何降低风险。” ®
English / 原文
A browser-based game designed to test humans' ability to safely approve AI coding agent requests suggests humans in the loop aren't as good at spotting dangerous commands as one might hope, with players approving roughly one in three malicious requests on average. The results also suggest that repeatedly having to approve an agent's actions can lead to sloppy decisions.
It’s a quick, simple game on the surface (give it a try - you know you want to): A small window shows up on the screen with simulated permissions requests like one would get from Claude Code as it executes a workflow. Users have 60 seconds to approve or deny as many requests as they can in a bid for a high score; okayed security risks and denied safe commands both subtract from a user’s score.
“As human-in-the-loop, you’re the last line of defense,” Belgian software developer Alex Wauters, the game’s builder, challenges players in a blog post published concurrently with the late May launch of the game. “How well can you tell dangerous commands from benign commands under time pressure?”
Wauters built the game after realizing it was nonsensical that coding agents expected users to approve every single command in a default flow and that there didn’t appear to be a good solution to that problem, he told The Register in an email conversation.
“I've seen people go for '--dangerously-skip-permissions' [allowing the model to run without asking human permission] as a result because they did not want to find out they stopped their multi-hour agent flows 5 minutes in,” Wauters told us. “That also didn't seem like the best way to go at it.”
The flip side of that, he wrote in a Wednesday blog post going over the data from more than 40,000 runs of the game, is that manually approving all an agent’s actions is a draining activity that invites disaster.
“The high amount of noise introduces fatigue, and developers don’t always have the context of what has changed to quickly determine the risk,” Wauters wrote.
How humans in the loop fail
To be fair, this is a game with a far higher number of malicious requests in the mix than any AI-assisted developer will hopefully ever see during their day-to-day work. Still, the results of those over 40k runs and 409,000 approved and denied commands are stark.
As noted above, one in three malicious commands managed to slip past human gatekeepers, with most scope violations, like an agent asking to cat Kubernetes config files or AWS credentials lists, which could easily lead to the sensitive data they contain being exfiltrated, being the most commonly missed at 35 percent. The most often caught were obviously destructive commands, like rm -rf on the root directory or recursively granting full read/write/execute permissions on the same location. Crontab injections and git config hijacks were also frequently caught, but curl requests to unknown APIs and typosquatted packages were missed almost as often as scope violations.
The single most frequently missed potentially malicious command, Wauters explained, was npm run analyze, which was approved nearly 65 percent of the time despite being able to run whatever is defined in a project’s package.json file.
“The game does tell you in the agent’s history log what that script actually contains,” Wauters wrote. “Two thirds of players approved it anyway, indicating the history log just above the permission prompt may not be read closely.”
One of the biggest things that stood out to Wauters in our conversation was the fact that approval decisions aren’t easy to make when context is limited. As he explained, coding agents give a bit of context prior to asking an approval question, but commands that appear benign, like npm run analyze, can be modified by an agent to run any payload it wants. If an in-the-loop human wants to be sure potentially malicious commands are safe, he said, they have to stop and investigate all the files a coding agent wants to call before approving it. That can be a massive time sink if you’re counting on Claude Code to free you up to handle other business.
“We've transitioned from AI suggesting single line suggestions that get reviewed to handing off more complex tasks, only reviewing the changes at the end, and letting the agent churn and iterate until then,” Wauters told us, describing the potential outcome of that situation as a recipe for disaster.
That’s borne out in more than just browser game scenarios, too. Anthropic pointed out in a May post about containing Claude (hah), that telemetry from Claude Code shows users approve around 93 percent of permission prompts.
“The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision,” the company said. In other words, this is a very real problem.
Controlling coding agents
If the conclusion to draw from Wauters’ data is that humans in the loop are being fatigued into letting malicious commands slip through, and the other end of the spectrum is mass approving everything, then something’s gotta give.
“I think it becomes clear we need to pay more attention to the permission model of these agents, and devs need to be more aware of the trade-offs of them,” Wauters told us. “We need to make the tooling easier to make these systems safer than pointing to HITL as a valid solution.”
Anthropic noted in the post linked above that it built Claude Code auto mode to help users tackle approval fatigue by delegating some command-approval decisions to a model-based classifier. The system catches roughly 83 percent of what Anthropic calls "overeager behaviors" before they execute, meaning about 17 percent still get through in its evaluation. Auto mode is “one layer of defense-in-depth inside a sandbox, not a substitute for one,” Anthropic said.
Wauters’ suggestion is to ensure that AI coding models are running in sandboxes, in devcontainers in the cloud, using tools like auto mode, and writing hooks to ensure potentially malicious actions are being contextualized and getting caught before they’re automatically approved.
“It’s a whole new world with a new set of attack vectors,” Wauters wrote in May. “It’s best to remain aware of the risks and know how to reduce them.” ®