Agentic AI AI AI Assistant

像穴居人一样与 Agent 对话，真的能节省 65% Token 吗？我们测试了

对 Token 压缩技能 Caveman 在 Claude Code 上的配对 A/B 基准测试，测试运行于 SkillsBench：它是否真的能节省 Token，以及是否会降低 AI Agent 的输出质量？

宣称节省：65%。实测节省：8.5%。

这是在真实 Agent 任务上、强制激活该技能后实现的输出 Token 节省。这是其效果上限，而非通常使用场景下的结果。

我们为何进行这项测试

JetBrains 正越来越多地投资于对编程 Agent 周边工具进行严谨测试，其中有一个技能引起了我们的注意：“Caveman”。用它自己的语言来描述，其宣传大致如下：

技能让 agent 像穴居人说话。能少用 token，何必多用。废话死。代码、命令保持字节完全一致。每次回复。永远。省 65% 输出 token。支持 30 多个 agent。GitHub 星多。

我们的想法是：

宣称很便宜，验证很昂贵。Agent 不是聊天窗口。Agent 输出大多是工具调用、文件编辑、代码；技能承诺不碰这些。因此，我们测量 README 没有测量的两件事：在多步骤 Agent 工作中的真实节省，以及压缩 Agent 的思考过程是否会损害任务结果。

测试设置

**为何“强制启用”很重要：**Caveman 由用户激活。它会被“caveman mode”或“be brief”等短语触发。我们在每一次回复中都强制启用它，这意味着下面的每个数字都是该技能的最佳情形。在正常使用中，Agent 必须自行决定是否激活它，因此实际节省只可能等于或低于这里测得的约 10% 上限。

发现 1：节省约为 8.5%，而非 65%

宣传中的节省来自聊天式的散文回答。Agent 输出则不同：代码、差异（diff）、工具调用和精确错误字符串主导了 Token 流，而 Caveman 正确地将所有这些内容保持原样。只有工具调用之间的叙述性文字会被压缩，而这类文字并不多。

相对于基线的输出 Token 节省：

这是强制启用 Caveman 组在三次运行中的输出 Token 节省。第一次小规模测试中醒目的 -29.5% 未能复现；随着规模扩大，节省收敛至 -8.5%（在 82 个配对任务中，输出 Token 从 59.2 万降至 54.2 万）。宣称的 -65% 已经超出图表范围。

发现 2：未检测到质量下降

我们真正关心的问题是：让 Agent 更简短，会不会让它变差？在完整运行的 82 个配对任务中，答案是否定的：两组在统计上无法区分。

每任务配对结果：

对 18 个非平局结果进行的符号检验显示：p = 0.82，远未达到任何显著性差异标准。基线组的平均任务得分为 0.326，技能组为 0.311；在 0-1 量表上，相差 -0.015。

各次运行的平均任务得分：

随着样本量增长，早期看起来令人担忧的差距不断缩小：这符合噪声模式，而不是真实效应。两组中，单个任务都可以在重复尝试之间自由地从通过变为失败，或从失败变为通过。

风格迁移本身完全按设计工作：强制启用组的记录明显呈现穴居人式表达，而代码产物保持不变且正常。

发现 3：成本节省真实存在，但很脆弱

成本大体跟随约 8.5% 的 Token 节省，因此理论上技能组应便宜约 10%，按单个任务计算也确实如此。但在我们的完整测试中，两组原始总成本却显示技能组贵了 11.6%：40.60 美元，对比 36.39 美元。

这一反转完全由一次试验造成：一项依赖审计任务在技能组中膨胀到超过 20 万 Token 的长上下文定价档，计费为 8.29 美元，而非 0.33 美元。在更早的一次运行中，同一任务则在基线组产生了一个 3.25 美元的异常值。这是该任务本身的特性，而非技能造成的结果。

结论

**风格安全、效果诚实，但节省被过度宣传。**在强制启用时，Caveman 能稳定改变 Agent 的表达方式，而不会对 Agent 实际产出的结果造成可测量的损害：82 个配对任务，符号检验 p = 0.82。但在真实 Agent 工作中，它至多只能削减约 8.5% 的输出 Token 和约 10% 的成本，因为主导 Agent 会话的 Token 是代码和工具调用，而该技能有意保留这些内容。宣称的 65% 属于聊天式问答，不适用于编程 Agent。

**建议：**如果你喜欢它，就使用它。它很有趣，而且不会带来可测量的质量损失。但不要期待它能在日常 Agent 任务中带来巨大节省：个位数百分比的高端区间，才是现实的效果上限。

• **质量：**未检测到下降：8 个任务更好，10 个更差，64 个平局；在 0-1 量表上，平均任务得分相差 0.015（p = 0.82）。
• **Token：**在强制激活时，输出 Token 减少 8.5%，这意味着它已接近效果上限；自动触发的使用方式节省会更少，或完全没有节省。
• **成本：**预期可降低约 10%，但经常会被单次试验的方差抵消。
• **方法论补充：**我们第一次仅含 10 个任务的运行“显示”Token 节省了 -30%。随着样本量增长，这一结果消失了。永远不要相信 k=1 的评测。

你希望我们下一次测试哪个技能？在评论区留下名称即可。少字够。我们测。

运行详情：Harbor 0.17；claude-sonnet-5，推理强度 low；SkillsBench 86/87 个任务；约 240 次试验；总支出约 106 美元。

订阅 JetBrains AI Blog 更新

Agentic AI AI AI Assistant

Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? We Test

A paired A/B benchmark of the token-compression skill

Caveman

on Claude Code, run on

SkillsBench

: does it actually save tokens, and does it degrade AI agent output quality?

Advertised saving: 65%. Measured saving: 8.5%.

Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.

Why we ran this

We at JetBrains are investing more and more into proper testing of the tooling around coding agents, and one skill got our attention: “Caveman”. Its pitch is best described in its own dialect:

Skill make agent talk like caveman. Why use many token when few do trick. Filler die. Code, commands stay byte-exact. 65% output token saved. Every reply. Forever. Work with 30+ agents. Many GitHub star.

We think:

Claim cheap to make. Verify expensive. Agent not chat window. Agent output mostly tool call, file edit, code: skill promise not touch those. So we measure two things README not measure: real saving on multi-step agent work, and whether squeezing agent think-out-loud hurt task outcome.

Setup

Why “forced” matters: Caveman is user-activated. It triggers on phrases like “caveman mode” or “be brief”. We forced it on in every reply, which means every number below is the skill’s best case. In normal use, where the agent must decide to activate it on its own, the realized saving can only be equal or lower than the roughly 10% ceiling measured here.

Finding 1: the saving is about 8.5%, not 65%

Advertised savings come from chat-style prose answers. Agentic output is different: code, diffs, tool invocations, and exact error strings dominate the token stream, and Caveman correctly leaves all of it verbatim. Only the narration between tool calls gets compressed, and there is not much of it.

Output-token saving vs. baseline smoke: 10 tasks, k=1 re-run: 10 tasks, k=3 full: 86 tasks, k=1 -29.5% -6.7% -8.5% small-sample noise headline result, 82 clean pairs advertised -65% Output-token saving of the forced-Caveman arm across the three runs. The eye-catching -29.5% from the first small run did not replicate; at scale the saving converges to -8.5% (592k to 542k output tokens over 82 paired tasks). The advertised -65% is off-chart.

Finding 2: no detectable quality degradation

The question we actually cared about: does making the agent terse make it worse? Across 82 paired tasks in the full run, the answer is no: the arms are statistically indistinguishable.

Per-task paired outcomes 8 64 tied 10 skill scored higher identical score in both arms skill scored lower Per-task paired outcomes, full run. Sign test over the 18 non-ties: p = 0.82, far from any significant difference. Average task score was 0.326 for baseline vs. 0.311 for the skill arm, a -0.015 gap on a 0-1 scale. Average task score per run no-skill with-skill-forced 0.5 0.25 0 0.38 0.25 0.45 0.39 0.33 0.31 smoke: k=1 10 tasks: k=3 86 tasks: k=1 looked like a regression gap shrinks statistically flat Average task score per arm. The scary early gap shrinks as sample size grows: the pattern of noise, not of a real effect. Individual tasks flip freely between passing and failing on repeat attempts in both arms.

Style transfer itself works exactly as designed: forced-arm transcripts are unmistakably caveman, while code artifacts stay untouched and normal.

Finding 3: the cost saving is real but fragile

Cost tracks the roughly 8.5% token saving, so the skill arm should come out roughly 10% cheaper, and per task, it does. But the raw arm totals in our full run showed the skill arm 11.6% more expensive: USD 40.60 vs. USD 36.39. The entire inversion came from a single trial: one dependency-audit task ballooned past the 200k long-context pricing tier in the skill arm and billed USD 8.29 vs. USD 0.33. In an earlier run the same task threw a USD 3.25 outlier in the baseline arm. It is a property of the task, not the skill.

Outcome

Safe, honest about style, oversold on savings. Forced on, Caveman reliably changes how the agent talks without any measurable damage to what the agent produces: 82 paired tasks, sign test p = 0.82. But on real agentic work it trims about 8.5% of output tokens and about 10% of cost at absolute best, because the tokens that dominate agent sessions are code and tool calls, which the skill deliberately preserves. The advertised 65% belongs to chat-style Q&A, not to coding agents.

Recommendation: use it if you like it. It is fun, and it costs you nothing measurable in quality. Just do not expect huge savings on daily agentic tasks: a high-single-digit percentage is the realistic ceiling.

• Quality: no detectable degradation: 8 tasks better, 10 worse, 64 tied; average task score differs by 0.015 on a 0-1 scale (p = 0.82).
• Tokens: -8.5% output tokens with activation forced, meaning this is the ceiling; auto-triggered usage saves less or nothing.
• Cost: roughly -10% in expectation, routinely erased by single-trial variance.
• Methodology bonus: our first 10-task run “showed” a -30% token saving. It dissolved as sample size grew. Never trust a k=1 eval.

You want next skill tested? Drop name in comments. Few word enough. We test.

Run details: Harbor 0.17;claude-sonnet-5 with reasoning effort low; SkillsBench 86/87 tasks; about 240 trials; about USD 106 total spend.

Subscribe to JetBrains AI Blog updates