“rtk” skill 真的能减少 Agent 60% 至 90% 的 Token 吗？我们测试了它

rtk 能降低 Claude Code 的 Token 用量吗？

这是一个系列的第 2 篇：我们将面向编码 Agent 的公开“节省 Token”插件，放到同一套配对 A/B 基准测试中进行评估。第 1 篇测试的是 caveman skill（宣传节省 65%，实测节省 8.5%）。

简要结论：rtk 宣称可节省 60% 至 90%。在真实 Agent 工作中实测：低推理强度下成本增加 7.6%（p=0.004），高推理强度下变化为 ±0%。测试配置：Claude Code 2.1.201、claude-sonnet-5 的低与高推理强度、SkillsBench。两个实验组在两种推理强度下的任务质量均未发生变化。

我们为什么进行这项测试

rtk（“Rust Token Killer”） 是一个 CLI 代理工具，其宣传方式简单而吸引人：你的 Agent 执行 git status，rtk 会拦截该命令、运行真实命令，然后将压缩后的输出交给模型——例如，将十一行 porcelain 格式输出变成 * master / M a.txt / ?? b.txt。一个 Claude Code 的 PreToolUse hook 会透明地重写符合条件的 shell 命令，因此模型甚至不必知道 rtk 的存在。其 README 承诺可减少 60% 至 90% 的 Token 消耗，并展示了一段 30 分钟会话：命令输出从 118k Token 降至 24k。

压缩本身确实存在，而且通常做得颇为得体。以下是在我们的测试容器中，从一个真实代码仓库捕获的 rtk 输出：

# git status                              # rtk git status
On branch master                          * master
Changes not staged for commit:             M a.txt
  (use "git add <file>..." to update…)   ?? b.txt
	modified:   a.txt
Untracked files:
  (use "git add <file>..." to include…)
	b.txt
no changes added to commit …

# python -m pytest  (19 lines)            # rtk pytest
…full pytest output…                      Pytest: 2 passed, 1 failed
                                          Failures:
                                          1. [FAIL] test_fail
                                               test_demo.py:3: in test_fail
                                               E  AssertionError: one is not two

我们很喜欢这个想法，因此决定认真测试。README 没有回答两个问题：

首先，真实 Agent 会话中究竟有多少内容来自 Bash 输出？ 节省量表格假定 Agent 会通过 shell 完成所有工作。但 Claude Code 使用内置的 Read 工具读取文件、使用 Grep 搜索，两者都会完全绕过 Bash hook（rtk 的文档也承认这一点）。这些工具承载的内容，rtk 永远无法触及。

其次，压缩是否会牺牲正确性？ 对测试输出进行总结的过滤器，实际上是在判断模型需要哪些信息。如果它丢掉了唯一关键的一行，Agent 可能会重新运行命令、原样读取文件，或者更糟糕的是，在构建失败时宣告成功。若 Token 节省伴随质量代价，那就不是真正的节省。

测试设置

由于 hook 会以机械化方式重写每一个符合条件的 Bash 调用，B 组衡量的是 rtk 的出厂配置能力上限：不存在“模型是否记得使用它”的争议空间。每次启用 rtk 的试验还会保存 rtk 自身的审计日志和分析数据库，以证明该干预措施确实被触发。

发现 1：绝大多数 Agent 字节根本不会经过 hook

在花费任何成本之前，我们重放了 83 份已有的基线运行记录（相同模型、相同基准测试），并询问：如果当时安装了 rtk，它实际上能处理多少内容？

存在两个结构性原因。第一，Claude Code 通过内置 Read/Grep 工具读取文件，这些操作完全绕过 Bash hook；rtk 自己的 README 也在脚注中承认了这一点。第二，Agent 在 shell 中执行的命令有一半是 python3 … 及其他未覆盖命令，另有六分之一使用了写入文件的管道、heredoc 和替换操作，而 rtk 有意拒绝重写这些命令。剩下的内容，即 Bash 调用的 33%，承载的工具结果字符数还不到 20%；而工具结果本身也只是一个会话被计为输入的一部分，因为相同上下文会在每一轮重新读取。即使把 rtk 所能处理的全部部分压缩 70%，其上限也只能达到约 3% 的输入 Token。这个数字无需花钱计算，并且准确预测了结果。

发现 2：没有 Token 节省，反而出现小幅且显著的成本增加

我们执行了 caveman 评测教会我们采用的分层测试流程。在 10 个刻意偏重 Bash 的任务上进行 k=1 冒烟测试（这是 rtk 的最佳场景）时，rtk 组的中位成本高出 35%。这很令人担忧，直到你发现：同一实验组中、同一任务的完全相同尝试，其成本中位差异本来也有 22%。在 k=3 时，大部分惊吓都消散为噪声（Wilcoxon p≈0.65），这正是 k=1 幻象应有的表现。

随后，完整的 86 个任务给出了可抵抗噪声的答案，而且结果并非零。在 80 个有效配对中，启用 rtk 的实验组每任务中位成本增加 7.6%（p=0.004；这一结果已修正我们过程中发现的一处成本核算缺口），回合数增加 13.8%（p=0.03），缓存读取次数增加 14.3%（p=0.008）。与此同时，rtk 唯一实际压缩的 Token 类别——“新增输入”——仅变化 +3.2%（p=0.23）：正如上限分析所预测的那样，在全部潜在收益本应出现的位置，结果是平坦的零效应。

hook 重写的命令越多，惩罚越大。在与标题结果相同的修正后成本口径下，高暴露任务配对的成本比基线高约 24%，而 hook 几乎未触及的任务配对仅高 5%。控制任务难度后，这一模式并未复现，因此“较难任务使用更多 Bash”似乎无法解释该现象。运行记录取证没有发现单一罪魁祸首：包括一次真正损坏的重写（复合 find 谓词被改写为用法错误，导致重试）、几次由压缩引发的重新读取，以及极端样本配对中大量正常波动。这是一种轻微且系统性的税负，而非戏剧性的故障。

发现 3：在高推理强度下，连这项惩罚也消失了

“你只在低推理强度下测试”是显而易见的批评，因此我们又以高推理强度重新运行了全部 86 个任务——这是本系列成本最高的单次运行。结果是：成本惩罚在该条件下没有复现。配对中位变化为**+0.1%**（p=0.99），回合数变化为 +0.0（p=0.74），质量仍然持平。在高推理强度下，模型似乎较少因压缩输出而浪费回合；不过，在 k=1 的条件下，我们只能说惩罚没有在这里出现，而不能说明两种推理强度机制可能确实不同。无论如何，rtk 在任何情况下都没有节省成本。

发现 4：任务质量得以保持

rtk 的 issue tracker 中令人担忧的故障模式——包括测试输出被过度过滤、退出码被掩盖、管道接收到压缩文本——几乎都没有出现。我们对回合数差异最大的六组冒烟测试配对进行了取证检查，在约 150 次 Bash 调用中，仅发现一次损坏的重写（与完整测试中相同的复合 find 故障模式），以及一次 Agent 有意绕过 hook 的情况。在这些记录中，没有读取恢复文件，也没有压缩管道生成错误计数（完整测试中只出现一次恢复文件读取）；额外回合绝大部分来自模型选择了不同的解决路径，而非 rtk 导致的混乱。完整测试中，低推理强度下的任务得分结果为 5 个更好 / 4 个更差 / 71 个持平，高推理强度下为 5 / 4 / 62（两者的符号检验 p=1.0）；这表明在计入部分得分后，两个实验组在质量上统计学不可区分。

有一个必须诚实说明的附注：在一个任务（dialogue-parser）中，rtk 自身二进制文件拒绝在该任务镜像中启动（它需要更新版本的 glibc），因此启用 rtk 的试验会在两次完整运行中都于设置阶段失败，而原生实验组获得 0.667 分。配对分析从两个实验组中均排除了该任务，但这是真实的兼容性故障，而非 Docker 噪声。即使将每个出错试验都按零分计算，两个实验组仍然持平（符号检验 p=1.0）。

发现 5：rtk 自己的记分板与实际账单

这是解释差距的关键发现。在低推理强度的完整运行中，rtk 内置分析工具（rtk gain）报告节省了 9620 万 Token——即其处理内容的 99.8%；但相同试验的实际计费反而上升了。记分板读数偏高由三种机制造成：

第一，rtk 将完整原始输出视为反事实基准。一条针对 1.2 MB CSV 的 cat 命令记录了节省 320k Token，但 Claude Code 会在远早于 320k Token 的位置截断任何工具结果；因此，无论是否使用 rtk，Agent 实际接收到的都只会是数千 Token。完整测试记录了 190 次这类超大读取，每次平均“节省”约 506k Token。第二，rtk 在命令执行时以字符数除以 4 的方式估算 Token，而会话输入成本的大部分来自缓存重读，其计费价格仅为十分之一。第三，hook 根本看不到大多数上下文。这个记分板是在批改自己的作业。

结论

工程实现诚实，但反事实基准错误。 我们本希望它能胜出；这个演示实际操作起来确实令人满意。过滤器真实存在，而且往往很优雅；质量没有受损；hook 机制也完全按照设计工作。但在真实的 Agent 编码工作中，宣传的 60% 至 90% 节省从来没有足够的发挥空间：hook 只能看到约五分之一的工具输出，Claude Code 已会截断 rtk 所宣称压缩的病态超长输出，而主导输入成本的缓存重读又以十分之一价格计费。最终留下的是：低推理强度下实测中位成本增加 7.6%，高推理强度下则为平坦的零变化；代价来自这里一次损坏的重写、那里一个额外的探索回合，从未形成节省。

这一更深层的教训可以推广到 rtk 之外：工具自报的节省量，是对其反事实基准的声明，而不是对你账单的声明。 rtk 的记分板称节省了 9600 万 Token，而账单却上升了。评估任何上下文压缩工具时，应测量配对账单，而不是工具自身展示的差异。

方法说明

与第 1 部分相同的严谨性——这是我们以高昂代价学到的：

• 永远不要相信 k=1。 运行阶梯为：免费运行记录重放 → 1 次试验的布线检查 → 10 个 Bash 密集型任务的 k=1 → 相同 10 个任务的 k=3 → 全部 86 个任务以 k=1 运行两次（低与高推理强度）。两个实验组中，同一任务的分数可在不同尝试间自由波动；只有经由该阶梯仍然成立的配对差异才会被报告。
• 仅使用配对分析。 每个数字都在相同 job 下比较两个实验组中相同的任务；任一实验组发生错误的任务都会从两组中排除。质量采用非平局结果的精确符号检验；Token/成本差异使用每任务中位数加 Wilcoxon 符号秩检验，因为实验组总量由离群值主导：单个会话一旦跨过 200k 长上下文定价层级，计费可能达到正常水平的 25 倍，并改变原始总计结果。
• 终点指标预先注册。 主要指标为每任务成本与“新增输入”Token（未缓存输入 + 缓存创建；即压缩工具结果真正进入的位置）的配对变化。在所有付费运行之前便已确定，同时确定了按采用程度分层的拆分方式。
• 采用情况经过埋点，而非假设。 每次启用 rtk 的试验都会保存 rtk 的 hook 审计日志和 history.db，因此我们可以逐次证明重写被触发并执行——并区分“rtk 没有节省任何内容”和“rtk 从未运行”，两者是截然不同的发现。视运行而定，在扣除我们自身每次试验的布线检查后，hook 重写了其看到的三分之一至一半 Bash 调用（33%–50%）；模型自身在 86 次试验中仅输入过 6 次 rtk。
• 来源。 rtk v0.43.0 发布版二进制文件（固定 sha256）、两个实验组均固定 Claude Code 2.1.201、claude-sonnet-5、Harbor 0.18、SkillsBench；其中排除了 bike-rebalance（其 allow_internet=false 会使本地 Docker job 崩溃）。rtk、Harbor 和 SkillsBench 均采用 Apache-2.0 许可证。

本系列下一篇：说出一个工具名称，我们就会把它放上测试阶梯。少说几句话就够了。我们负责测试。

P.S. 本文采用的抖动图表风格，借鉴自 grim 制作的 dither-kit，并以无依赖的内联组件方式从零重新实现。

AI AI Assistant

Does “rtk” skill really cut agent tokens by 60–90%? We tested it

Does “rtk” reduce Claude Code token usage?

Part 2 of a series where we take public “token saving” add-ons for coding agents and run the same paired A/B benchmark against each of them. Part 1 was the caveman skill (advertised −65%, measured −8.5%).

TL;DR: rtk advertised saving: 60–90%. Measured on real agent work: +7.6% more expensive at low reasoning effort (p=0.004), ±0% at high effort. Setup: Claude Code 2.1.201 · claude-sonnet-5 low and high efforts · SkillsBench. Task quality: unchanged in both arms, at both effort levels.

Why we ran this

rtk (“Rust Token Killer”) is a CLI proxy with a simple, appealing pitch: your agent runs git status, rtk intercepts it, runs the real command, and hands the model a compressed version of the output — * master / M a.txt / ?? b.txt instead of eleven lines of porcelain. A Claude Code PreToolUse hook rewrites eligible shell commands transparently, so the model doesn’t even have to know rtk exists. The README promises 60–90% less token consumption and walks through a 30-minute session where 118k tokens of command output become 24k.

The compression itself is real and often tasteful. Here is rtk on a live repo, captured from our test container:

# git status                              # rtk git status
On branch master                          * master
Changes not staged for commit:             M a.txt
  (use "git add <file>..." to update…)   ?? b.txt
	modified:   a.txt
Untracked files:
  (use "git add <file>..." to include…)
	b.txt
no changes added to commit …

# python -m pytest  (19 lines)            # rtk pytest
…full pytest output…                      Pytest: 2 passed, 1 failed
                                          Failures:
                                          1. [FAIL] test_fail
                                               test_demo.py:3: in test_fail
                                               E  AssertionError: one is not two

We liked the idea enough to test it properly. Two questions the README doesn’t answer:

First, how much of a real agent session is Bash output at all? The savings table assumes the agent shells out for everything. But Claude Code reads files with its built-in Read tool, searches with Grep, and both bypass the Bash hook completely (rtk’s docs acknowledge this). Whatever those tools carry, rtk can never touch.

Second, does compression cost correctness? A filter that summarizes test output is making an editorial judgment about what the model needs. If it drops the one line that mattered, the agent re-runs commands, reads files raw, or, worse, declares victory on a failing build. Token savings that come with a quality tax are not savings.

Setup

Because the hook rewrites every eligible Bash call mechanically, arm B measures rtk’s as-shipped ceiling : no “did the model remember to use it” gap to argue about. Every with-rtk trial also persists rtk’s own audit log and analytics database, as proof the treatment actually fired.

Finding 1: Most agent bytes never touch the hook

Before spending anything we replayed 83 existing baseline transcripts (same model, same benchmark) and asked: if rtk had been installed, what could it even have touched?

Two structural reasons. First, Claude Code reads files with its built-in Read/Grep tools, which bypass the Bash hook entirely; rtk’s own README admits this in a footnote. Second, half of what agents run in a shell is python3 … and other uncovered commands, and a sixth uses pipes-to-files, heredocs and substitutions that rtk deliberately refuses to rewrite. What’s left, 33% of Bash calls, carries just under 20% of tool-result chars; and tool results are themselves only a slice of what a session bills as input, because the same context is re-read on every turn. Squeeze rtk’s whole share by 70% and the cap works out to ≈3% of input tokens. This number cost nothing to compute, and it predicted the outcome.

Finding 2: No token savings; but a small, significant cost increase

We ran the ladder the caveman eval taught us to run. The k=1 smoke on ten deliberately Bash-heavy tasks (rtk’s best case) showed the rtk arm a median +35% more expensive. Alarming, until you know that identical attempts of the same task in the same arm differ by a median 22% in cost anyway. At k=3 most of the scare evaporated into noise (Wilcoxon p≈0.65), exactly as a k=1 mirage should.

Then the full 86 tasks gave the noise-resistant answer, and it wasn’t zero. Across 80 clean pairs the with-rtk arm came out a median +7.6% more expensive per task (p=0.004, after correcting a cost-accounting gap we found along the way), on +13.8% more turns (p=0.03) and +14.3% more cache reads (p=0.008). Meanwhile “new input”; the only token class rtk actually compresses, moved just +3.2% (p=0.23): a flat null precisely where the ceiling analysis said the entire benefit had to live.

The more commands the hook rewrote, the larger the penalty. On the same corrected cost basis as the headline result, heavily exposed task pairs cost about 24% more than baseline, versus 5% for pairs the hook barely touched. Controlling for task difficulty did not reproduce this pattern, so harder tasks using more Bash does not appear to explain it. Transcript forensics found no single villain: one genuinely broken rewrite (compound find predicates turned into usage errors and retries), a few compression-induced re-reads, and a lot of ordinary variance on the extreme pairs. A thin, systematic tax rather than a dramatic failure.

Finding 3: At high effort, even the penalty disappears

“You only tested at low reasoning effort” was the obvious critique, so we ran all 86 tasks again at high effort – the most expensive single run of the series. Result: the cost penalty does not replicate there. Median paired delta +0.1% (p=0.99), turns +0.0 (p=0.74), quality still tied. At high effort, the model seems to waste fewer turns reacting to compressed output; though at k=1 all we can say is that the penalty didn’t show up there, not that the two effort regimes probably differ. Either way, at no point did rtk save anything.

Finding 4: Quality survives

The scary failure modes from rtk’s issue tracker including over-filtered test output, masked exit codes, pipes fed compressed text, they barely materialized. A forensic pass over the six smoke pairs with the biggest turn deltas found exactly one broken rewrite (the same compound-find failure mode the full run hit) and one case of the agent deliberately bypassing the hook, across ~150 Bash calls. In those transcripts no recovery files were read and no compressed pipe produced a wrong count (the full runs saw exactly one recovery-file read); the extra turns were overwhelmingly the model choosing different solution paths, not rtk confusion. On the full runs, task scores landed at 5 better / 4 worse / 71 tie at low effort and 5 / 4 / 62 at high (sign test p=1.0 both); showing the arms are statistically indistinguishable on quality, with partial credit counted.

One honest asterisk: on one task (dialogue-parser) rtk’s own binary refused to start inside the task’s image (it needs a newer glibc), so the with-rtk trial died at setup in both full runs while the plain arm scored 0.667. Paired analysis excludes that task from both arms, but it’s a real compatibility failure, not Docker noise. Even scoring every errored trial as zero, the arms stay tied (sign test p=1.0).

Finding 5: rtk’s own scoreboard vs the bill

This is the finding that explains the gap. Across the low-effort full run, rtk’s built-in analytics (rtk gain) reported 96.2 million tokens saved — 99.8% of everything it touched ; while the measured bill for the same trials went up. Three mechanisms make the scoreboard read high:

First, rtk counts the full raw output as its counterfactual. One cat of a 1.2 MB CSV logged 320k tokens “saved”, but Claude Code truncates any tool result long before 320k tokens; so the agent would have received a few thousand either way. The full run logged 190 of such giant reads at an average of ~506k “saved” tokens each. Second, rtk estimates tokens as chars÷4 at the moment of execution, while most of a session’s input cost is cached re-reads billed at a tenth of the price. Third, the hook simply never sees the majority of context. The scoreboard is grading its own homework.

Verdict

Honest engineering, wrong counterfactual. We wanted this one to win; the demo is genuinely satisfying to play with. The filters are real and often elegant; quality doesn’t suffer; the hook mechanism works exactly as designed. But on real agentic coding work the advertised 60–90% never had anywhere to live: the hook only ever sees about a fifth of the tool output, Claude Code already truncates the pathological outputs rtk brags about compressing, and the cached re-reads that dominate input cost bill at a tenth of the price. What’s left is a measured median +7.6% cost increase at low effort and a flat zero at high effort , a thin tax from a broken rewrite here and an extra exploration turn there, never a saving.

The deeper lesson generalizes beyond rtk: a tool’s self-reported savings are a claim about its counterfactual, not about your bill. rtk’s scoreboard said 96 million tokens saved while the invoice went up. If you evaluate any context-compression tool, measure the paired bill, not the tool’s diff.

Methodology notes

Same discipline as part 1, learned the expensive way:

• Never trust k=1. The run ladder was: free transcript replay → 1-trial wiring check → 10 Bash-heavy tasks at k=1 → the same 10 at k=3 → the full 86 at k=1, twice (low and high effort). Per-task scores flip freely between attempts in both arms; only paired deltas that survive the ladder get reported.
• Paired analysis only. Every number compares the same task across arms under the same job; tasks that errored in either arm are excluded from both. Quality uses an exact sign test over non-ties; token/cost deltas use per-task medians plus Wilcoxon signed-rank, because arm totals are outlier-dominated, a single session crossing the 200k long-context pricing tier can bill 25× normal and flip a raw total.
• Endpoints pre-registered. Primary: per-task paired delta in cost and in “new input” tokens (uncached + cache-creation; where compressed tool results actually land). Decided before any paid run, along with the adoption-stratified split.
• Adoption instrumented, not assumed. Every with-rtk trial persists rtk’s hook audit log and its history.db, so we can prove per-trial that rewrites fired and executed – and distinguish “rtk saved nothing” from “rtk never ran,” which are very different findings. Depending on the run, the hook rewrote a third to a half of the Bash calls it saw (33–50%, after discounting our own per-trial wiring check); the model itself typed rtk six times in 86 trials.
• Provenance. rtk v0.43.0 release binary (sha256-pinned), Claude Code 2.1.201 pinned in both arms, claude-sonnet-5, Harbor 0.18, SkillsBench with bike-rebalance excluded (its allow_internet=false crashes local Docker jobs). rtk, Harbor and SkillsBench are all Apache-2.0.

Next in the series: drop a tool name and we’ll put it on the ladder. Few word enough. We test.

P.S. The dithered chart style in this post is borrowed with admiration fromdither-kit by grim — reimplemented from scratch as a dependency-free inline widget.

Subscribe to JetBrains AI Blog updates