使用 Claude 实现评估设计与爬坡优化的自动化

评估（Evaluation）能为你提供应用或技能在特定任务中表现的量化信号。然而，设计合理的评估体系，并在不自欺欺人的前提下据此提升性能，往往是一件难事。我们已在 claude-api skill 中加入了针对这两方面的指导方案。

通过该技能，你可以运行 /claude-api build-eval 在你的代码库内构建评估体系，并运行 /claude-api hillclimb 针对该评估逐步优化你的应用；整个过程通过引入保留数据集（held-out set）来防范过拟合。

在本文中，我们将首先阐明良好评估设计与爬坡优化（Hillclimbing）的核心原则，随后展示配备 claude-api 技能的 Claude Code 如何践行这些原则，最后以这些命令的实际示例作结。

评估设计（EVAL DESIGN）

设计良好的评估通常具备以下几个共同要素（图 1）：

1. 评估任务反映真实生产环境（Eval tasks mirror production）。从你关心的“生产环境”中抽样任务，即被测能力或应用最终投入使用的实际场景。有时人们选择某些任务只是因为它们容易生成或容易打分，但至关重要的是确保任务分布能够代表你真正关心的业务场景。
2. 性能随模型变强和思考深入而提升（Performance improves with stronger models and more thinking）。能力更强的模型和更高的思考投入（effort levels）通常应在评估中获得更好的表现。如果并非如此，往往意味着模糊不清的任务描述或失准的打分器正在拖累整体评测表现。
3. 前沿水平具备“可通过”的上升空间（There is “passable” headroom at the frontier）。在最高思考投入下，最强前沿模型的得分也应明显低于 100%，否则你将无法可靠地判断改动对性能带来的影响。需要注意的是，这一差距不应归咎于不可能完成或表意模糊的任务：一个常见的表象是某个任务无论重复测试多少次都始终失败。一个好的任务应当满足：两位领域专家能够得出相同的评判结论，且打分器检查的所有要素均已在任务描述中清晰说明。
4. 多次运行之间的方差低（Low run-to-run variance）。高方差通常由设计粗糙、模糊的任务，或是对相同输出给出不同判决的打分器所导致。方差也可能隐蔽存在于配置之中。例如，思考投入未能保持一致。此外，运行环境也会干扰评估结果：先前测试遗留的状态（如某个文件、git 历史记录）可能会直接把答案“喂”给 Agent。

消耗 Token 对应的得分情况

三种模型尺寸在三种思考投入设置下的示意图

满分 / 低投入 / 中投入 / 高投入 / 每次尝试消耗的 Action token（对数刻度） / 得分 / 前沿模型 / 中型模型 / 小型模型

1. 能力更强的模型得分更高
2. 更多的思考投入得分更高
3. 距离满分保留一定的上升空间（Headroom）
4. 运行间方差较低

图 1 优秀评估体系的四个要素

对抗性抽样（Adversarial sampling）

模型的能力表现往往是参差不齐的（jagged）。如果你仅仅因为当前模型在某些样本上失败而挑选它们，你实际上是在对单一模型能力曲面的“低谷”进行抽样（图 2）。这会导致评估最终测出的只是该特定模型的“失败指纹”，而非应用场景中本质上困难或有价值的核心能力。

图 2 对抗性抽样

挑选困难用例应基于人类专家的难度评判：一个有效的检验标准是在将任务纳入评估集之前，你能够明确陈述它究竟难在哪里。同时，纳入源自生产环境真实流量、Bug 报告或工单中暴露的具体失败用例。但是，不要盲目信任用户流量：用户有时只会尝试他们预期能够成功的功能，因此完全取自用户流量的任务分布往往会偏简单。

/CLAUDE-API BUILD-EVAL

claude-api 技能中的 build-eval 命令将上述原则转化为了引导式工作流。当你在 Claude Code 中运行 /claude-api build-eval 时，Claude 会对你进行访谈，在你的代码库中构建评估，并在特定关键节点暂停等待你的批准。

设计样本用例

Claude 会按以下优先级协助你抽样输入以构建评估：

1. 生产环境调用记录（Transcripts），在确认数据留存与敏感信息规范后进行。
2. Bug 报告与技术支持工单。
3. 由你手工编写的 5 到 10 个用例。
4. 从你的代码库中合成生成的用例。

该技能优先使用生产流量，但它也能基于你提供的少量真实样本来生成锚定合成数据。该技能会指示 Claude 生成一个简易页面，展示所有输入样本并等待你的确认。作为演示，下图展示了该技能可能要求用户审核的邮件路由应用示例输入集（图 3）。

图 3 技能生成的示例输入审核页面

验证打分器（Grader）

在确定输入之后，Claude 会推荐契合你应用输出特性的成本最低的打分方案：

• 程序化验证（Programmatic verification）：如果输出可能性的范围是受限的，它将采用基于代码的检查（精确匹配、固定集合中的标签、符合 Schema 的 JSON、通过单元测试等）。
• LLM 即评审员（LLM-as-judge）：如果输出空间是开放式的——存在多种有效答案但有清晰的质量标准——它将默认采用此方式。在此模式下，第二个模型会读取输入、输出以及以可验证断言形式编写的标准量表（Rubric，而非 1 到 5 分的主观打分），并返回带有推理过程的得分。如果你有可供对比的基准线（Baseline），评审员将以随机顺序读取两种输入（且不被告知哪个是基准），进而选出更优的一方。你可以指定评审模型，且该模型不应是你正在测试的被测模型。

Claude 会试评少量用例，并询问你是否会对其中任何一项给出不同的分数（图 4）。一般来说，在彻底信任你的评估器之前，阅读一部分带评分的记录样本至关重要；打分机制失效是评估配置错误的最常见根源之一。

当你验证通过打分器后，该技能会告知你评估集的规模（用例数 × 重复次数 × 模型数，以及预估耗时），运行基线测试，并输出带有置信区间的得分。你最终将获得：用例集、打分器、运行器、每个用例对应的单行 JSON 和完整记录，以及一个列出每个用例得分并附带记录链接的纯网页。如果你需要页面展示更多信息（例如图表），只需提出要求，Claude 就会在其旁生成额外的配套页面。默认情况下，这些额外页面是本地打开的静态文件，不会从网络加载任何内容。

图 4 为各输入用例生成建议评分的结果页面示意图

诊断检查

在上述基准运行期间，Claude 会执行一系列排查项：

• 打分器一致性（Grader）：Claude 对同一输出运行两次打分器，并报告判决结果是否发生了改变。
• 底层通路稳定性（Plumbing）：Claude 检查是否存在超时、API 报错以及答案被截断的情况，以确保基础设施层面的噪声不会被误判为模型方差。
• 上升空间（Headroom）：如果基线得分已经达到约 95% 或更高，该技能会提醒用户并警示爬坡优化应将探索重点放在成本或延迟上，而非质量。

爬坡优化（HILLCLIMBING）

一旦拥有了对应用在特定任务上性能的可靠评分手段，你就可以尝试对其进行优化。爬坡优化（Hillclimbing）是一种用于微调诸如思考投入（effort）或提示词等参数的高效方法，它能够在成本与性能之间进行权衡。关于选择优化切入面的一些通用建议：

• 低迭代成本（Cheap iteration）：针对你在爬坡优化中所聚焦修改的任何层面，修改成本（在时间、资金与精力维度上）应当是低廉的。许多内部尝试和客户都将爬坡优化集中在文本层面，例如提示词和技能描述。这些内容极易修改与回滚。相比之下，在爬坡过程中对 Agent 框架代码（harness）进行开放式的修改可能涉及大规模的代码变动。
• 因果可归因（Attributable）：评估分数的变动应当能够明确归因于你在爬坡过程中修改的层面。例如，爬坡优化的若干成功落地案例都聚焦于技能触发（skill triggering）。其评估指标（技能的触发率）直接与被修改的技能描述相绑定。
• 目标边界清晰（Well-scoped objective）：一个常见的失败模式是提出开放式的“提升性能”要求，却没有仔细评估评测中所具备的上升空间（headroom）；一个接近饱和的评测体系或边界不清的修改面（例如开放式地要求更新框架）更容易陷入停滞。跨不同场景下普遍有效的一个强目标是成本优化：即便评估已经饱和，你依然可以要求 Claude 寻找降低成本的途径，同时维持性能持平。

过拟合（Overfitting）

即便是设计精良的评估，也难以完全复刻生产环境中真实的关注任务分布。因此，对评估集产生“过拟合”是一个常见陷阱，它会导致系统在评估集上的表现优于实际生产流量下的表现。

评估特征可能通过许多途径“泄漏”到你的框架代码中（harness，即模型周围的代码，包括提示词、工具以及调用 Claude 的循环）。例如，假设某个评估任务能够从 OCR 中获益，但 OCR 在你的生产任务中极少有用。评估框架可能会为你的应用添加一个 OCR 工具，这虽然拉高了基准得分，却对生产环境毫无助益。更广泛地看，爬坡优化可能会为框架添加专门解决你所选特定评估样例中极端边界情况（edge cases）的特性。这些框架层面的补充拉高了评估得分，却无法转化为生产环境中的收益（图 5）。

图 5 框架代码过拟合的常见成因

有三项措施有助于解决这一问题：

• 划分测试用例（Split the cases）：划分为爬坡优化算法可读取的训练集（train set）和完全未见过的测试集（test set）。如果训练集得分提升而测试集得分停滞不前，这就是典型的过拟合警报信号。
• 切勿将失败用例直接粘贴至提示词中（Never paste failures into the prompt）：如果爬坡优化模块读取了失败的运行记录，它绝不应将具体的失败内容直接硬编码粘贴到提示词中。
• 在结构层面上将答案置于模型无法触及之处（Keep the answers structurally out of the model's reach）：模型有时会通过直接定位评估答案来实现“奖励作弊（reward hack）”。

正如后文所述，claude-api 技能会自动为你应用这些原则。

/CLAUDE-API HILLCLIMB

claude-api 技能中的 hillclimb 命令将这些原则转化为了一套引导式工作流。当你在 Claude Code 中运行 /claude-api hillclimb 时，Claude 会针对给定的评估指标执行循环迭代优化。你可以选择允许其进行修改的范围，包括：

• 你的系统提示词（System prompt）
• 技能（Skills）或指令文件
• 工具描述（Tool descriptions）
• 模型选择、思考投入级别（effort level）及其他 API 参数
• 你的框架代码（Harness code）

在启动前，Claude 会询问你期望优化的目标（例如：纯性能提升，或在维持现有性能的前提下降低成本），随后将评估集随机划分为测试集与训练集。面对成本目标时，它会综合考量几个常见的成本影响要素，包括提示词缓存（Prompt Caching）、针对所选模型的提示词兼容性审计，以及甄选最适配的模型与思考投入级别。

在第一轮迭代之前，Claude 会检查评测噪声（仅凭随机偶然性分数可能波动的幅度）是否小于你所期望关注的最小提升幅度；如果不满足，它会给出明确警示并建议增加重复次数或测试用例数。

在每一轮迭代中，Claude 会读取上一轮训练集的运行记录，并以补丁（Patch）的形式提出一项修改。它在每一轮中都致力于实现效果足以突破评测噪声的改动：从根本上修复导致失败的行为（例如重写诱发错误的章节或补齐遗漏的规则），而非对某一行文字进行细枝末节的措辞修饰。随后，它会使用打上补丁的代码运行评估。在此阶段，Claude 会执行一道检查：如果 train 训练集得分提升但 test 测试集持平，Claude 会判定存在过拟合嫌疑并主动回滚该补丁。如果出现性能回退（Regression），Claude 同样会执行回滚。只有当训练集和测试集均取得提升时，它才会保留该补丁（图 6）。

Evaluations provide a signal on how your app or skill is performing on specific tasks. But designing evaluations, and improving performance on them without fooling yourself, is hard. We've added guidance for both to the claude-api skill.

With the skill, you can run /claude-api build-eval to build an evaluation inside your codebase, and run /claude-api hillclimb to improve your application against it, one change at a time, with a held-out set of examples to catch overfitting.

In this article, we highlight the principles of good eval design and hillclimbing first, then show how Claude Code with the claude-api skill applies those principles. We’ll close by showing a few examples of these commands.

EVAL DESIGN

Well designed evaluations have a few common elements (Figure 1):

1. Eval tasks mirror production. Sample tasks that you care about in “production,” or the setting in which the capability or application you are testing will be used. Sometimes tasks are picked because they are easy to generate or they are easy to grade. But it’s important to ensure that the task distribution represents what you actually care about.
2. Performance improves with stronger models and more thinking. More capable models and higher effort levels typically should perform better on an evaluation. If they don’t, ambiguous tasks or a miscalibrated grader often are hobbling performance.
3. There is “passable” headroom at the frontier. The most capable model at the highest effort should be well below 100% on the evaluation, otherwise you can’t reliably judge how changes impact performance. Importantly, the gap should not be explained by impossible or ambiguous tasks: a common tell is that a task fails every evaluation run, regardless of the number of replicates. A good task is one where two domain experts would reach the same verdict and everything the grader checks is stated in the task.
4. Low run-to-run variance. High variance is often due to poorly designed, ambiguous tasks or a grader that produces different verdicts on identical output. Variance can also hide in the configuration. For example, effort may not be applied consistently. Also, the environment can affect the results of the evaluation: leftover state from an earlier trial (a file, a git history) can hand the agent the answer.

Score against tokens spent

png

Three model sizes at three effort settings, illustrative

Perfect score low effort medium effort high effort Action tokens per attempt, log scale Score Frontier model Mid-size model Smaller model 1 2 3 4 1 A more capable model scores higher 2 More effort scores higher 3 Headroom below a perfect score 4 Low run-to-run variance Perfect score low medium high Action tokens per attempt, log scale Score Frontier model Mid-size model Smaller model 1 2 3 4 1 A more capable model scores higher 2 More effort scores higher 3 Headroom below a perfect score 4 Low run-to-run variance

FIG 1 The four elements of a good eval

Adversarial sampling

Model capability is jagged. If you pick cases because today's model fails them, you are sampling the valleys of one model's capability surface (Figure 2). The evaluation can end up measuring that model's failure fingerprint rather than what is intrinsically hard or valuable for your application to do.

FIG 2 Adversarial sampling

Pick hard cases because a human judged them hard: a useful test is to be able to say why a task is hard before you include it. Include cases that are specific failures in your application derived from production traffic, bug reports, or tickets. However, don’t blindly trust user traffic: users sometimes try what they expect to work, so a task distribution drawn strictly from user traffic may skew easy.

/CLAUDE-API BUILD-EVAL

The build-eval command in the claude-api skill turns these principles into a guided workflow. When you run /claude-api build-eval in Claude Code, Claude interviews you, builds the eval inside your codebase, and pauses for approval at specific points.

Designing examples

Claude helps you sample inputs to build evaluations in this order:

1. Production transcripts, after asking about retention and sensitive data.
2. Bug reports and support tickets.
3. Five to ten cases you write by hand.
4. Cases synthesized from your codebase.

The skill prioritizes production traffic, but it can also generate synthetic data anchored in a few real examples that you provide. The skill instructs Claude to generate a simple page that shows you every input and waits until you confirm them. As an illustration, below we show an example set of inputs for an e-mail router application that the skill may ask the user to review (Figure 3).

FIG 3 Example inputs review generated by the skill

Validating the grader

After the inputs, Claude proposes the cheapest grader that fits your application’s output:

• Programmatic verification : If the output possibilities are constrained, it uses a code based check (exact match, a label from a fixed set, JSON that matches a schema, tests that pass).
• LLM-as-judge : It will default to this type of check if the output space is open-ended, with many valid answers but clear quality criteria. In this case, a second model reads the input, the output and a rubric written as checkable claims (not a 1-to-5 scale), and returns a score with its reasoning. If you have a baseline to compare against, the judge instead reads both inputs in random order, without being told which is the baseline, and picks the better one. You pick the judge model, and it should not be the model you are testing.

Claude grades a handful of cases and asks whether you would have scored any of them differently (Figure 4). In general, it is important to read a sample of scored transcripts before believing your evaluator; scoring failures are among the most common ways an evaluation is misconfigured.

When you’ve validated the grader, the skill tells you the size of the evaluation set (cases × repeats × model, and roughly how long it will take), runs the baseline, and prints the score with a confidence interval. What you get back: the cases, the grader, the runner, one JSON line and one full transcript per case, and a plain page that lists each case's score with a link to its transcript. If you want more than that page shows (e.g., a chart), just ask and Claude will build it as an extra page next to it. By default, these extra pages are static files that open locally and load nothing from the network.

FIG 4 Schematic of the results page generated with suggested grades for each input.

Diagnostic checks

During the baseline runs mentioned above, Claude checks a number of things:

• Grader : Claude runs the grader twice on the same output, and reports whether the verdict changed.
• Plumbing : Claude checks for timeouts, API errors, and cut-off answers to ensure infrastructure noise doesn't pass as model variance.
• Headroom : if the baseline already scores about 95% or higher, the skill warns the user and alerts that the hillclimb should aim to explore cost or latency rather than quality.

HILLCLIMBING

Now that you have a reliable means of grading your application’s performance on a task, you can try to improve it. Hillclimbing is an effective way to tune parameters like effort or prompts, which trade-off cost and performance. Some general tips for choosing where to apply it:

• Cheap iteration - It should be inexpensive (in terms of time, cost, and effort) to modify whatever surface you are focused on for hillclimbing. Many internal efforts and customers have focused hillclimbing on text, such as prompts and skills. These are easy to change and revert. In contrast, open-ended modifications to an agent harness during hillclimbing may involve extensive code changes.
• Attributable - Changes in the score on your evaluation should be attributable to the surface you are modifying during hillclimbing. For example, several successful applications of hillclimbing have focused on skill triggering. The evaluation metric (the trigger rate for the skill) is directly coupled to the skill description that is being modified.
• Well-scoped objective - One common failure mode is an open-ended request to improve performance without careful consideration of the headroom available in the evaluation; an evaluation that’s near saturation or a poorly scoped surface (e.g., an open-ended request to update the harness) is more likely to stall. One generally strong objective across various efforts is cost: even if an evaluation is saturated, you can ask Claude to find ways to reduce cost while keeping performance at parity.

Overfitting

Even a well-designed evaluation rarely matches the exact task distribution you care about in production. As a result, "overfitting" to an evaluation is a common problem and results in a system that performs better on an evaluation than on production traffic.

There are many ways an evaluation can "leak" into your harness (the code around the model, including prompts, tools, and loop that calls Claude). For example, consider an evaluation task that benefits from OCR, but OCR is rarely beneficial in your production tasks. The evaluation harness might add an OCR tool to your application, which improves on the benchmark without any impact on production. More broadly, hillclimbing may add features to that harness that address edge cases in the particular evaluation examples you’ve chosen. These harness additions improve your evaluation score, but don’t translate to improvements in production (Figure 5).

FIG 5 Common causes of harness overfitting.

Three things can help address this:

• Split the cases. Use a train set that the hillclimber may read and a test set that is never seen. If the train set scores improve while the test set scores stay flat, then that is a common overfitting warning sign.
• Never paste failures into the prompt. If the hillclimber reads the failing transcripts, it should never paste the failure content into the prompt.
• Keep the answers structurally out of the model's reach. Models can sometimes “reward hack” by directly finding answers to evaluations.

As discussed below, the claude-api skill applies these principles for you.

/CLAUDE-API HILLCLIMB

The hillclimb command in the claude-api skill turns these principles into a guided workflow. When you run /claude-api hillclimb in Claude Code, Claude iterates to improve against a given evaluation. You choose what changes it can make including:

• Your system prompt
• Skills or instruction files
• Tool descriptions
• Model choice, effort level, and other API parameters
• Your harness code

Before it starts, Claude asks what you want to optimize (e.g., performance, or cost while performance holds) and then splits the evaluation set at random into test and train. With a cost goal, it considers a few common cost drivers, including prompt caching, auditing the prompt for compatibility with the selected model, and picking the model and effort setting.

Before the first round, Claude checks that the eval's noise (how far the score can move by chance alone) is smaller than the smallest improvement you'd act on; if it isn't, it says so and suggests more repetitions or cases.

Each round, Claude reads the previous round’s train transcripts and proposes one change as a patch. It aims each round at a change whose effect can show above the eval's noise: it fixes the failing behavior at its root (e.g., rewrites the section that causes it or adds a missing rule) rather than rewording a line. It then runs evaluation with the patched change. At this point, Claude applies a check: if the train set improves but the test set is flat, Claude suspects overfitting and reverts the patch. If there is a regression, Claude reverts. If train and test sets improve, it keeps the patch (Figure 6).

FIG 6 The process used by the hillclimber.

When the score stalls for two or three rounds, Claude reads each remaining train failure and sorts it by cause. It does the same early if no single fix could gain more than the eval's noise, and suggests more repetitions or cases, rather than spending rounds on changes too small to measure. This step can catch ambiguous evaluation cases, harness errors, or run-to-run variance.

Only legitimate failures are included in more hillclimbing rounds.

When hillclimbing completes, Claude leaves your code at the version that did best on the test set for your goal. It reports the test result against the baseline with confidence intervals (Figure 7). If the gain is within noise, it says so and recommends against merging.

FIG 7 Schematic of the report generated following hillclimbing.

EXAMPLES

Hillclimbing for cost reduction

We ran /claude-api hillclimb on an internal customer support benchmark with the goal of reducing cost and improving performance. The benchmark included 44 tickets, with 30 used for the search and 14 held out. It started on Opus 4.8 at default (high) effort settings with 74.4% decision accuracy on the search tickets and a token cost of 4.6 cents per ticket.

The hillclimb first audited the prompt, removing mandatory tool-call rituals, a scratchpad step, and contradictory rules. Then it tried Opus 5.5 on low effort. This cleared the baseline accuracy bar at 87.8% and cut cost to 1.9 cents per ticket, less than half the starting cost.

Part of that saving comes from Opus 5.5's pricing: input and output tokens cost 20% less than on Opus 4.8, and cache reads cost 60% less. Because Opus 5.5 cleared the bar, the hillclimb then stepped down a tier to check whether a cheaper model could clear it too. Sonnet 5 on low effort scored about the same, 88.9%, at about half the cost, 1 cent per ticket (Figure 8).

The customer support benchmark

png

Four configurations along the adopted path

70% 80% 90% 100% 1¢ 2¢ 4¢ Cost per ticket, log scale Decision accuracy, search tickets Starting accuracy 1 Opus 4.8, high effort Baseline, 74.4% at 4.6¢ 2 Opus 5.5, low effort Audited prompt, 87.8% at 1.9¢ 3 Sonnet 5, low effort Same prompt, 88.9% at 1¢ 4 Sonnet 5, low effort Improved prompt, 98.9% at 1¢ 70% 80% 90% 100% 1¢ 2¢ 4¢ Cost per ticket, log scale Decision accuracy, search tickets Starting accuracy 1 2 3 4 1 Opus 4.8, high effort Baseline, 74.4% at 4.6¢ 2 Opus 5.5, low effort Audited prompt, 87.8% at 1.9¢ 3 Sonnet 5, low effort Same prompt, 88.9% at 1¢ 4 Sonnet 5, low effort Improved prompt, 98.9% at 1¢

FIG 8 Cost-focused hillclimbing.

Finally, Claude improved the prompt with routing rules and a refund-cap cross-reference, bringing Sonnet 5 to 98.9% at about the same cost. On the 14 held-out tickets that the search never saw, the final configuration scored 90.5% against the original setup's 78.6%, at about one fifth of the cost.

Hillclimbing for performance improvement

Another example is our claude-api skill, which provides guidance on using our APIs and general tips for working with Claude (including the sub-commands discussed in this article). We want to ensure our skill can correctly implement code that uses our APIs, and we built an evaluation set derived from our documentation to test the skill.

On our evaluation, the skill started at 66%. We gave the hillclimber access to documentation and our SDKs, allowing Claude to identify errors and self-correct them (Figure 9). Claude found that the skill was missing coverage of eight features.

Adding sections for them in the skill improved performance to 74%. It then found errors in C# and Java type tables, boosting performance to 77%.

The claude-api skill eval

png

From 66.1% to 87.9% in three phases of work

65% 70% 75% 80% 85% 90% 0 6 9 11 13 17 22 24 Hillclimbing round (0 is the baseline) Pass rate 1 2 3 66.1% 74.3% 74.0% 77.5% 77.2% 80.1% 84.0% 87.9% 1 Adding missing sections and type tables 2 Fixing how the skill tells Claude to write code 3 Fixing graders, plus more skill edits 65% 70% 75% 80% 85% 90% 0 6 13 17 24 Hillclimbing round (0 is the baseline) Pass rate 1 2 3 66.1% 74.3% 74.0% 77.5% 77.2% 80.1% 84.0% 87.9% 1 Adding missing sections and type tables 2 Fixing how the skill tells Claude to write code 3 Fixing graders, plus more skill edits

FIG 9 Performance-focused hillclimbing.

After the score stalled for two rounds, Claude analyzed the remaining failures and bucketed them by root-cause. A normal round makes one edit for the most common failure. This step makes no edit; it only sorts every remaining failure by cause. This reflection step was useful in a few ways:

• Reflecting across a collection of failures, the hillclimber found that the skill content was present but Claude was simply writing older API shapes (e.g., from its trained priors). To address, the hillclimber added a table near the top of the skill that guided Claude from the forms it remembered to the current ones: for example, from extended thinking with a fixed token budget, which the API now rejects on recent Opus models, to adaptive thinking, and from older versions of the web search and web fetch tools to the current ones. It also moved the C# and Java warnings against fixed-budget thinking above their adaptive-thinking examples. This improved performance to 80%.
• Tasks that never improved in performance despite addressing obvious content gaps are tells that the example or grader is flawed. One task asked for code that catches one error type, while its grader wanted a chain of at least three. Claude reworded the task. Another grader's instructions contradicted our docs, and testing the real API showed the docs were right. Addressing these, along with more skill edits, brought performance to ~88%.

GETTING STARTED

Note: Run claude update first. The claude-api skill ships inside Claude Code, so updating gets you the latest version of these commands.

CODEShellCopy

claude update

Then, in Claude Code:

PROMPTCopy

/claude-api build-eval
/claude-api hillclimb

These sub-commands can be used directly in Claude Code via the claude-api skill:

Run /claude-api build-eval if you want to generate an evaluation set for a particular problem. You can steer it by providing access to examples (e.g., traces). Claude will employ the guidance shared in this article to design the examples and grader, and ensure you approve the examples and the grader.

Run /claude-api hillclimb if you have an evaluation and want Claude to improve on this, guided by your goal (e.g., better performance, or lower cost while performance holds). Claude will employ the guidance shared in this article to check for overfitting while climbing and check for bugs in the eval itself, such as a grader that marks a correct-looking answer wrong or a harness error, both before the first round and whenever the score stalls.

With special thanks to Misha Khalman for skill development. With thanks to Misha Khalman, Michael Segner, Matt Bell, Matt Thanabalan, and Punit Shah for reviews, contributions, and product support.