smevals：用于评估模型、提示词与 harness 的小型评测套件

2026年7月31日 - 链接博客

smevals：用于评估模型、提示词与 harness 的小型评测套件。我一直在与 Jesse Vincent 的 Prime Radiant 应用 AI 研究实验室合作，构建这一评测框架，以帮助回答不同模型具备哪些能力的问题。

最终成果是 smevals：一款新工具，可在不同模型配置下运行小型评测套件，并对结果进行评分。

这篇博客详细介绍了该工具。以下是 10 秒钟版本：

1. 告诉你的编程 Agent 执行 run uvx smevals docs 来了解这个工具（该命令会输出 README）
2. 然后让它为你构建一个评测套件

创建评测后——其形式是一个包含若干 YAML 文件的目录——你可以像下面这样针对模型运行它：

uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6

运行与评分操作被分别处理——你可以使用以下命令，按照定义好的检查项为运行结果评分：

uvx smevals grade path-to-eval/

然后，你可以运行一个 localhost Web 服务器来浏览结果：

uvx smevals serve path-to-eval/

或者运行 smevals build 命令，将该报告构建为静态 HTML，随后可将其托管在任何地方。这里有一个示例，展示了我构建的一套评测，用于评价模型创作俳句的能力。

这个项目中最耗时的部分，是为它确定一套术语！以下是我最终确定的定义，引用自公告：

• 一项 eval（评测） 是为回答某个模型相关问题而设计的一组挑战，例如：该模型生成 SVG 的能力有多强？
• 每项 eval 由多个 tasks（任务） 组成。任务是一项具体挑战，例如“生成一只骑自行车的鹈鹕的 SVG”。
• 运行 eval 时，你会针对一个或多个 configs（配置） 执行它。每个配置都会指定待评估的模型，但也可包含其他待测试参数，例如不同的系统提示词、模型参数或 Agent harness。
• 一次 run（运行） 会记录特定配置被用于执行特定任务时发生的情况。runner（运行器） 则是执行一次运行的脚本。
• 收集到一次或多次运行后，你需要评估结果，以了解模型（或配置）的表现如何。这一过程由 grader（评分器） 完成，并产出一个 grade（评分结果）。
• 每个评分器会依次运行一组 checks（检查）。这些检查可以是简单操作，例如检查输出中是否包含特定字符串，或确认输出是否为有效 XML。它们也可以是更复杂的自定义操作——通过称为 checkers（检查器） 的脚本实现——包括使用其他模型来回答有关该次运行的问题。

多年来，我一直在尝试找出一种自己认可的评测方法。smevals 是我对这一想法的第三次迭代，而这次它让我感觉是对的。我期待未来进一步扩展它，也会将它用于我自己的一些项目。

31st July 2026 - Link Blog

smevals - a small eval suite for evaluating models, prompts, and harnesses. I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models.

The result is smevals , a new tool for running small eval suites across different model configurations and grading the results.

This blog entry describes the tool in detail. Here's the 10 second version:

1. Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README)
2. Then tell it to build you an eval suite

Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this:

uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6

Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using:

uvx smevals grade path-to-eval/

Then you can run a localhost web server to explore the results:

uvx smevals serve path-to-eval/

Or run the smevals build command to build that report as static HTML, which you can then host anywhere. Here's an example showing an eval suite I built to evaluate how well models can write haikus.

The most time-consuming part of this project was figuring out the vocabulary for it! Here's what I settled on, quoted from the announcement:

• An eval is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?
• Each eval is a collection of tasks. A task is a specific challenge, for example "Generate an SVG of a pelican riding a bicycle".
• When you run the eval you do so against one or more configs. Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses.
• A run records what happened when a specific config was used to execute a specific task. A runner is the script that executes a run.
• Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a grader , which produces a grade.
• Each grader runs a sequence of checks. These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called checkers), including using other models to answer questions about the run.

I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects.