Parsimony

现有基准测试会衡量任务解决率和成本，却不会对过量代码作出惩罚，而过量代码会增加代码库的长期维护难度。Parsimony 用于衡量 AI 模型新增了多少代码。

图表和排行榜需要 JavaScript。数据可在 GitHub 上查看。

该工具使用各基准已经发布的代码补丁，统计编码单元：语句、调用、比较、名称和字面量等语法元素。它不统计代码行数，因此把代码挤在一起不会获得优势；删除的代码也会被扣除：净单元数 = 新增单元数 − 删除单元数。

比较模型

图表 散点图：比较两个指标 柱状图：比较一个指标 显示我的运行结果

X 轴 Y 轴

如何解读

模型按其成功解决的任务中平均净新增单元数排序，从低到高。「净新增单元数（平均值）」也包含失败的尝试。任务是否解决取自 SWE-bench Verified，不重新运行测试。「已测量」表示能够测量的成功尝试次数。

Artificial Analysis Intelligence Index 是一项外部评分，按模型和推理力度匹配，不影响排名。

颜色较淡的模型由作者在前 3–10 个 DeepSWE Python 任务上运行（OpenAI 模型使用 pi，Anthropic 模型使用 Claude Code），并使用各任务自带的测试进行评定。这些结果未经验证，也不参与排名。GPT-5.6 Sol 以两种方式都运行过：作者运行结果的新增代码量是其已发布运行结果的 0.88 倍。

排行榜

任务
所有任务

方法

• 将每个补丁应用到任务代码上，并解析每个被修改文件的修改前后两个版本。不运行任何代码。
• 排除测试、文档和生成的文件，仅测量 Python 代码。
• 通过或失败结果来自 SWE-bench 已发布的数据，不重新运行测试。

局限性

• 代码量更少并不总是更好：单元数衡量的是规模，而不是可读性或设计质量。
• 编写更多代码的模型往往也能解决更多任务（在包含 26 个以上模型的排行榜中，Spearman 相关系数为 0.33–0.60）。
• 所有模型都使用同一个 Agent（mini-SWE-agent），但 Agent 仍会影响生成的代码量。

添加你自己的基准

有其他基准的代码量数据？下载示例文件，填入结果并打开，即可获得相同的图表和表格。文件始终保留在你的浏览器中。格式及分享方法。

下载示例 JSON 清除

打开基准 JSON

示例：检查是否存在负数

以下两段代码返回相同结果。

python def has_negative(values): found = False for value in values: if value < 0: found = True return found

def has_negative(values): return any(value < 0 for value in values)

Parsimony

Existing benchmarks measure solve rate and cost, but don't punish excess code, which makes a codebase harder to work with in the long term. Parsimony measures how much code AI models add.

The charts and leaderboard need JavaScript. The data is on GitHub.

It takes the patches that benchmarks already publish and counts coding units : syntax elements like statements, calls, comparisons, names and literals. Lines aren't counted, so cramming code isn't rewarded, and deleted code is subtracted: net units = added − removed.

Compare models

Chart Scatter: compare two metricsBars: one metric Show my runs

X axis Y axis

How to read this

Models are ranked by mean net units added on the tasks they solved, lowest first. "Net units added (mean)" also includes failed attempts. Solved comes from SWE-bench Verified; tests aren't rerun. Measured is how many solved attempts could be measured.

The Artificial Analysis Intelligence Index is an outside score, matched by model and reasoning effort. It doesn't affect the rank.

Faded models were run by me on the first 3–10 DeepSWE Python tasks (OpenAI models with pi, Anthropic models with Claude Code) and graded with each task's own tests. They aren't verified or ranked. GPT-5.6 Sol ran both ways: my run added 0.88× the code of its published run.

Leaderboard

Task
All tasks

Method

• Each patch is applied to the task's code, and both versions of every changed file are parsed. Nothing is run.
• Tests, docs and generated files are left out. Only Python is measured.
• Pass/fail comes from SWE-bench's published results; tests aren't rerun.

Limits

• Smaller isn't always better: units measure size, not readability or design.
• Models that write more code also tend to solve more tasks (Spearman 0.33–0.60 on the boards with 26+ models).
• Every model uses the same agent (mini-SWE-agent), but the agent still affects how much code gets written.

Add your own benchmark

Have code-size numbers from another benchmark? Download the example file, fill in your results and open it to get the same chart and table. The file stays in your browser. Format and how to share it.

Download example JSON Clear

Open benchmark JSON

Example: check for a negative number

Both return the same result.

def has_negative(values):
    found = False
    for value in values:
        if value < 0:
            found = True
    return found


def has_negative(values):
    return any(value < 0 for value in values)