🔶 用 10,745 美元预算让 Claude Code 循环运行，能改进企业 AI Agent 吗？

过去几个月，Agent harness（智能体运行框架）优化已经变得相当主流。我们过去几年一直在研究 Agent harness 优化，并且经常被问到：

“为什么不直接让 Claude Code 循环运行，以改进你的 Agent？”

这是一个非常诱人的想法，但我们对此有所保留，因此我们在一个已部署到生产环境的企业 AI Agent 上进行了基准测试（测试时该 Agent 尚未经过我们的优化）。我们还找到了其他一些流行的优化方法，并让它们相互对比：

1. 循环运行的 Claude Code

2. Autoresearch（Andrej Karpathy 的个人项目）

3. Autoagent（ThirdLayer 的 Kevin Gu 开源的 Agent 自动改进工具）

它们所优化的指标，是在设定召回率下限条件下的精确率。三种方法均从相同的 Agent 配置开始，使用同一份数据集和相同的评测器进行优化。我们总计花费了约 6.7 万美元来探索这项实验（包括在数据集上运行 Agent、评测器成本等），结果如下：

基线精确率为 0.734，三种优化器都明显超过了这一水平。

Claude Code 达到 0.818。

AutoResearch 达到 0.843。

AutoAgent 达到 0.877。

我们观察到一个有趣的现象：尽管三个优化器都获得了数万美元级别的计算预算，但它们很早就找到了各自的最佳方案。

数据表明，这些自动优化器能够发现一些容易获得的改进，但很快就会达到能力上限。如果只是让它们持续运行，并期待它们找到能够突破上限的 Agent 配置，你也可能浪费大量计算时间和资金。

要持续改进一个 Agent，不能仅靠增加算力和预算来暴力尝试。还有许多可做的优化，例如有策略地运行测试（这是成本最高的部分）、跳出局部最优、决定哪些候选方案值得继续推进，等等。

注：Claude Code 的启动时间晚于另外两个优化器，而且运行速度也更慢。我们还测试了 GEPA，但它的结果存在非常严重的过拟合。

*这项实验消耗了大量额度（credits）

Over the past few months, agent harness optimization has become quite mainstream. We’ve been researching agent harness optimization for the past few years and have often been asked:

“Why can’t you just put Claude Code in a loop to improve your agent?”

It’s a really enticing idea, but we had some reservations, so we benchmarked it on a production-deployed enterprise AI agent (before we optimized it). We found some other popular optimization methods, and we pit them all against each other:

1. Claude Code in a loop

2. Autoresearch (Andrej Karpathy’s side project)

3. Autoagent (Kevin Gu at ThirdLayer’s open source agent auto-improvement)

The metric they optimized for was precision with a recall floor. They all started with the same agent configuration and were optimized on a dataset with the same judge. We ended up using a total of ~$67k exploring this experiment (including agent runs on the dataset, judge costs, etc.), and here are the results:

The baseline precision was 0.734. All three optimizers comfortably beat it.

Claude Code reached 0.818.

AutoResearch, 0.843.

AutoAgent, 0.877.

An interesting observation we made was that even though the 3 optimizers were given tens of thousands of dollars in compute, they found the best solution very early on.

The data suggests that these auto-optimizers can pick up low-hanging fruit, but cap out quite quickly. You can also waste a lot of compute time and money if you just let them run, hoping they find an agent configuration that breaks through the ceiling.

To improve an agent consistently, you can’t brute force it with more compute and budget. There are many more optimizations to be made, such as strategically running tests (most expensive part), escaping local minima, deciding which candidates to continue, etc.

Note: Claude Code was started later than the other two optimizers and was also slower. We also tested GEPA, but its results were very overfit.

*A lot of credits were hurt in this experiment