🔶 代理化测试流程、LLM 基准测试及其他代理编码笔记

从去年11月开始，我相当大量地使用AI，整体体验十分有趣。一个代理会做一些事情，如果这些事情是人做的，你会立刻将其解雇。当然我的反应是假装这很了不起，并启动上千个代理，让它们做更多这样的事。

去年年中，我曾让GPT（可能是5.0或5.1）尝试寻找一个bug的来源。不出所料，那段代码没有测试，git bisect 也用不了，而且那是一个UI交互bug，我甚至不太有资格为它写测试。于是我请Codex在日期X和Y之间进行二分查找，找出引入这个bug的提交。Codex立刻告诉我，存在问题的提交在这个日期范围之后（这根本不可能是正确的）。我告诉Codex这不对，它接着告诉我一个提交，那一两次显然也不是出问题的提交。在我告诉它那些也不对之后，它又说某个看起来合理的提交就是问题所在。当我要求它证明或证伪它的理论时，它告诉我它编写了一个测试，并已确认所指的提交就是导致问题的提交。

然后，我请它通过在普通浏览器测试环境中展示完整的开发者端到端栈来为我演示，声称录制视频。它声称自己并没有权限这样做（这是个谎言），但它可以在playwright中用适当的测试代码生成提交前后复现过程的视频。视频很有说服力，显示该功能在提交前工作正常，提交后则失效。这让我感觉有些不对劲，于是我尝试在提交前后手工重现这个问题，结果发现这一切都是伪造的。视频让它看起来像是Codex复现了这个bug，但那个浏览器环境是人工设计的，旨在制造一个假的复现，而非真实环境。

正如我所说，因为这次经历不带讽刺意味地如此“美妙”，我立刻想：“我怎样才能得到更多这样的体验？”并开始越来越重度地使用代理，直到去年中后期我便在大量使用编码代理。

由于本文涵盖的主题较为分散，这里给出一个简要大纲。

测试背景

在测试方面，LLM 具备极高的杠杆效应。就所需投入的精力而言，达到特定质量门槛比以往任何时候都容易，然而软件质量似乎却比任何时候都低。十年前，我们曾回顾我在闲散一周里遇到的bug。那时就有不少bug，如今我遇到的bug更多，但我认为这种情况并非必然。

一个原因是，在bug被发布后，采用数据驱动的方法来查找和修复它比以往更容易。举个工作中的例子，我尝试构建了一条从支持工单（聊天或邮件）到拉取请求（PR）的流水线。据我所知，它运行良好。由于我所在的公司采用传统工作流，所有这些修复都会经过人工审核，并且迄今为止我们没有收到已知的误报。

就单位投入时间而言，进行更彻底的测试也是可能的。个人而言，我认为这可以足够有效，以至于我对通过“ 软件工厂”工作流交付大量代码感到相当安心，因为我见过一种测试密集、无审核的工作流，其产出质量远超我见过甚至听说过的任何依赖审核的工作流。

和每个人一样，我的偏见来自于我的经历。碰巧的是，我职业生涯的头十年是在一家公司度过的，那家公司的测试流程恰好适用于今天的LLM环境。我在Mastodon上提到过将Fuzzing作为默认测试方法，而一位怀疑者尝试后立刻发现了一些bug：

于是我重读了那篇博客，脸上写满了“怀疑”，但是没用，没错，Claude Fuzzing 找到了好几类值得修复的 bug。

我谈过的其他一些人也尝试采用了类似我们将在此讨论的测试流程，他们全都很快在他们所处的软件中发现了bug，包括那些仅仅要求Codex或Claude审计代码、查找bug、“测试”、“更多测试”等方式无法浮现的bug。例如，Dennis Snell 提到，他和队友 Jon Surrell 不仅在他们维护的代码中找到了bug，还以相当低的投入在“上游依赖中，包括HTML规范、三大浏览器以及其他开源项目”里找到了bug。

通常，当我和软件从业者聊测试时，我仿佛来自另一个星球，立刻被投以异样的目光。那么让我聊聊我曾任职的硬件公司 Centaur 如何进行测试，它塑造了我对工作方式的偏见。我们做过的在软件界曾经或依然非正统的事情包括：

1. 聘用专职QA/测试工程师，测试作为一等职业发展路径，与开发者平起平坐
2. 默认没有代码审核
3. 几乎没有手写测试
4. 通过程序员有时称之为基于属性的测试、随机测试、Fuzzing 等方式持续测试，尽管我们当时只称其为测试（手写测试被称为“手工测试”）
5. 庞大的回归测试套件（在计算集群上要跑3个月墙上时间）
6. 没有单元测试

仅为了让你对总体结构有个概念，当我离开时（2013年），我们拥有大约1000台机器，全天候不间断地生成并运行测试，支撑着大约20名逻辑设计人员和20名测试工程师。这些都是本地部署，这些机器占据了我们所在建筑半层楼。

整体结构是，我们约20%的机器运行回归测试，80%的机器生成并运行新测试。三个月的回归测试太长，无法用来卡提交，所以会有一个小得多的测试列表，大约10分钟就能跑完，人们在提交前会运行它们。那些预提交测试会在特别定制的、超频的、金钱所能买到的最快机器，以及不同的模拟器设置上，尽可能快地运行。

新的失败一旦发生就会被发现并报告，一到两名工程师负责梳理这些失败并进行分类（剔除误报、修复测试生成器中导致误报的问题等）。

就影响程度而言，除非把文化单独作为一项来考量，（1）可能是我们和典型软件公司之间最大的差别，但对本文读者来说也最无关，所以我把讨论放到了脚注里，只简短说一句：测试和任何其他技能一样；投入越多时间练习，技能就越强。由于在大多数大型科技公司，测试并非一等职业发展路径，软件公司里人们的测试技能通常无法与某些职业CPU测试工程师相提并论。正如一位花20年时间从事分布式系统或UX的工程师，会比一位天赋相当但只花5%时间在分布式系统或UX上的工程师强得多一样，花20年做测试的人，也会比在测试上只花5%时间的人强得多。

（2）是使得我们芯片公司使用的某些测试实践非常适合AI工作流的原因之一。我们默认不做代码审核，是因为我们对测试实践足够信任，以至于审核通常不会在可靠性上增加太多价值。我们每年平均交付不到1个用户可见严重bug，审核只在有人希望为某些他们认为特别棘手的代码多找一双眼睛时按需进行。借助AI编码工作流，一个人轻松就能生成比任何一个人甚至十个人手工审核还要多的代码。人们对于未经审核就交付代码的舒适程度各不相同。我个人对于没有人工审核就交付代码非常坦然，因为我见过这在比大多数软件公司的多数软件技术难度更高的产品上成功实施。

我常听到人们说，“那风险太大了；我们有几百万用户”，但从经验上看，他们所谈论的工作流在绝对数量上人均交付的 bug 率大约要高出一千倍，若按严重度调整，比率会更高。如果一家公司在交付bug的速率上仅有我们在 Centaur 时的百分之一，且主要依赖审核来捕捉bug，我或许能理解他们的观点。但在典型软件公司并非如此，人们不想放弃人工审核，是因为他们假想的交付bug风险。

（3）和（4）是相辅相成的。我认识的几乎所有认真对待可靠性的软件团队（各种交付可靠数据库、分布式数据库等的团队），至少都在方向上做着同样的事，尽管他们可能有更高比例的手写测试。同理，依赖通过亲自操作软件并观察软件是否似乎正常运行来进行测试被认为是不好的做法，依赖直接手动键入测试输入和期望输出也是不好的做法。 如之前讨论过的，手工编写测试效率极低。对于任何给定的可靠性水平，如果你优先选用随机测试生成而非手写测试，你会更快达到目标。

（5）源于大量测试找出大量bug。通常，如果某个测试发现了我们后来修复的bug，我们会将该测试永远保留在回归测试套件中。事实证明，如果你用好的测试找到了许多bug，你最终会得到一个大型测试套件。撇开这点不谈，单从测试效率来看，软件界那种在每次PR的CI中运行相同测试集合的标准做法效率极低，你只要想想，在同样的测试时间内，运行同一个测试一千次，与运行一千个不同的测试，哪个更有可能发现bug？

（6）也出自对测试效率的考量，因为我们的团队比竞争对手小得多。这是公司得以存续如此之久的一个原因。当Intel将所有x86设计商逼得除了AMD之外都无法生存时，我们的运营成本够低，使得公司坚持到了2021年，在那时以1.25亿美元被 Intel 收购。以公司如此微小的团队规模，若靠单元测试获得合理覆盖率是不可能的，而雇佣足够人手去做单元测试恐怕意味着公司会像 Transmeta、Rise、Cyrix、TI、UMC、NEC、VM 等 x86 项目一样，早一二十年就消亡了。从效率角度看，单元测试表现相当糟糕。

总结起来，我们做了好些大多数软件人告诉我是坏主意的事情（专门的测试工程师、无单元测试、无代码审核等），而且我们交付的质量比我工作过的任何软件公司或我用过的任何软件都要高得多。每当我谈起这些，人们总会说这不适用于软件，因为CPU只有X种关注点，而你不能对Y做同样的事。当我最初从CPU设计转行到软件时，觉得那可能是真的，但如今我已经用这种测试方法尝试过人们提及的每一种被认为不可能的Y，且每试必灵，所以我已觉得这不太可信（而各种X通常包含对硬件开发面貌的错误假设）。虽然硬件和软件之间存在真实差异，但我所见过的以此为由声称测试技术无法搬用的情形，通常是那人依赖某种想象出来的因素，因为对硬件开发不够了解才会觉得相关。

一个显著差异是测试与开发投入精力的比例，但Fuzzing的固定成本相当低，因此可以按任意投入水平伸缩，效率增益依然存在。而且，由于测试效率的提升，实际投入比例并不像软件工程师通常想象的那样大。我们的测试工程师与开发人员比例约为1:1，此外我们大约10%的时间处于“冻结”状态，目标是找bug而不交付新特性，因此对这个开销的零阶估计是，我们把55%的精力花在测试上，45%花在开发上，或者如果我们在测试上零投入，便可在开发上投入2.2倍的精力。如果你看看一家交付显著bug速度比我们快好多倍的软件公司，然后宣布紧急状态，让每个人将55%的精力花在测试上，我认为比率并不会改变太多。或许他们能把之前的bug率降低一半或类似幅度，但投入力度并非真正的差异所在。

如今，人们还会说，既然可以让LLM找bug，何必费劲Fuzzing呢？我多次尝试了这两种方式，我的经验是，Fuzzing 在发现bug的延迟上普遍胜出，并且在发现更多bug和拥有更低的误报率方面占据绝对优势。LLM 的方差相当高（后面会细说），所以有时直接让Codex或Claude找bug或许能赢，但平均而言，Fuzzing 都赢了。

关于测试的一些细节

尽管我对LLM测试说了这么多积极的话，LLM 似乎并不擅长测试。然而，对于任何层级的……

I've been using AI fairly heavily since last November and the whole thing is a funny experience. An agent will do something that, if a human did it, you'd immediately fire them. My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that.

Mid-last year, I had GPT (maybe 5.0 or 5.1) try to find the source of a bug. Naturally, this code didn't have tests and git bisect wouldn't work, and it was a UI interaction bug for which I'm not even really qualified to write a test for, so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug. Codex immediately told me the offending commit was after this date range (which couldn't possibly be correct). On telling Codex this was wrong, it then told me some commit that was obviously also not the offending commit once or twice. On telling it those were wrong, it then told me the offending commit was some plausible looking commit. When I asked it to prove or disprove its theory, it told me that it wrote a test and confirmed that the alleged commit was the breaking commit.

I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. It claimed that it didn't have permissions to do that (which was a lie), but it could make video of the execution of the repro before and after the commit in playwright with the appropriate test code. The video was convincing and showed the feature working properly before the commit and failing to work after the commit. Something about this didn't feel right, so I tried reproducing the issue by hand before and after the commit and found out that the whole thing was a fabrication. The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment.

Like I said, because this was non-ironically such a great experience, I immediately thought to myself, "how can I get more of this?" and started using agents more and more heavily until I was using coding agents heavily mid-late last year.

Since this post covers a relatively disparate set of topics, here's a brief outline.

Testing background

LLMs are highly leveraged when it comes to testing. In terms of the amount of effort it takes, it's easier than ever to hit a particular quality bar and yet, software seems to be lower quality than ever. A decade ago, we looked at the bugs I ran into in an arbitrary week. There were quite a few bugs then and I run into more bugs now, but I don't think this has to be the case.

For one thing, after a bug has been shipped, it's easier than it's ever been to use a data-driven approach to find and fix the bug. Just for example, at work, I tried creating a pipeline that goes from support ticket (chat or email) to pull request (PR). As far as I can tell, this works ok. Since I work for a company that has a traditional workflow, all of these fixes get reviewed by a human and, so far, we've had no known false positives.

Per unit of time invested, it's also possible to do more thorough testing. Personally, I think this can be effective enough that I'm fairly comfortable trying to ship a large volume of code via a "software factories" workflow because I've seen a testing-heavy no-review workflow that results in much higher quality than any review-reliant workflow I've seen or even heard of.

Like everybody, I have biases that fall out of my experiences. It just so happens that I spent the first decade of my career at a company whose test processes happen to work well in today's LLM environment. I talked about fuzzing as a default testing methodology on Mastodon, and a skeptic tried it out and immediately found some bugs:

so I reread the blog post and was very "dubious face" but no yeah, Claude fuzzing found several classes of bugs that are worth fixing

A number of other folks I've talked to have also tried adopting something like the testing flow we'll discuss here and they've all immediately found bugs in the software they work on, including bugs that don't get surfaced by just asking Codex or Claude to audit the code for bugs, find bugs, "test", "test more", etc. For example, Dennis Snell mentioned that he and a teammate, Jon Surrell, not only found bugs in the code they're working on, but also "in upstream dependencies, including the HTML specification, big-three browsers, and other open-source projects" with fairly low effort.

In general, when I talk to software folks about testing, I'm coming from such a different place that they immediately look at me like I'm an alien, so let's talk about how we tested at this hardware company I worked for, Centaur, which informs my biases about how I like to work. Some of the things that we did that were or are unorthodox in the software world are:

1. Hired dedicated QA / test engineers, with testing being a first-class career path on par with being a developer
2. No code review by default
3. Virtually no hand-written tests
4. Constant testing via what programmers sometimes called property based testing, randomized testing, fuzzing, etc., although we just called those tests (hand-written tests were called "hand tests").
5. Large regeression test suite (3 months wall clock to execute on compute farm)
6. No unit tests

Just to give you an idea of the general structure, when I left (in 2013), we had about 1000 machines generating and running tests at all times for roughly 20 logic designers and 20 test engineers. This was on prem and the machines took up half a floor of the building we were in.

The general structure was that we had maybe 20% of machines running regression tests, and 80% generating and running new tests. Three months of regression tests is too much to gate commits on, so there was a much shorter list of tests that took maybe 10 minutes or so to run that people would run before committing. Those pre-commit tests would run on a special setup to run as quickly as possible, with overclocked machines that were the fastest machines money could buy, as well as a different simulator setup.

New failures would get found and reported as they happened and one to two engineers had a job of sorting through failures and triaging them (rejecting false positives, fixing issues in the test generator that caused them to generate false positives, etc.).

In terms of the magnitude of the impact, unless you count culture as a separate item, (1) was probably the biggest difference between us and a typical software company, but also the most irrelevant for readers here, so I'll relegate the discussion to a footnote, except for this brief comment that testing is like any other skill; spending more time doing it improves skill and, since testing isn't a first-class career path at most major tech companies, people generally don't have the same level of testing skills at software companies as you see in some career CPU test engineers. In the same way that an engineer who who spends 20 years working on distributed systems or UX is going to be much better at it than an equally talented engineer who spends 5% of their time on distributed systems or UX, someone who spends 20 years working on testing is going to be much better at it than somebody who spends 5% of their time on testing.

(2) is one of the things that makes some of the test practices we used at the chip company suited to AI workflows. We didn't review code by default because we trusted our test practices enough that review didn't, in general, add much reliability. We were shipping fewer than 1 significant user-visible bug per year, and review was done on an as-needed basis when someone wanted an extra set of eyes on something they thought was particularly tricky. With AI coding workflows, it's easy for one person to generate more code than any human or even any ten humans can review by hand. People have different levels of comfort with shipping code without review. Personally, I'm very comfortable shipping code without human review because I've seen it done on products that are technically more challenging than most software at most software companies.

I often see people say things like, "that's too much risk; we have millions of users" but, empirically, they're talking about a workflow that ships bugs at a rate that's maybe a thousand times higher per capita on raw count, with the ratio being much higher if you adjust for severity. If a company were shipping bugs at, say, a hundredth the rate we were at Centaur while relying primarily on review to catch bugs, then I could see their point, but that's not what's happening at the typical software company where people don't want to move away from human review because of the perceived risk of shipping bugs.

(3) and (4) go hand in hand. Almost every software group I know of that's serious about reliability (various teams that ship reliable databases, distributed databases etc.) are at least directionally doing the same thing, although they might have a larger fraction of hand written tests. For the same reason it's considered a bad idea to rely on testing by interacting with the software yourself and observing whether or not the software appeared to work, it's a bad idea to rely on directly typing out the inputs to a test and the expected outputs. As previously discussed, it's just really inefficient to write tests by hand. For any given level of reliability, you'll get there more quickly if you prefer randomized test generation over hand-written tests.

(5) fell out of having a lot of tests find a lot of bugs. In general, if a test found a bug that we later fixed, we'd keep the test in our regression test suite forever. It turns out, if you find a lot of bugs with good tests, you'll end up with a large test suite. But putting that aside and just looking at it from a test efficiency standpoint, the standard setup in software of having the same set of tests run in CI for each PR is extraordinarily inefficient if you think about the what's more likely to find a bug, running the same test a thousand times in a day or, in the same amount of test time, running a thousand different tests.

(6) came out of test efficiency concerns as well, in that we had a much smaller team than our competitors. That was a reason the company managed to survive for so long. While Intel was putting every x86 designer out of business other than AMD, our operating cost was low enough that the company survived until 2021, at which point it was acquired by Intel for $125M. With the company's tiny team size, it wouldn't have been possible to get reasonable test coverage with unit tests and hiring enough to do unit tests probably would've meant the company would've gone the way of the x86 efforts of Transmeta, Rise, Cyrix, TI, UMC, NEC, VM, etc., a decade or two sooner. From an efficiency standpoint, unit testing does pretty poorly.

To sum it up, we did quite a few things that most software people tell me are bad ideas (dedicated test engineers, no unit tests, no code review, etc.) and we had much higher quality than any software company I've worked for or any software I've used. Whenever I talk about this, people will say that this doesn't apply to software because CPUs only have X concerns and you can't do the same thing with Y. When I first switched from CPU design to software I thought that might be true, but I've since tried this testing methodology with every kind of Y that someone has mentioned this can't work for and it's worked for every single one, so I no longer find this very plausible (and the Xs generally involved incorrect assumptions of what hardware development is like). While there are real differences between hardware and software, when I’ve seen people lean on that as a reason that testing techniques don’t carry over, it’s been the case that the person is relying on some imagined factor that only seems relevant because the person doesn’t know much about hardware development.

One significant difference was the ratio of effort that went into testing vs. development, but the fixed costs of fuzzing are fairly low, so this is scalable to any level of effort and the efficiency gains are still there. And, due to the gains in test efficiency, the ratio of effort wasn't as large as software engineers generally imagine. We had about a 1:1 ratio of test engineers to developers and then spent maybe 10% of our time in a "freeze" state, where the goal was to find bugs and not ship new features, so a zeroth order estimate for the overhead here is that we spent 55% of our effort on testing and 45% on development, or we could've put 2.2x the effort into development if we spent zero effort on testing. If you look at a software company that's shipping significant bugs many times faster than we did and you declare an emergency and get people to spend 55% of their effort on testing, I don't think the ratio changes too much. Maybe they get to half the previous ratio or something, but the level of effort isn't really what's making the difference.

Nowadays, another thing people will say is, why bother with fuzzing when you can just ask an LLM to find bugs? I've tried doing both quite a few times now and my experience has been that fuzzing generally wins on latency to find a bug, and it dominates on finding more bugs and having a lower false positive rate. LLMs have fairly high variance (more on this later), so just asking Codex or Claude to find a bug can sometimes win but, on average, fuzzing has won.

Some details on testing

Despite the very positive things I've said about LLMs testing, LLMs seem pretty bad at testing. However, for any level of testing effort, LLMs let you apply testing effort a lot more easily than before if steered properly.

An extreme example of this is that everybody I've talked to who cares about quality or testing at all finds the tests LLMs generate by default, or if you tell them "Write tests", "Write more tests", etc., to be poor. People tend to rate the tests as somewhere between worthless and marginally useful, depending on their standards.

For example, Em Chu (a compiler engineer) says:

The existing tests I'm working with aren't perfect, but are still above the bar LLMs seem to aim for, which I would describe as "thorough enough to smuggle a feature through human code review." For a compiler (compared to e.g. UI), where I'm guessing it's easier to write the average test, but a higher bar of correctness is generally expected of the end product, LLMs just suck. They are painfully bad at the adversarial "now, what if I do this" or "let's try the cross-product of everything" process humans use to write tests that actually find bugs

At the same time, I've seen a number of folks rave at how amazingly good LLMs are at testing when you tell them "Write tests", "Write more tests", etc. When I've looked into why people say LLMs are great at testing, what I've found is that people who did essentially no testing at all find LLMs to be great at testing. Well, that makes sense. If you go from basically zero testing effort to a tiny bit of testing effort, that's a huge win.

As of June 2026, directing LLMs to do fuzzing / randomized testing feels similar. I've tried using an LLM to generate a fuzzer and, for most projects, this will turn up real and often serious bugs within minutes. However, on looking at what the LLM-created fuzzer is trying to test, I have the same reaction as a normal programmer who cares about quality looking at LLM-created tests. The coverage of the LLM-generated fuzzer is curiously bad and misses all kinds of basic things you'd expect a hastily human-written fuzzer to cover. Depending on whether you're a glass half empty or a glass half full person, you might say that this says something about the test coverage of most projects, or that it says something about the unreasonable effectiveness of fuzzing.

At a high level, LLM-generated fuzzers from SOTA models today don't do a good job of "thinking about" how inputs should be varied to elicit bugs. Then, if you naively tell it about how inputs should be varied and to combine these, it will also not combine bug ingredients in a reasonable way. It's possible to give instructions that will work well, but this heavily relies on the user to provide direction.

If you're using randomized testing as "extra credit", to catch a few more bugs, or to replace traditional software testing processes, you can just tell an LLM to look for risky areas of the code and find invariants that might be violated and fuzz them. This works ok. When I've convinced people to try some randomized testing, they usually start here and find quite a few bugs they're happy to have found. Due to the nature of who's interested in trying out novel-to-them test techniques, this is often from people who've worked on some of the most well-tested and reliable code at the company and they can find bugs in their own relatively well-tested code.

If you want to use randomized testing to keep an agentic "software factories" workflow honest, then you need to have a way to deal with gaps in SOTA models because, when you're shipping the equivalent of hundreds or thousands of PRs a day into a project, everything that's not constrained from degrading will rapidly degrade.

At a high level, the entire system needs some kind of feedback that finds gaps and instructs whatever loop is making adjustments to the fuzzer to close the gaps. Recently, I've been testing things where I don't understand the domain and don't understand the project or the code, so I've been flying relatively blind and know that there will be a lot of gaps in what I come up with (and, as noted above, LLMs are terrible at this). But even in areas where I'm familiar with the domain and understand the code relatively well, there will still be some gaps because humans miss things and make mistakes, so there always needs to be some kind of feedback into the test setup that can find gaps and allow you or an agent to close the gaps.

I've been playing with various ways to have agents convene and reconvene to get agentic loops running better and, while that kind of thing helps, I haven't figured out a way to do do this well enough to create some kind agentic software quality improvement loop that doesn't rely on outside feedback, whether that's occasional human input, or shipping something (ideally only fractionally and with staged rollout) and then having the system monitor metrics/logs/traces/support tickets/whatever to use that as feedback. The support ticket to PR pipeline I mentioned above is one such feedback loop. The pipeline not only tries to generate a PR, it also tries to get the test setup to add test coverage that will find the bug and possibly surface other bugs, or will re-find the bug if there's a future regression. This seems to work ok-ish, in that it finds real bugs and improves test coverage, but I'm sure there's a lot of room for improvement.

Relatedly, I've been wondering why LLMs are so bad at writing tests. On asking around a bit, I'm told that this is because the capabilities that LLMs have come out of people building RL environments which allow models to improve at tasks, sometimes in a generalizable way and sometimes not. I'm also told that there's a market for selling RL envs, but it's fairly thin because there aren't all that many buyers for them, and you really want to know someone at a lab who's a buyer or close to it. If you are such a person or can connect me with such a person, could you do me a favor and reach out to me (I'm fairly easily reachable on X, Mastodon, email, etc.). I'm curious about how this works and how plausible it is to sell an RL env for something like testing, optimization, or the longer horizon tasks discussed in this post, where it's easy to observe significant gaps.

Back on the topic of testing, when fuzzing or doing any kind of bug auditing, detecting false positives is a critical part of the process. At least for now, having access to a model that's better than anything you can publicly use won't save you. A while back, Dennis Snell told me, frustratedly, that he spent the day wading through AI slop forwarded to his employer by Anthropic that came from their vaunted Mythos model that's too dangerous to release from the public. Anthropic was apparently doing the company some kind of favor or maybe doing some kind of EA security improvement, except that they didn't bother with having a reasonable false positive rejection process so they were just forwarding garbage to us. At the time, I was using a model that, if Fable is any indication, appears to be moderately less capable than Mythos, but I had no problem generating an endless stream of bugs (some of which were security issues) with no known false positives, which seems to indicate that having a reasonable setup around the model is a least as important as having the latest and greatest model.

I've been trying custom workflows on a per-project / per-problem basis and don't exactly have a generic false positive rejection scheme, but there are various things that seem to be semi-generalizable. If you don't mind spending tokens, having independent agents repeatedly check an alleged bug reproduction (repro) substantially cuts the false positive rate. A couple months ago, I mentioned that I had good luck using different "personas" for reviews as well as for managing agentic loops and I got some responses with theoretical reasons this doesn't work well but, in practice, it seems to work fairly well. My workflow changes regularly and maybe a week after that discussion I started adding "contrarian" personas to the mix, which improved performance given the same wall clock or token budget.

For anything human reviewed, having some kind of artifact (e.g., a video if it's a bug that's expected to be apparent in the UI) is necessary. Without really explicitly trying to have the agent review this, just producing this at all seems to reduce the false positive rate somewhat, and then having the agent review the artifact reduces the false positive rate further. Asking agents to independently review the artifact (e.g., looking at the test code that produces the video vs. looking at the video itself) also reduces the false positive rate further. In general, getting independent perspectives seems to help a lot with reducing false positives. In the experiments I've run, this has been less effective than having agents with different personas / perspective per wall clock time or per dollar, but just asking the same question multiple times improves results, for reasons that should be obvious from the graphs in the LLM variance section.

Pretty much everything I've tried to reduce false positive rate has worked, so if you're not scaling up a workflow to the point where optimizing the costs matters, doing anything remotely reasonable seems to work fairly well.

Caveman mode

I keep getting various tool and workflow recommendations and, when I look into it, I can almost never find good information on whether or not it makes sense to adopt the recommendation. Just for example, I've seen "caveman mode" recommended multiple times at work. Caveman mode allegedly reduces token usage and speeds up prompt resolution (the README claims 75% reduction in token usage, 65% reduction token usage, and 2x fewer tokens used, as well as a 3x speed increase).

Searching for information (just googling 'caveman mode', no quotes, the top hit that wasn't a link to caveman mode was this reddit thread where, of the three top comments, one is joke and the other two highly recommend it:

Just extreme brevity in a refreshing way... and dramatically lowered token count without any seeming impact on the analytical thinking... but i have no way to benchmark before and after.

Someone at work is testing it and it seems to actually save tokens AND work just as well.

Most of the rest of the top hits were also positive recommendations for caveman mode that purported to do some kind of eval (although they read like unfiltered LLM text) and the top hit on YouTube was one of the the biggest programming YouTubers saying

it actually works; it actually works quite well ... no, I'm not exaggerating

In a slack thread at work where people were recommending caveman mode, I asked if anyone had done a comparison, noting that the creator of caveman mode responded to the HN thread about caveman mode by saying it's a joke. Someone linked to an analysis of caveman mode that claims a significant win, but the analysis was an LLM-generated SEO spam article with numerous errors. When I politely pointed this out, the person who posted the link said "I only skimmed it".

At that point, I decided to spend about 15 seconds apiece generating some caveman mode benchmarks (it seems like people call benchmarks evals now, so I should call these evals?) (previously discussed in more detail here).

To start with, I'd been using a lot of GPT-5.5 xhigh when this discussion came up a couple months ago and I benchmarked this thing, so let's look at how this looks for GPT-5.5 xhigh on the first benchmark, a simple benchmark where we ask the agent to optimize some code in wasm. Since this is something I spent 15 seconds prompting an agent to generate, I don't think it's worth spending a ton of time discussing the details, but one thing to note is that it's possible to do much better than any of the results an agent achieved here. I would expect a human doing this by hand or a human who's being prescriptive about what the agent should to do to get much better results than any of the AIs). For the optimization chart, 1.0 is no speedup and higher is better (below 1.0 means the "optimization" slowed things down). To give you an idea of what this looked like when running the experiments, you can click the buttons to follow along interactively or just play an animation.

We can see that, for the first benchmark (optimize a non-trivial algorithm in wasm), after one run, caveman is looking good. We get 1.027 speedup vs. 0.987, $12.10 vs. $23.10, and a wall clock time (of how long the agent took) of 8m51s vs. 14m9s. But we know that LLMs are stochastic, so we should probably run again. After a second run, we can see a big change in the results, with an average of 1.0 speedup for both, but caveman mode coming in at $12.45 in 8m64s vs. $40.38 in 17m57s. That's a huge cost savings that's in line with the claimed savings from caveman mode. It's a bit silly to narrate each step of the animation, but if we skip to the end, we can see that the average after 50 runs is in favor of caveman, with 1.03 vs. 1.01 speedup, and $17.97 in 13m46s vs. $24.21 in 16m52s. That's not as good as what we saw after two points, but that's still solidly in favor of caveman mode.

I asked GPT-5.5 xhigh to do classical and Bayesian statistics on this and it produced a script that says that, for the Optimization 1 benchmark, the p-values for caveman having better speedup, cost, and wall clock time, are 0.1, 0.005, and 0.001, respectively. With Bayesian stats, we have P(caveman better) 0.958, 0.999, and 1.000, respectively. We can look at the plots for the other two benchmarks I spent 15 seconds on as well. Optimization 2 is another "optimize this code in wasm" benchmark, and Game AI is a task where the agent is asked to implement a board game AI for the game Lost Cities with a deadline of 10ms per move.

The results with these benchmarks look mixed. For Optimization 2, we have P(caveman better) = 0.17, 0.999, and 1.000, respectively, and for Game AI, we have P(caveman better) = 0.04, 0.79, and 0.73, respectively, so caveman actually gives worse results for Optimization 2 and Game AI, which is the opposite of what we saw for Optimization 1. BTW, I didn't cherry pick the order of these results to present some kind of surprising narrative reversal. If we were to stop here, we might think that caveman gives worse outcomes but saves money, or maybe it gives better outcomes on some tasks and worse outcomes on some tasks and saves money.

If we try a few more models (GPT-5.4 mini, GPT-5.4, GPT-5.5) at every effort level, we get the following averages, which makes the overall picture less clear (in the graphs below, up and to the left is better, down and to the right is worse; the arrows point from the baseline to caveman):

What are the patterns here? To name a few, for Optimization 1, caveman generally has better results than standard, but for Optimization 2 and Optimization 3, it's mostly the other way around, although there are exceptions. For cost, we can see a variety of patterns as well. There's enough variance between conditions (tasks as well as models and effort levels) that it's clear that we'd have to run a lot more conditions to get a clear picture of what's going on and, overall, the difference averages out to be small enough that it doesn't seem worth using caveman mode.

LLM Variance

Recently, when new models have been released, I've done a search to see what people are saying about them. In general, there are a lot of contradictory comments out there. For example, when GPT-5.5 was released people said, variously, GPT-5.4 is better than 5.5 because it's better at staying on task while 5.5 wanders off and overthinks the problem, making 5.5 much more expensive and pointless; 5.5 is so much better than 5.4 that it's cheaper to use because it doesn't mess up and then get stuck fixing its own issues as much; 5.5 is cheaper than 5.4 because it works so well you can run at a lower effort level; 5.5 "just works" while 5.4 often fails and needs handholding, etc. Often, someone will run a benchmark and show that their statement is true.

Looking at these benchmarks, we can see support for all of these statements that I saw on reddit when searching for comments on GPT-5.5 shortly after release. In Optimization 1, GPT-5.4 has better results than 5.5 and is much cheaper. But in Game AI, GPT-5.5 is substantially better than 5.4, so much so that 5.5 high costs about as much as 5.4 xhigh, but with better results, and 5.5 medium is cheaper than high with significantly better results. With just these three evals, you can find support for every statement I saw people making about GPT-5.5 on release because all of the statements are sometimes true. And that's when we're averaging out variance with quite a few more runs than any reasonable person is going to make to support some comment they're throwing out on the internet.

In general, this kind of thing is why, when I see a metric or graph that summarizes a set of benchmarks, I think, "show me the distribution". Benchmarks of models often reduce to a single, nice, neat number, where you see that X is better than Y, which is better than Z. I find these to be basically meaningless, in that, if we're looking at the latest and greatest from OpenAI and Anthropic, we know there are reasonable benchmarks where X is better than Y and vice versa. If the set of benchmarks had a few more benchmarks that favored Y instead of X, the results would be flipped. For some kind of summary metric like that to be useful to me, it would have to be the case that the set of benchmarks perfectly mirrors the distribution and weight of tasks I do and that I can only choose a single model to use for all tasks. Since neither of those is true, it’s not clear what actionable information I can take away from these benchmarks.

If we look at public benchmarks in more detail, the situation seems worse than it appears from the abstract argument above. Results are generally presented in fairly high precision, as if that's meaningful. For example, on one benchmark, we might see that (for example) GPT-5.5-xhigh is 1% better than Fable 5 medium, but at 19% lower cost. And then if we compare to Opus 4.8 maybe it's 13% worse than GPT-5.5 xhigh at 11% higher cost. If we want to know what this means, we can dig into the data and see that we have some benchmark that claims to be meaningful because it has this big set of diverse tasks, but they're all pass/fail tasks that get run 4 times and most tasks are either very easy and get 4/4 with the best models (except, due to some random noise, you sometimes see a random 3/4) or are very hard and mostly get 0/4. Then there's some small subset of tasks that actually determine the relative scores of these SOTA models. If you change out one of these for a different one, the results between the two highest scoring models can get flipped. If you change a few tasks (out of about 100), then you can see the apparently much worse Opus 4.8 move ahead of GPT-5.5. Change a few more tasks and GLM-5.2 can pull ahead.

When I see things like this, it reminds me of Miguel Indurain, who was enough of a household name when I was a kid that I'd heard of him even though I don't follow cycling. A few years ago, I was curious why household names in cycling since Indurain are all different archetypes from Indurain and it turns out the answer is that it's arbitrary. For arbitrary reasons, the Tour de France has become the most famous cycling race in the world and someone who has a dominant streak can become famous enough that they become known outside of cycling circles. For other arbitrary reasons, there was a period of time where the TdF had much longer time trial stages than it does now, which suits someone of Indurain's archetype. You tweak the benchmark a bit and Miguel Indurain goes from being a once household name to an all-time great time trialist that pretty much nobody has heard of unless they follow cycling.

Back on the topic of coding agents, it's not clear who really needs to pay attention to these benchmarks that present summary metrics of how models are doing. As we noted above, as a user of models, these benchmarks don't meaningfully tell me which model I should use.

If many other users based their decisions on these benchmarks, then AI labs would need to care about their results on these benchmarks. But even though GPT-5.5 has been handily beating the various Opus 4.x models during 5.5's tenure on most of these benchmarks, Anthropic's business grew much faster than OpenAI's during the time period that the best publicly available models were GPT-5.5 and Opus 4.6/4.7/4.8, so much so that OpenAI has been giving companies free tokens to try to convince people to use GPT. My company was one of many to get months of free tokens and, during that time period, most people still primarily used Claude and Opus. Anthropic's revenue trajectory is incompatible with these benchmarks being major determinants of user choice, so I don't know why anyone should really care what these summary metrics show.

The last set of graphs we looked at shows how much variance we see across tasks, but from the prior set of graphs, we also saw a lot of variance within tasks with the same model and effort level. If we look at a small number of individual runs, then pretty much any conclusion is possible due to the variance between runs. Just for example if we look at Optimization 1, for GPT-5.5 xhigh, one standard deviation between runs is 0.075 (i.e., 7.5% performance increase / decrease). If we look at the average difference between the best and worst tested GPT, that's 1.055 (GPT-5.4 xhigh caveman) and 0.986 (GPT 5.4 mini low), which is less than 1 standard deviation (SD) across GPT-5.5 xhigh. For the actual graphs and not just summary statistics, we have:

For every task, whatever the best condition is, it's easy to get a result where the best condition actually scores worse than a result from the worst condition. When I was chatting to Max Bittker, who's done a lot more benchmarking than I have because he runs an RL environment startup said:

Yeah - the level of noise from task to task and run to run is so high that it's no wonder the discourse ends up confused. Easy to make mistakes like "Wow, $new_model is amazing" -> "Oops, I was still using the old model the whole time", or "this new harness / prompting trick works great!"

But on the other hand, benchmarking has given me confidence in statements like "Opus models recover and debug confusing conditions much better than Sonnet models" and "Chinese models score well on SWE-Bench but underperform on novel tasks" with more statistical significance.

Another comment, this time loosely paraphrased because this is from memory, was that Matt Mullenweg said that, if you look at people who undertake high variance activities, like gamblers, they're often superstitious. You'll see somebody wear their lucky socks or have a specific routine they do before they sit down to play the slots. Using caveman mode or deciding which model is good because a coding agent coughed up a good result after trying it is not so different.

I did the above and wrote the draft of this post before Fable was released, and then Fable was released, so just out of curiosity, I ran a few Fable benchmarks, at which point it was unreleased, so I also did a few runs with Opus 4.8.

Like I mentioned above, these are little benchmarks I created by typing to an agent for about 15 seconds a piece. With how coding agents let you scale processes, you could create a very large number of reasonable benchmarks in 15 seconds of a human time a piece, but this isn't some carefully designed setup for stamping out a bunch of these; it's just something I did because I wanted to get some kind of caveman mode comparison, and when I tried to have agents do this with no supervision, the benchmarks were completely worthless. With a little bit of time spent, the benchmarks seemed good enough for the purposes of evaluating caveman mode and determining that it's not worth further investigation. If I was planning on writing this up publicly, I would've spent a few more minutes making these benchmarks better in multiple ways, but I think they're good enough for the purposes of this variance discussion. I wouldn't read too much into the reuslts, but I did do a few basic checks such as looking at outlier results to make sure agents weren't cheating or there wasn't some issue (such as noise on the box) causing unusually poor runtime results, and the agents weren't cheating and the slowest results did reproduce on a quiet box, etc. And, that disclaimer aside, just like with the GPT benchmarks, we can see variance in these benchmarks that reflects the discussion that's happening online, e.g., Fable is great at some tasks and not so great at other tasks, and quite a few people are making claims about Fable's overall performance based on the performance on some small set of things they tried.

Above, we discussed why it's not clear how I or any particular user of agents can usefully use these benchmarks that produce a ranking or a small tuple of numbers that allegedly tell you how good a model is. But even when drilling down into a benchmark, it's still not clear to me why, as a user, how I should change my behavior as a result of the benchmarks.

Just for example, something I've observed is that Opus 4.8 makes up bad rationalizations to explain things much more often than GPT-5.5. I've tried having them approach the same problem a number of times (I'll often ask both to solve a real problem I'm running into just to see what happens). These are often much larger problems than these benchmark problems, things like real debugging tasks or, in the case of building a game AI, instead of a simple prompt to produce an AI, an agentic loop and/or a series of instructions to get the agent to produce a fairly strong AI. I've asked a small handful of people and they've all observed the same thing with GPT-5.x vs. Opus 4.6/4.7/4.8, including people who prefer Claude and primarily use Claude and are more experienced with Opus than with GPT.

On the flip side, I've seen two benchmarks that measure how good models are at detecting false information and/or not making things up and those benchmarks both show that Opus is much better than GPT at this, to the point where, if the benchmarks show what one might expect them to show, the experiences I and other people have had seem impossible. For example, in one benchmark, a naive reading of the benchmark presentation is that Claude Opus and Sonnet are much better at detecting false information than any other model and Opus 4.8 is the absolute best and detects false information 95% of the time, including 95% of the time in software. The GPTs are terrible at this and rank below Qwen, Grok, Kimi, Minimax, Mimi, and Nemotron.

What's going on here? I'm not sure.

Maybe those benchmarks are measuring a different aspect of making things up than what the people I've talked to (and I) run into while working on real programming problems, or maybe this is a small sample size and the people I've talked to (and I) have just randomly gotten bad rolls of the dice from Opus, or maybe we're "holding it wrong" when it comes to using Opus. I suspect what's happening is that the benchmarks are measuring something different from what happens when you encounter incorrect rationalizations while coding or debugging, but to have any confidence in this I'd have to run some experiments that will be fairly expensive. If you want to support my doing experiments in general, you can subscribe to my Patreon or, if you're at an AI lab, give me some free credits to run experiments with.

BTW, there's something analogous in the Game AI benchmark here, where Opus is substantially better than GPT, but when I tried to have to manage the process of creating an actually strong (superhuman) game AI, GPT seemed to do better because Opus kept falling into making things up / rationalizing nonsense failure mode. With no supervision, both were worthless and had their own failure modes, but when directed, GPT took less prodding to stay on track and keep doing things that could work. This is not reflected at all in the small Game AI benchmark where an xhigh run is a single prompt that "only" costs on the order of $10 and the small benchmark turns up the oppposite result as we see in doing the real version of the task.

But, the point here is that, even when you find benchmarks that measure something that seems to be the exact thing you care about, those benchmarks often don't end up matching what you see in practice. If you're a weirdo like me maybe you'll decide to spend a bunch of time and/or money running some experiments to figure out what's going on but, if you have some job you want to get done (maybe for your actual job), that's a fairly unreasonable thing to do.

Benchmarking and data analysis

For no particular reason, I've always liked designing experiments and measuring things. This was true long before I ever thought about careers and it's still true today. As discussed here, measurement is one of the primary themes of this blog, maybe the primary theme. Lucky for me, this lifelong hobby has been something I've been able to make a career out of. And even luckier, this skill seems to have been made relatively more valuable by coding agents.

As we discussed above, testing is a rate-limiting factor in highly agentic workflows because when you let agents start doing a lot of things, anything that's not well tested gets stochastically degraded. The more you ship with agents, the worse this gets. If everything you do is like one of these model evals where every task is a pass/fail task and there's no way to really do better than simply not getting the task wrong, then you can make sure your tests are good enough and let agents go to town until tests pass and everything will be great. In practice, for almost all non-contrived problems, it's possible to have a solution that passes a strict correctness eval but is still better or worse in some kind of non-strict correctness sense. If you want to let agents go wild on a problem (whether this is fast and loose vibe coding or running autonomous agentic loops), now you have a benchmarking problem.

When left to their own devices, running in self-improving loops, agents seem fairly bad at this. There are various little things I've been doing for a few months that get some improvement here (having lots of agents think about things independently and reconvene over multiple iterations with a mix of contrarian agents, etc.), but they still don't seem great at this without human intervention.

One reason for this seems to be that agents are really bad at understanding data and doing data analysis. Doing this in the context of some kind of agentic loop or large problem is harder than doing a standalone data analysis, but when people just have an agent do a standalone data analysis, the results are generally terrible without a lot of guidance. Because it's so easy to have an agent do data analysis, I've seen quite a few agent led analyses and, at least so far, every one that I've seen is completely bogus. I've also done a fair number myself (I think people really underrate the value of getting completely bogus output from an LLM; more on this shortly) and found the same thing.

When I say complete bogus, I mean things like finding different numbers that aren't really related and somehow inferring something deep about their relationship, picking two numbers or examples out of many and coming up with a theory that contradicts other numbers in the data, making plots that show something meaningless (but often look pretty), etc. For a random concrete example, the last time I looked at an analysis produced by an agent (I glanced at this as I was writing this), someone had asked an agent to analyze the resource utilization of something. This was Opus 4.8 on max (Fable was disabled, so this was the best thing money could buy from Anthropic at the time) agent determined that 514% of the resources were being consumed by some task where it was impossible that more than 100% of the resources could possibly be consumed in any way. I have a colleague who sometimes comments on these things and will ask innocuous questions like "what does X mean?". Generally, he either gets no response or what appears to be some kind of AI-written response that's as incorrect as the first thing he replied to.

Anyway, when I started using agents for data analysis, maybe in November 2025, I found the speedup to be pretty incredible. There are analyses that, roughly speaking, would've taken weeks that instead take hours.

I hesitate to even describe this next thing I've been doing because of how many completely bogus data analyses I'm already seeing, but something I've been doing a lot of, which I find to be a much larger speedup than doing a "normal vibe coded data analysis" is to run a simple agentic loop where the agent (or agents) "understand" and analyze the problem and then fix up the parts that need to be fixed for me to extract a good analysis from what the agent has produced.

How much this bogus LLM loop speeds things up depends on the problem, but the last time I tried it, I think a traditional pre-LLM analysis of the issue would've taken me some number of days. Let's say two days. With a workflow where I'm multitasking and poking the LLM when it needs poking, my guess is that it would've taken one to three hours of my time over the course of a day or two. With this "have the LLM loop on producing a result I know will be wrong" workflow, it took about five minutes of my time over the course of a few days to get an acceptable result. Literally every time I looked at any part of the analysis I hadn't previously corrected, the analysis was wrong (and, in some cases, I had to issue multiple corrections) but, somehow, the whole thing still moved a lot faster than if I tried to steer the LLM.

I think people are really underestimating the value of getting completely incorrect results out of an LLM. This is, non-ironically, a game changer in a positive way, when directed correctly.

There's something a bit odd about how incredibly bad SOTA models are at data analysis and how much they speed up human data analysis. I have this series of exercises in benchmarking, evals, and experimental design (part 1, part 2, part 3, part 4, part 5, part 6). When GPT-o3 was released, Tyler Cowen said that o3

wipes the floor with the humans, pretty much across the board ... I don’t mind if you don’t want to call it AGI. And no it doesn’t get everything right, and there are some ways to trick it, typically with quite simple (for humans) questions. But let’s not fool ourselves about what is going on here. On a vast array of topics and methods, it wipes the floor with the humans. It is time to just fess up and admit that.

This prompted me to try asking the questions from these exercises to SOTA models. They generally underperform what I'd expect from a reasonable junior colleague unless you specifically word the question so that models can answer the questions. If you just ask the question the way you'd ask the question to a reasonable human colleague in real life, they generally don't do well. This isn't just a theoretical problem that comes up when someone asks an evals homework question. This issue comes up any time you ask an agent to do some kind of empirical, open ended, longer horizon improvement task.

I've seen this issue come up on a wide variety of real world problems, but it even comes up on contrived problems that don't have almost any of the messiness of real world problems. An example of this that I think is a nice problem is building board game AIs. This is a much easier problem to evaluate than most real-world problems because, at the end of the day, you want to beat humans and other AIs on a well-defined game that has a simple win/loss/draw result.

Agents can't really figure out how to do this when left to their own devices. To be fair to AIs, this problem is somewhat harder than internet commenters give it credit for. For example, if you just search for information on how to implement these things, you see all kinds of exchanges like this or this where someone who hasn't done it will explain why it's easy and someone who has will tell them it's harder than they think. Most of the online tutorials aren't very useful, to the point that the author of the well-regarded bulle library suggests that beginners avoid looking at "[a]lmost any article/blogpost/book on [a board game AI technique] that isn't backed by the author's strong [board game AI]". While the problem isn't exactly hard, it seems to have enough pitfalls to trip up most people (and all publicly avialable LLMs).

When I first started trying to implement a board game AI (this was in the GPT-5.1 to 5.2 days), I didn't know anything about board game AIs and tried having coding agents implement what they thought would work. Now that I know a bit about board game AIs, I can say that every direction GPT tried to suggest was bad and couldn't work. As an experiment, I tried this again recently, when GPT-5.5 and Opus 4.8 were the best publicly available models and they failed in the exact same way and suggested many of the exact same bad ideas (just to really make sure, I let them implement the bad ideas and they failed as expected).

I think, at some level, this is a fairly well understood gap in LLMs today. Coincidentally, around the time I was trying GPT-5.1 or 5.2 at this, the Code Clash eval was created, which also tests the same thing and found the same result

Unable to Iterate: Models struggle to improve over rounds, exhibiting a variety of failure modes.

Despite agents being terrible at setting direction and me not knowing anything about board game AIs, I was somehow able to cobble together a superhuman AI for Azul in about 5 hours of my time and get it to a crushingly strong level that's well above any other AI that was out there at the time (and I think still out there now) in about 20 hours of my time. The Azul world champion and probably strongest player in the world overall said:

the bot is quite insane now. def seems stronger than me, though to really test it i would have to play it like a turn-based game. with limited time on my end, i stand no chance 🙂.

As much as I'd like to be able to say there was "one weird trick", there were really two tricks, but two tricks isn't so bad, especially when they generalize to other kinds of projects as well. The two tricks were:

• Look at the data / have some reasonable-ish evals
• Solve problems systematically

None of this has to be very rigorous. In fact, I'd say this was all running on vibes (in the pre-vibe coding sense of the word). For the data side, for this project, I used a common habit for me, plotting a bunch of things that seemed relevant, eyeballing them, and then nudging things in a direction that I would hope improves things. A more rigorous (and possibly better) approach is to run a bunch of experiments from scratch, do "ablation" runs where you add or remove individual ideas, etc., and understand what each component does and what the impact is. But I wanted to get a strong AI while using as little of my time as possible and just training on my laptop and doing things rigorously seemed too expensive given those constraints.

I'm sure all of this would've been trivial for someone with ML experience, but there were a bunch of little things I had to observe and then find ways to deal with (which is something agents are currently terrible at). For example, in terms of figuring out a good set of evals to look at it, there are a few funny things. One is that you can easily make changes that reduce loss but don't change the win rate against actual opponents (humans or other bots) and vice versa. Another is that, if you try to track improvement in terms of how well your bot beats previous versions of itself, it's very easy to make a series of changes that allows the bot to beat all previous versions of itself and have an apparent Elo gain of 1000 or 2000 against prior versions that isn't any better against humans or bots that play in a different style. And, at the time, there was no superhuman bot that was available to play against, so you couldn't get a reasonable eval by just playing against a field of existing bots.

For a human, these aren't really hard problems. A reasonable human can look at one of these issues, think for a few minutes, and come up with a proposed solution that will probably work (and when it doesn't work, they can try another solution). But there's enough subtlety that agents don't do well at this today without supervision (to be fair to coding agents, many or most humans don't either—almost all of the bots I found to play against were fairly bad, despite people seeming to have spent a decent amount of time on a lot of them, with the one publicly available exception being the "PJF98" Azul bot in https://github.com/cestpasphoto/alpha-zero-general). There was one bot stronger than the PJF98 bot in existence, but it wasn't made generally available to play against and was just used to win one season on BGA (which was voided because using a bot is cheating) and then never seen again.

The other trick I mentioned was taking a systematic approach. At a high level, this is because when dealing with an opaque system with some complexity, every time you see a symptom of a bug, there's a good chance this is a window into one or more other bugs that you don't know how to observe. If you non-systematically close the window without fixing the other bugs, those other bugs are still there; you just don't know how to find them anymore.

This is specifically the case when building a board game AI that uses a neural net because you have this net that's doing who-knows-what and some kind of search function that's doing who-knows-what and you can sometimes spot specific obviously bad moves. This is also generally the case with vibe coding and/or agentic loops because you have agents doing who-knows-what and you sometimes get a window into what's going wrong as the giant, overly complex, pile of code the agents produce will emit some kind of obviously bad behavior.

Specifically in the case of board games, I looked at the other Azul AIs that are out there. A lot of people had built bots, especially recently with coding agents making it so easy to build a bot. For some of those, you can actually see how the person tried to nudge the bot (for example, because they included the instructions they gave to Claude in their repo) and, AFAICT, the main reason these other bots were much worse despite more human time spent on the effort was a combination of using the wrong evals and fixing the symptom of a problem and not the cause.

For example, in one case where the author left the Claude instructions in the repo so you can see what they did, at one point, their bot was making a very bad move that's worse than a move that any human would play even if they'd just learned the game. The author instructed Claude to make a series of straightforward fixes that would stop the bot from making that move, which it did, but this didn't fix all of the other bad moves that fall out of the systemic problems the bot has; it just fixed the one very obvious symptom of a bad move that the author could see.

A vaguely analogous issue my bot had is that, at one point, my bot would almost always try to open in column 2. A simple fix that I think would've worked would be to tweak the bot to play in column 2 less often. I tried the various "obvious" fixes like increasing noise to increase exploration in self play, etc., but none of the obvious fixes worked other than directly reducing the value of column 2, but that has this other issue we discussed of not really solving the problem. In that specific case, a more systematic fix that worked and made that bot's play a lot more generally robust was to, at various points in the game, fork the game into a new game with different column permutations. The idea here is, the problem is that, on average, column 2 is the best column, so even if the bot opens in a different column, it will have this tendency to go back to the average best column due to the games it's seen in the past. If you're DeepMind, you can throw a bunch of money at this problem and have the bot play a lot more games and learn something else. But if you're running on a laptop, that could take a while and if you're trying to tweak the weight at which the bot should be nudged away from column 2, there's no way you're going to get a reasonably optimal set of weights to do this with. But if you (for example) sometimes, near the endgame, permute column 2 to column 4, the bot will learn that if you have a nearly filled column 4, it's a winning move to complete it, which will help the bot learn that earlier in the game, if you have a partially complete column 4, it makes sense to advance it to near completion, and so on and so forth, until, in the opening, the bot learns that in the opening, it should then open with a reasonable distribution.

For someone like me with no ML/AI background, getting the bot to superhuman performance was a series of puzzles like this that are a combination of figuring out what the right view of the data is to see problems and then tweaking this systematically to solve the problem. This is not so different from managing agentic loops for general problems. Michael Malis has said that running a "software factories" workflow feels like playing Factorio and I can see what he means by that.

Misc

I'm not sure how much value there is in talking about specific workflow tips and tricks since the half-life of these is so short. Before I was running more autonomous loops and was doing more human-in-the loop work, "my" agents would often do something that's in direct contradiction of AGENTS.md or other instructions. When I offhandedly mentioned this to Yossi Kreinin, he suggested adding a note at the bottom of my AGENTS.md to re-read instruction after compaction. For all I know, that's a superstitious act that did nothing, but it seemed to reduce the rate of this issue from a few times a day (when juggling a handful of agents at once, so maybe once every few agent-days) to once every week or two (maybe once every 100 agent-days or so). It seems like this isn't necessary anymore, so either this was a placebo that never did anything in the first place and I was just getting unlucky for a while, or the AI labs have added this as well. A trick we discussed in August 2024, back when LLMs would produce code but not run it by default, was to just have a loop that runs the code to make sure that it works and passes tests. That trick had a surprisingly long useful life considering how obvious and useful the trick was, but of course the big AI labs eventually realized how useful that was and they all have tools that do this now.

For the past few months, the "trick" I've been using is to have something that's vaguely in the same space as something like Gas Town, but with various little things to try to make it more reliable. It seems like Claude's Dynamic Workflows put half of these tricks into an easy to use package (I haven't used it since I'm told that it uses a waterfall model where things from one stage wait on the last one, but either that will be fixed at some point or someone will release something without that limitation) and Codex has some advertised but yet to be released features that seem targeted at doing the same thing. In principle, goal mode should package up most of the other half of the tricks, although it doesn't work nearly as well for now (besides sometimes failing due to a bug, it seems less good at pursuing the goal than a manually written loop, and it can also really get off track; Dennis Snell mentioned letting what he expected to be small task go in goal mode and finding that, according to codex, it spent what would've been $60M in tokens in a few days had his employer not been on an unlimited token deal). Presumably these things will improve and most of what I'm doing now will be available off-the-shelf within a couple of months? Like the tricks mentioned above, these are all things that are so obvious enough I don't know that there's much value in spending a lot of time talking about the details, but I'll put some brief comments in a later appendix in case anyone is curious.

The meta techniques discussed above have generalized fairly well for me on a variety of projects and they seem like they should be durable as long as agents can't just take a non-trivial problem description and fully solve the problem better than I can, at which point I won't have to worry about it because my economic value may be close to zero.

Another meta idea that's been useful is the very obvious thought that coding agents are highly non-uniform in their effectiveness relative to humans. When discussing about traditional productivity, Fabian Giesen made this comment on how velocity improvements change how he works:

There are "phase changes" as you cross certain thresholds (details depend on the problem to some extent) where your entire way of working changes. ... ​​There's a lot of things I could in theory do at any speed but in practice cannot, because as iteration time increases it first becomes so frustrating that I can't do it for long and eventually it takes so long that it literally drops out of my short-term memory, so I need to keep notes or otherwise organize it or I can't do it at all. Certainly if I can do an experiment in an interactive UI by dragging on a slider and see the result in a fraction of a second, at that point it's very "no filter", if you want to try something you just do it. Once you're at iteration times in the low seconds (say a compile-link cycle with a statically compiled lang) you don't just try stuff anymore, you also spend time thinking about whether it's gonna tell you anything because it takes long enough that you'd rather not waste a run. Once you get into several-minute or multi-hour iteration times there's a lot of planning to not waste runs, and context switching because you do other stuff while you wait, and note-taking/bookkeeping; also at this level mistakes are both more expensive (because a wasted run wastes more time) and more common (because your attention is so divided). As you scale that up even more you might now take significant resources for a noticeable amount of time and need to get that approved and budgeted, which takes its own meetings etc. There's something analogous about agentic coding velocity improvements. People sometimes make claims about how agentic coding is 100x or 1000x more productive. When I look at tasks I do, it's hard to really pin down a number because you can't effectively lift-and-shift a human workflow to agents, so I'm doing something very different from what I would've done in the first place. For example, we look at the idea discussed above of having agents look at every support ticket to convert support issues into PRs. If I were to claim some kind of speedup here, it would be a huge number, easily more than 1000000x. But of course I wouldn't read every support ticket myself, so such a claim is meaningless. A more reasonable way to estimate this would be to try to figure out what kind of traditional organizational structure would generate that many bug fixes using classical techniques and estimate the ratio, but the error bars on an estimate like that will be more than one order of magnitude and it's also not something that would've been done at any normal software company, so the comparison is once again meaningless. It's like someone who drives 20000 miles a year saying they saved 6000 hours and got a 15x speedup on their commute by driving because that's how long it would've taken to walk the same distance. It's clear that the driving enabled them to do things they wouldn't have been able to otherwise do, but there's no point in discussing the ratio.

Another example of a place where you can claim a massive speedup number, possibly more than 1000000x, that's a meaningless number is, when trying to get an LLM to generate a fuzzer in a semi-automated way, I've tried having an LLM look at the entire commit history and all bug fixes, as well as every plausibly related support ticket in history, to try to get the LLM-written fuzzer to be able to reproduce the bugs in some general way. This seems to have been useful to do, but it's hard to put a real number on the value.

For me, the effective value of LLMs is that there are a lot of tasks that would've been too time consuming to reasonably do that can now be done fairly trivially. In a footnote, I mentioned that there are lots of apps you can just trivially build now, so I sometimes play online board games on a custom app that's just a nicer experience than I can get on any of the major board game platforms (which is also nicer than the typical dedicated app for a board game).

In things that I mostly do to satisfy my curiosity, but that can also have some relevance to a business, I'll also do data analyses that didn't really seem worth the time before. For example, I've long been curious about the ratio of user-experienced bugs to support tickets. With coding agents, it took minutes of my time to get agents to produce a list of major incidents where we know that every user that tried to access the site or use a feature couldn't use the feature or site and to get a list of tickets that appeared to be associated with each incident. On joining this data, a typical ratio was 200 impacted users per ticket, with the ratio often being between 100:1 and 1000:1 (and, I'm sure, a wider range would've been found if more incidents were included). FWIW, I pre-registered a guess of 100:1 to 1000:1, leaning more towards the 1000:1, so I was somewhat off here as the typical 200:1 is closer to 100:1 than it is to 1000:1.

Of course, there are a lot of holes in this analysis. There's no way to tell for sure that there aren't missing tickets (I did check for false negatives by independently cross-checking with tickets that were manually tagged in the bug tracker as being associated with an incident and the main thing that turned up was that agents were able to find many more tickets than humans found, but that doesn't mean that that aren't false negatives the eluded both humans and agents). I also randomly sampled a handful of tickets to see if the associated tickets were reasonable and they seemed reasonable, but this was just a quick order-of-magnitude analysis, so I didn't manually inspect as many tickets as would be necessary to really validate a high precision analysis.

It's also not clear how the ratio changes for less severe issues. It should generally be the case that, for less severe issues, the ratio is higher, and also for less invested users (such as the typical user in the sign-up flow). My guess is that, for many less severe issues, we generally see ratios well above 1000:1, with the ratio being easily above 10000:1 or even above 100000:1 for many subtle issues.

This analysis doesn't let me say anything too precise, but it does work as a response to when someone says "only six users were impacted" because an issue had six tickets. That typically means that there's an internal bug tracker ticket for which someone attached six support tickets. On using an agent to search for related tickets, you might find twenty-five tickets and then, looking at the issue severity, you might estimate a ratio of 10000:1, so "only six users were impacted" may turn into "it's overwhelmingly likely that at least 5000 users were impacted and plausible to likely that 25000 users were impacted".

If I was already familiar with every system I'd need to query for this analysis, I suspect getting a rough guess for this would've taken hours and it would've been a rougher guess. But given that I wasn't familiar, I would guess that this would've taken a couple of days. Instead, this took maybe 15 minutes of my time. Like Fabian mentioned, this kind of speedup where an app or analysis that would've taken hours or days can be done is a phase change that really changes how you work, and that's not even including autonomous loops that only need occasional maintenance.

Another thought, and it wasn't obvious to me in advance that this would be the case, is that LLMs seem to be a larger productivity multiplier for people who are relatively more expert at the thing they're trying to do. On looking around for commentary on this, I of course got pages full of LLM generated SEO spam and, after wading through it, I saw quite a few comments indicating the opposite. Basically, that anyone can do anything now. While that's more true than it used to be, it seems to be even more true that expertise in an area has become more valuable. Max Bittker had a comment that this is at least partially because LLMs are good at counterfeiting things, but that counterfeits aren't that good (yet?), and experts can tell the difference between a counterfeit and the real thing, which I thought was an interesting framing.

And following up on this 2015 post about how many people are underestimating how much AI will displace humans, in the past year, my median LLM-driven remote customer service interaction has been better than my median human remote customer service interaction (a well-paid human would provide a better experience than all but maybe one of my LLM-driven experiences, but companies don't want well-paid humans). The exact same line of reasoning that was rebutted in that post 2015 was repeated in 2022 on the release of ChatGPT. The line of reasoning is so popular that it's been a repeat thought leader viral hit every year since 2022 and I don't see that changing any time soon, but it still seems wrong to me today.

By the way, as we previously discussed, there are lots of ways to be an effective programmer, so I'm not saying what's discussed here is the best way to do things or even a very good way (I'm changing my workflow regularly as I figure out new things) but this is what I've been trying that seems to work better than a purely traditional workflow for me. Just given my background and interests, it's natural for me to take a systematic, evals-driven, approach to agentic coding. I've seen people use what is pretty much the opposite approach and make it work for them as well, where they move very quickly without almost any understanding of what's happening and, when they strike gold, they recursively put all of their effort into that. Not only is that not a great fit for my background, I don't think it would really work well for most of the problems I've worked on, which I select because they seem like a reasonable fit for how I approach problems. No doubt my workflow, ported to the problems the opposite kind of workflow is suited to, would also not work very well.

Appendix: people talking past each other

It's always been the case that people talk past each other when they disagree, but this seems to be more of an issue when talking about agentic coding today than it has been for a lot of other topics.

Some of the major reasons I've seen for this are:

• General incredulity
• Workflow-based reliability differences
• General workflow issues, i.e., "skill issue"
• Differences in expectations

On general incredulity, moreso than at any other time in my life, I see people saying that X is impossible, nobody does X, and anyone who claims that they're doing X is a liar, when I know people (who I trust and are credible) doing X or I'm doing X. There's a lot of incredulity out there about AI. Eleven years ago, we looked at how people were saying AI can't possibly replace or displace humans even as it was already happening, but the kind of displacement there wasn't a serious fear for a typical middle-class person or programmer, so it wasn't as salient for folks I might run into "on the street" as it is today. Now that this is more "in your face" for people I'm running into, the amount of denying reality has shot up.

BTW, I find it quite reasonable that a lot of people are incredulous. For one thing, a lot of the positive claims about AI are incorrect. On average, wild, incorrect, claims get more traction than boringly precise and correct claims, so someone who isn't following this stuff is going to see a lot of incorrect claims that are easy to dismiss. It doesn't help that a lot of the people making these claims have a direct financial interest in the general success of AI companies, which makes it easy for skeptics to conclude that these people are all self-interested liars.

On workflow-based reliability differences, what I mean is that different workflows demand different levels of reliability and people who use a workflow that demands a higher level of reliability will often claim AI is useless because it can't do X reliability, missing that there are many effective uses that don't require that level of reliability. To take an extreme example, some people have done novel mathematical research and solved open problems using AI. Let's say, hypothetically, AI solves some open problem you're working on 1 out of every 100 times you try it and returns completely gibberish 99 out of 100 times. That seems great. Solving a serious open problem in math research 1 in 100 times is a pretty awesome result, even if doing a reasonable code review 1 in 100 times and producing a bad code review 99 out of 100 times would be a bad result.

For a less extreme example, for programming work, if you don't assume that the agent is 100% reliable and build systems to handle this, you can tolerate a much lower level of reliability than somebody who assumes the agent is reliable and needs some kind of human-in-the-loop correction to deal with cases where the agent isn't reliable. I regularly see people give advice saying, "don't use AI for X [because it won't work 100% of the time]" where I know someone who does X all the time and have some boring system for handling X not being 100% reliable. This is something you commonly have to do in programming regardless of whether or not you're dealing with agents, so why not use the same techniques you'd use to handle agents being unreliable?

Differences in scale also cause people to talk past each other as people have very different reliability demands at different scales. There's an old rule of thumb that, for traditional software systems, for every order of magnitude increase in scale, you need a different architecture. That's not really true, but it gets at this directionally correct idea that you really want different architectures at different scales. This is also true for using agents.

Just for example, one that I've seen come up repeatedly related to the earlier note that, on tasks that I do, adding a clear, strict, instruction in AGENTS.md to not do something fails once every week or two when juggling 10 agents, a failure rate of something like one per 100 rule-agent-days. Does this work? If you're doing human-in-the-loop coding with a relatively small number of agents and rule (say, 10 agents, 10 rules) with failures being non-catastrophic things that are usually caught by human review, this works fine. If you're running hundreds or thousands of agents or if you're shipping without review or other guardrails even with only a few agents and failures can cause serious problems, this doesn't work at all. While some techniques are scale invariant, many aren't.

Another kind of workflow issue is when people have a workflow that can't work for X and then conclude that nobody can do X because they weren't able to do X. I haven't thought too much about the specifics of what doesn't work, but Max Bittker had the following comments:

A few patterns I've noticed, not in order:

1. People sometimes have misconfigured environments: They're on an old model, they have a bunch of MCPs or after-market system prompts turned on, or the LLM is trying to deal with Windows or some unusual environment or IDE.

2. People sometimes trash their context window by using a single super long chat with lots of backtracking or single super-messy working directory.

3. Injecting accidental requirements via imprecise language, sending the LLM off on a harder task than they intended like building a matchmaking system before the core of the game

4. If you don't have a strong idea what you want or when you're done, you might under-specify and then collaboratively drift and change requirements over time and going nowhere in particular (many AI-psychosis vibe coders do this)

Appendix: agentic loops and writing this post

As I noted in the alt text at the start of this post, I was very hesitant to write about AI for years because I'm up here in Galapagos Island, highly disconnected from what's happening because I haven't been reading social media much and I've also been doing a bad job at keeping in touch with folks who are really in the thick of it in SF.

I was convinced that it might make sense to write something in May, when I chatted with a couple folks from SF who were in town for PGConf. Even though they are in the thick of it and seem to be close to the cutting edge of what people are doing with agents, there were still things from my workflow that they thought were interesting. But, like I said, the half-life of particular workflow tips and tricks is short enough that I didn't think it would be useful to write up detailed notes on an exact workflow, which is why the main post has focused more on higher level ideas. I'm not sure what might actually be useful, but I'm going to try writing up something about the evolution of my workflow. As noted above, I don't think there was any point in time where my workflow was great, exactly, but it was useful and there always seemed to be a natural next step to make it a bit more productive.

Back in the stone ages, before tools like Claude and Codex, I would sometimes just run a very simple loop that would just keep compiling the code and running the tests and re-prompting until everything passes, which I mentioned here in mid-2024. At the time, I didn't find this useful enough to use for anything where I knew what I was doing, but it enabled me to embed a little web game into that post and do other tasks that would've required me to learn something about an area where having actual expertise will probably never be particularly interesting to me, such as building a web app.

That kind of thing was mildly useful for quite a while when I wanted to accomplish some simple task, but I didn't start using agents much until running into the bisect/video story at the top of this post, and doing some other data analysis and seeing how much it could speed up data analysis. BTW, the bisect/video story came out of an analysis where I was wondering what the half-life of a working feature and/or bug fix is. Coincidentally, that analysis is linked to a couple of ideas in the post. First, the half-life turned out to be fairly short pre-LLM, which seemed like a point in favor of more testing if we wanted to turn the dial up on velocity (with or without LLMs). And, second, this is yet another analysis that would've taken long enough that I wouldn't have done it pre-LLM.

At some point mid-late 2025, I tried vibe coding some personal projects. I used Codex just because people said it gives you more quota on the subscription plans (I've since tried both and that seems to be accurate). At that point, my goal was to use as little of my time as possible on personal projects while getting the output I wanted, but I hadn't really done any vibe coding before and didn't have any idea what would work, but it quickly became apparent that I did the most naive possible loop, where I would just queue up a bunch of copies of "if X isn't done, implement X until complete; if X is done, ...". I wasn't reading anything about what other people were doing and the people I was personally talking to generally weren't using AI much, so I had no idea that the term "Ralph loop" had been coined in mid-2025 and people were doing various, less cumbersome, variants of this, although this wasn't really all that cumbersome (maybe 5 seconds or so to queue up the copies of the prompt once they're written).

And, in retrospect, this workflow wasn't so bad in terms of steering things in the right direction compared to a Ralph loop. I'd usually look at this a handful of times every day (right before bed, on waking up, maybe a couple other times) and, in general, every time I looked, things would not quite be right and I'd queue up some commands to nudge things in the right direction after looking at the logs/metrics/graphs/etc. that explained what was going on. Since this was for personal projects I was spending a small number of minutes a day on, I wasn't trying to maximize throughput and was just running a few things on each of two different laptops I have lying around. It would've been better to have more autonomous loops that can keep going indefinitely without refilling the queue by hand, but the actual time savings from that isn't really all that much, I think less than 10 minutes per day.

After doing that for a while, I started (sometimes) trying to run more autonomous loops earlier this year. Once again, I think this must've been somewhat behind the curve compared to whatever people are excited about at any given time, but it seemed to work ok. On starting to do that, I found that a lot of the tooling people were recommending wasn't really a great fit for managing agentic loops and the tooling that was designed for it seemed a bit "vibe"-y and would have reliability issues.

An example of the first thing is that Conductor was widely recommended at the time and it wasn't really made to support a lot of things I'd want to do. For example, Conductor has the concept of a workspace to separate things. I often want to invoke what one might call "higher order workspaces", where an agent (or a set of independent agents) decide what should be done with a list of items, after which each item should be fed into another agent (or independent set of agents) (possibly with some limit on agent concurrency), after which, etc., with some kind of graph structure determine how tasks move around, where there are reduce-like steps that look at the output from a bunch of tasks, and so on and so forth.. While it's technically possible to make Conductor do this, it wasn't really designed for it and, given that you can just tell agents to write scripts to create a structure like this, it seemed a lot simpler to just use custom vibe-coded scripts.

The kinds of tools that were designed for use of a lot of agents, like Gas Town, mostly seemed to be for less structured workflows, and they often seemed highly vibe coded and somewhat unreliable as a result, to the point that I wasn't sure why I would want to use one of them and not just cobble together something myself. What I mean there is, per the discussion in the long footnote about board games, if something doesn't meet a certain bar for quality and complexity, it's fairly easy to just make a version of it yourself. Even if the thing you make for yourself is buggier than the product, you can make sure that, given your workflow, it doesn't have bugs that impact your particular workflow, so you don't even have to meet the same overall quality bar as the product does for it to be better for you.

As a result, what I've ended up doing is making one-off loops for tasks that need to run for a while. There's probably an efficiency gain from building some kind of orchestrator that fits what I want, but I don't think I've really set up enough of these loops to know what I want, and what I want varies a lot depending on the problem. Sometimes I want to have certain metrics trigger a health check by an agent that can fix issues and sometimes I want an agent or set of agents to automatically be invoked after each iteration (whatever that means) to check on progress and sometimes I want agents to continuously check on progress. Sometimes I want a graph structure with some kind of multi-level triage, etc.

Because it's fairly easy to have an agent set up whatever arbitrary structure you want, I'll generally start with some kind of simple structure with some basic health checks and monitoring and then I'll check in on it periodically and try to make structural fixes to address issues so that the issue doesn't recur. In principle, something like this that's well set up could produce useful work indefinitely but, at least for what I've set up, these loops tend to degrade in productivity over time. Even if nothing is badly wrong, the loops are a lot more productive if I'm actively at the keyboard typing instructions to nudge things in various ways, while monitoring what's happening. If I leave one of these alone for a few days or a week, it will usually still be producing a non-zero amount of useful work, but not nearly as much as if I'd check in daily. I've tried various strategies to try to keep these loops more on track without intervention and I've found various things that can help, but I haven't figured out how to replace myself (yet?).

For example, in April/May 2026, I tried having a bunch of "personas" convene and reconvene to try to nudge the loop instead of doing it myself. At the time, I was trying to get a loop running smoothly that was doing some work on some code that uses CRDTs (which I know nothing about, BTW), so I tried diagnosing issues with something like "use independent agents to review as linus torvalds, kyle kingsbury, marc brooker, tptacek, dan luu, and 4 contrarian personas. have each think for a long time", with multiple iterations of this (yes, I do find it quite silly to invoke my own name). Each of these personas seemed to keep things on track in a different way, e.g., the "linus torvalds" persona would tend to push back against the loop's "natural" tendency to have things spiral out of control due by repeatedly adding unnecessary complexity, the "dan luu" persona would force the loop to measure things before implementing and also sometimes say things like "we must not rationalize XYZ" and push back on some bad reasoning. None of this was enough to replace human observation and intervention, but doing variations of this was enough to keep these loops running more smoothly, with less intervention required.

This kind of persona setup also improved things for architectural design, debugging, etc. Again, not so much that you wouldn't want a human at all, but enough that the human can do a bit less work.

All of this seems to work best if the human understands the common failure modes. For example, when asked to debug and explain when a bug was introduced, if just prompted to explain, even with multiple independent rounds of analysis, the agents would often come back with a completely incorrect explanation (maybe 50% of the time or so). When asked to confirm their hypothesis by actually executing code to check if the hypothesis is correct, that removed most incorrect explanations. Just adding forced checking to a simple prompt has a better success rate than having agents do independent analyses and cross-check each other (doing both together seems to have an even higher success rate).

A lot of getting value out of agents seems to be having some kind of understanding of their failure modes and then working around them. Earlier in the post, we noted that people aren't really going to be left behind if they're not using coding agents now and get started in M months because they can be at most M months behind, but will likely be much less behind than that because of how quickly things are changing. A major reason for this is, AFAICT, a lot of the skill involved in using agents is working around their failures. Of course AI labs want to fix these failures, which means new releases are designed to obsolete these skills as much as possible. This is possible to observe directly, in some failure modes that were quite common a year ago are now much rarer.

BTW, this is another reason I don't find a lot of benchmarks useful for me personally. In general, benchmarks have moved to trying not to "over explain" and instead act like a naive user who isn't very good at using agents. This is a reasonable thing for AI labs to care about because they want to make a product that needs as little expertise as possible to operate but, as a professional programmer, if there's way to work around some failure mode of agents, you're probably going to use it instead of acting like you're a benchmark, and you therefore get a very different result even if you're doing the exact task from some benchmark.

Thanks to Max Bittker, Dennis Snell, Em Chu, Yossi Kreinin, Peter Geoghegan, Michael Malis, @caleb@goodfeeds.net, Misha Yaugdin, and Jason Seibel for comments/corrections/discussion. *[I've been using AI fairly heavily since last November and the whole thing is a funny experience]: I've been pretty disconnected from the mainstream tech world since moving to Vancouver in 2020 while the world has been undergoing the AI revolution. In principle, I could've kept up online, but I haven't read much social media (including news aggregators) for a few years, so I have no idea what's going on in the tech world (I didn't deliberately quit social media; the time just got crowded out by other things). I haven't written much about AI because I figure that anything I think of was already said by somebody two years ago in a post I haven't seen. But as I've started catching up with folks, a number of people have found my thoughts useful or funny or interesting or something, so I guess I'll post these and see what more folks think. *[the source of a bug]: this was part of a project we'll discuss a bit more later (to find the half-life of a feature and/or bug fix) *[which was a lie]: when I asked again later, it was able to do this *[non-ironically such a great experience]: The next commit Codex flagged was the correct commit, and having it do all of this work was still a huge time saver, despite the series of lies Codex told me *[I was using coding agents heavily mid-late last year]: BTW, for the past N years, people have been saying that if you don't get on board with using AI heavily TODAY you'll be left behind. This always seemed wrong to me in principle and it seems to have been wrong in practice. If you start M months later, you'll be at most M months behind. But because the landscape changes so quickly, you'll actually be less than M months behind because so much of what you learn becomes obsolete almost immediately; the real value has to be the productivity you get in the short term because the half-life of the knowledge is so short *[this post covers a relatively disparate set of topics, here's a brief outline]: Different parts of the post are relatively independent and should make sense if other parts are skipped *[for example]: I've noticed that, when someone says 'for example', an internet smart guy will drop in to say, you can't conclude that from that, seemingly not understanding that the example follows from the conclusion and not the other way around; this happens even when the writer provides multiple examples, including examples that invalidate the smart guy's point, so I'm not sure there's much to be done about this *[support ticket (chat or email)]: Ideally, this would start with metrics / logs / traces and not support tickets, but we don't have the right information in metrics / log / traces to do this in the general case *[this works ok]: There generally isn't enough information in support tickets to be sure that the PR fixed the bug the user encountered, but this at least finds and fixes bugs with similar symptoms *[no known false positives]: There have been a few issues where people disagree over what the right fix is and/or whether the issue is technically a bug or if it's just confusing behavior that should be clarified in some way but maybe not technically a bug, but these have all been things that reasonable people can disagree on, not something I'd consider a real false positive *[I'm fairly comfortable trying to ship a large volume of code via a]: Whether or not this is worth it today is another question *[hand tests]: alternately, sometimes called 'hand jobs', a pun that made sense because we called a run in our scheduling system a 'job', so every test ran in a 'job' *[test]: verification *[on prem]: Our machines, including hundreds that had other uses, were managed by our IT team of two people, who also had other IT duties, such as fixing Visual Source Safe every time it broke *[the machines took up half a floor of the building we were in]: At one point, we were using enough power to cause brownouts and the power company had to run more power to the area *[a different simulator setup]: it was using standard commercial simulators like VCS and NC-Verilog, as opposed to almost all other tests, which ran on a higher latency but higher throughput bit-parallel *[Personally, I'm very comfortable shipping code without human review because I've seen it done]: Michael Malis has mentioned he feels comfortable with this because when he was founder and CTO or CEO, he had to supervise a lot of code being shipped without really having the time to look at it; other people will have different reasons as well *[technically more challenging]: from a correctness standpoint *[much higher if you adjust for severity]: a similar sized group of developers will, maybe weekly or daily, ship a more severe bug than ever shipped in the history of Centaur; as a counterpoint, Yossi Kreinin notes that, adjusted by the cost of fixing the bug, the gap shrinks back by some amount, making this an incentive problem, which I think is not untrue, but still not a reason to test inefficiently given that any amount of time is spent on testing at all *[don't want to move away from human review]: there might be non quality (as in non bug rate) related reasons to keep human review, such as keeping a high bar for code quality or keeping the codebase human-understandable, which pretty much immediately stops being the case if you let a fleet of agents go wild on a codebase *[it's just really inefficient to write tests by hand]: a nice side effect of writing a test generator instead of writing a test directly even if you're writing the code is that the test generator is more structurally different than the code and you're less likely to encode the same incorrect behavior in the test generator and the code; this goes double for using agents for this problem since you should be using agents with different contexts for this *[unit testing]: A lot of software companies primarily rely on unit tests, but I think that's something they can get away with because they're not focused on quality and they have money to burn. *[does pretty poorly]: Yossi Kreinin says: unit testing doesn't get you very far correctness wise; it checks the code wasn't violently shat on, blowing a chunk of it apart completely, and it's 'effective' in being able to do that in a software context where correctness is inherently deprioritized and the desire to test and to act on test failures is very low. it is as effective as a 2-stick toilet where the 1st stick is for hanging your pants on and the 2nd is for driving the wolves away - nobody's dream but better than nothing in a certain environment; Dennis Snell has an additional point, that unit tests can be useful for documentation even if they shouldn't be relied on for quality *[the level of effort isn't really what's making the difference]: BTW, I've seen this done, and it didn't make all that much difference; companies might call this a 'code yellow' or a 'code red' or whatever, and then spend some number of months focusing on quality *[rate the tests as somewhere between worthless and marginally useful]: Some would say this is too generous. For example, Dennis Snell says: 'actively harmful, filling the test suites with bogus assertions, and oftentimes over-restrictive assertions about things that don't speak to behaviors or quality. just more noise to cloud the test suite. *[For example]: I've noticed that, when someone says 'for example', an internet smart guy will drop in to say, you can't conclude that from that, seemingly not understanding that the example follows from the conclusion and not the other way around; this happens even when the writer provides multiple examples, including examples that invalidate the smart guy's point, so I'm not sure there's much to be done about this *[that's a huge win]: This topic is covered more in a later section, but one reason people seem to arrive at deeply suboptimal workflows is that LLMs are so high leverage (for some things) that it's easy to get a massive improvement and stop looking for further improvement; I'm pretty sure this is true of everything I've found effective as well, as there are too many potentially effective things to really try them all when the situation is changing so rapidly *[possible to give instructions that will work well]: as with other tasks, the LLM will often ignore your instructions or say that it did what you said but not do it, so you either need to check or have a system for checking *[I'm sure there's a lot of room for improvement]: This is already an overly long post without going into more detail here, so I'll defer a very detailed discussion of this for another post which could easily be longer than this entire post *[At least for now, having access to a model that's better than anything you can publicly use won't save you]: I wrote this sentence when Fable wasn't released, after which it was released, and then unreleased, and then released again *[their vaunted Mythos model that's too dangerous to release from the public]: This was before Fable was released to the public (and then unreleased and then released again) *[agents]: or sub-agents *[the same question]: in a way that doesn't immediately return the same cached answer *[googling 'caveman mode']: logged out incognito window, etc. *[the top hit on YouTube]: The #2 hit was a video titled 'Caveman Claude Code Is the New Meta (Here's the Science)', *[saying it's a joke]: He still seems to be making strong claims for it, so this seems to be like when someone claims an off color comment was just a joke when called out on it? *[SEO spam article]: The entire blog was LLM generated SEO spam posts *[after 50 runs is in favor of caveman, with 1.03 vs. 1.01 speedup, and $17.97 in 13m46s vs. $24.21 in 16m52s]: This also shows you why people rarely benchmark these recommendations. At retail rate, this would've been around $2000 to run *[P(caveman better)]: Bayesian-bootstrap probability *[a deadline of 10ms per move]: to prevent the benchmark from being nearly totally a measure of how good the AI is at staying within the deadline, if a bot misses a deadline, a random move is selected instead of the standard mechanic of losing the game *[I didn't cherry pick the order of these results to present some kind of surprising narrative reversal]: I originally presented all of the results together in one big set of graphs, but draft readers suggested this was too busy and confusing, I broke out results in an order that seemed natural without looking at what the sub-results looked like, and it had been months since I looked at the results when I did this, so I didn't remember what the sub-results were *[quite a few more runs than any reasonable person is going to make to support some comment they're throwing out on the internet]: hey, I never claimed to be a reasonable person *[In general]: Not just in the context of agents or LLMs *[For example, on one benchmark]: an actual benchmark, which I'm using as an example here because it's been sent to me multiple times *[If many other users based their decisions on these benchmarks, then AI labs would need to care about their results on these benchmarks]: Yossi Kreinin notes that even though Dhrystone has been known to be a worthless benchmar for over 30 years, decision makers in some domains still pay attention to it, so CPU makers in those areas have to pay attention to it; as a user satisfaction metric, NPS might be more analogous to these model benchmarks, in that it's going to be somewhat correlated to things you care about, but it's not clear why you should care about NPS in particular *[during that time period, most people still primarily used Claude and Opus]: I wonder if people would've made different decisions if they were spending their own money to support their own business, but that's a general question about enterprise software purchasing and not something specific to coding models *[I don't know why anyone should really care what these summary metrics show]: Of course both the creators of the models and users care that the models are getting better, and very large differences on these sorts of benchmarks do generally indicate real capability differences, but that doesn't make one model being a bit ahead of another model on the summary metric for a benchmark meaningful *[an RL environment startup]: BTW, I'm really curious about the RL environment market. If you know something about it, please reach out to me! *[Chinese models score well on SWE-Bench but underperform on novel tasks]: This comment was made before GLM-5.2 was released *[undertake high variance activities]: Peter Geoghegan notes that microptimization is also a high variance activity that leads to superstitious beliefs, which reminds me of a story someone at one of the big game companies once told me about having to intervene when some senior devs were explaining a junior dev that they should unroll some loop for performance, but on actually looking at the compiled output or profiling the code, unrolling didn't improve performance, although it might have on the same code if it were 1985 *[gamblers]: Em Chu says: Using an LLM does feel like gambling. Maybe like a slot machine whose average payout is $5, but hits you with an unknown amount of x-ray radiation 20% of the time. *[wear their lucky socks or have a specific routine they do]: some athletes are also known for doing this kind of thing *[maybe we're "holding it wrong" when it comes to using Opus]: I don't find this last one super likely given that this includes people who are primarily Opus users and are more likely to be 'holding GPT wrong' than the other way around, but you never know *[my doing experiments in general]: I don't know that I'll necessarily run this experiment but, per https://danluu.com/why-benchmark/, I'll probalby do some kind of experiment, and LLM experiments end up being somewhat costly *[run a simple agentic loop where the agent (or agents) "understand" and analyze the problem and then fix up the parts that need to be fixed for me to extract a good analysis from what the agent has produced]: BTW, the LLM loop doesn't directly do data analysis—it modifies code that produces analyses and plots. In general, LLMs seem much better at modifying code iteratively than modifying their reasoning iteratively, so of course the loop is working on some code. *[my guess is that it would've taken one to three hours of my time over the course of a day or two]: I've done this enough on comparable problems that I think this is a reasonable estimate *[the whole thing still moved a lot faster than if I tried to steer the LLM]: for obvious reasons, this method of operation is much higher variance than normal vibe coding, so this can take more time than a normal LLM-assisted data analysis but, on average, I've found this to be a larger time saver *[where someone who hasn't done it will explain why it's easy]: On the Ark Nova discord, somebody claimed that if they were handed an Ark Nova simulator, they'd be able to build a strong AI in a day, but having implemented a handful of board games and AIs, building the simulator that implements the rules is the easier and less time consuming part by far; given agentic tools, someone who can't trivially do that for a board game the complexity of Ark Nova probably has little to no chance of building a strong AI *[strongest player in the world]: There's one higher rated player who only plays asyc/PBEM, so this is referring to the strongest player in the world at real-time games *[play it like a turn-based game]: This is referring to keeping the bot on some kind of fast real-time control, such as 15 seconds per move, while playing the game like an async/pbem game, where you can take as long as you want for a move *[just training on my laptop]: I don't know how reasonable this was, but my thought was that I'd learn more if I didn't just throw a bunch of resources/money at the problem and tried to do it in a resource constrained way. After achieving superhuman performance, I did grab a VM for a couple weeks to run some parameter tuning experiments, but I wanted to avoid doing that until I got fairly good performance *[then never seen again]: based on how well that bot played against humans vs. how well the slow/weak wasm version of my bot fared against humans, I think my bot is significantly stronger, but I was never able to compare directly *[you have this net that's doing who-knows-what and some kind of search function that's doing who-knows-what and you can sometimes spot specific obviously bad moves]: it's possible to really understand these things, but very time consuming *[you can actually see how the person tried to nudge the bot]: It's arguably unfair of me to make a comment like this when my bot isn't open source, but I haven't open sourced it because the Azul community asked me not to because open sourcing it would immediately lead to rampant cheating; at least for now, most attempts to build strong board game AIs take a long time or fail and competitive online play is still a thing *[open in column 2]: if you don't play Azul, it's fine to not know what this means; you can think of it as a bit analogous to always playing the same opening in chess *[fork the game into a new game with different column permutations]: done in a way to avoid poisoning the bot's notion of win probability from different positions *[you can't effectively lift-and-shift a human workflow to agents]: I've seen this attempted quite a few times and I have yet to see it succeed for a non-trivial workflow or task; no doubt this will become more effective as agents get better, but I'm doubtful that it will be a good way to use agents even if agents improve enough that it can work *[It's clear that the driving enabled them to do things they wouldn't have been able to otherwise do]: and many would say they'd be better off had they not done those things *[in some general way]: as opposed to hard coding some specific inputs that will reproduce the bug, which the LLM will often try to do if not prevented from doing this *[I was somewhat off here as the typical 200:1 is closer to 100:1 than it is to 1000:1.]: though, I was thinking about this as a log scale, so it's not off by as much it might seem, since the midpoint is 316:1 *[strike gold]: find product-market fit or w/e *[Galapagos Island]: Vancouver *[it wasn't really made to support a lot of things I'd want to do]: I haven't checked since I tried it; it may have more support for these kinds of workflows now