我们将 Agent 循环从 Anthropic 迁移至 GLM 后学到了什么

TL;DR：我们将 Unblocked 的大部分 Agent 流量从 Claude Opus 迁移到了 GLM 5.2。

按“每 token”计算，理论上可节省 95%；但按每项任务衡量的生产环境现实中，实际节省了 68%。要实现这一点，我们进行了基于真实代码审查的盲测 A/B 评估，构建了带熔断器的多服务商推理池，并花费数周应对各种“OpenAI-compatible”意外。

背景：Unblocked 赋予 Agent 顶尖工程师所具备的组织理解能力。其上下文引擎会连接并协调代码、对话、议题、文档和生产系统中的知识，随后在工作发生的任何位置提供这些理解。Agent 可通过 MCP/CLI/API 访问上下文；Code Review 和 Unblocked Code 等 Unblocked 产品则直接将这些上下文应用于代码审查和代码编写。最终结果是：用更少的 token、更少的迭代和更少的来回沟通，产出更好的软件。

过去近两年中，Unblocked 一直运行在 Anthropic 模型之上。我们的产品由一组 Agent 循环构成：回答工程问题、审查代码，以及搜索组织的机构性知识。Claude Opus 几乎处于所有这些流程的外层循环中。

今年夏天，我们将大部分流量迁移至 GLM——一种由第三方推理服务商提供服务的开源权重模型。本文将介绍我们为何这么做、测量了什么、遇到了哪些故障，以及目前的落点。我们不会披露系统内部实现，但这些发现应当适用于任何大规模运行 Agent 循环的团队。

问题#

审计 AI 支出时，一个数字格外突出：超过 80% 的支出来自主 Agent 循环中对前沿模型的使用，且几乎全部是 Opus。

我们列出了可调节的杠杆：

• 模型。 对代码审查的输出质量影响最大；对问答的影响则相对较小。
• 轮次。 在针对长循环优化缓存后，其重要性低于预期。
• 工具。 影响可变，并且会随着功能发布而增长。
• 缓存。 已完成优化。激进的 Prompt 缓存使我们在轮次之间保持超过 95% 的缓存命中率，因此在这方面已没有值得回收的空间。
• Prompt。 基本完成优化，并且随着每一代模型进步，其影响正越来越小。

缓存和 Prompt 的优化空间已经耗尽。删除功能并不在选项之中。剩下的只有模型本身，而前沿模型定价与开源权重模型定价之间的差距已经大到无法忽视。GLM 5.2 的报价不到前沿模型费率的三分之一；在各个服务商的原始每 token 定价上，相比 Opus 的节省看起来接近 95%。

问题是：在真实工作负载上、由真实用户评判时，质量能否维持？基准测试无法回答这个问题。

每 token 定价，是你对自己说的谎#

第一个修正很快到来。我们的一名工程师从 Fireworks 仪表盘提取了真实用量：在双方均计入缓存后，提供 23.5 亿 token 的服务，其成本约为同等用量使用 Opus 所需成本的 5%。从纸面上看，成本降低了 95%。

随后，我们对同一提交、同一个 PR、同一个 Prompt 运行了相同的代码审查：一次使用 Opus，一次使用 GLM。现实世界中的节省是 68%，而非 95%。

这个算术关系是乘法，而费率卡上只印出了两个项中的一个：

Copy

effective cost = token price × tokens required per task

仅从价格看，GLM 像是提升了 20 倍。但从账单看，大约只是 3.1 倍。

差异来自行为表现。GLM 在每项任务中使用的 token 远多于 Opus：

• 它会进行更多工具调用。它探索得更激进，有时这是好事，有时则是浪费。
• 它会犯更多错误，而错误会消耗轮次。一次格式错误的搜索或空结果，意味着循环要再经历一次往返。
• 它会摄入更大的工具结果，并花费更多 token 对其进行推理。
• 总体而言，它更健谈。

这并非来自少量样本的轶事。在迁移期间，我们搜索流程中的平均每次调用 token 数从约 18,500 上升至约 24,700，增长了三分之一；而 GLM 当时只处理了约 45% 的调用。探索开销在全服务集群账本中清晰可见。

因此，真正重要的指标是每项任务成本，而非每 token 成本。任何评估模型替换的团队，都应在端到端条件下运行完全相同的工作负载，并比较账单，而不是比较费率卡。68% 的降幅仍然是巨大的收益，只是并非费率卡所承诺的那种收益。

我们如何评估质量#

我们不相信内部的主观感受，也不相信公开基准测试。我们将比较机制直接构建进产品中。

针对代码审查，我们建立了双管线：两种模型针对完全相同的 PR、相同的提交和相同的 Prompt 运行，并发布两组评论。终端用户无法辨别每条评论由哪个模型生成。我们原本就会记录审查评论的反馈情绪，并可将其归因至相应管线，因此每一次反馈都成为一次盲投票。

情绪指标衡量的是精度：在获得反馈的评论中，有多少比例是负面反馈？在数千次生产代码审查中，三种模型的排序十分清晰：

模型	审查数	评论数	每次审查评论数
Claude Opus	3,708	1,146	0.31
GLM 5.2	3,384	1,568	0.46
基于 GPT 的替代方案	722	617	0.85

在精度方面，Opus 排名最佳，GLM 紧随其后，而基于 GPT 的替代方案则远远落后——尽管后者每次审查发布的评论数量接近 Opus 的三倍。GLM 只是更愿意发言。在代码审查中，这很危险：一个发布低质量发现的审查工具会训练开发者忽略它，而这正是该工具最糟糕的形态。考虑到价格，GLM 的精度处于我们可以接受的容忍范围内；第三个模型则不在这个范围内。

但也应坦诚这一指标实际测量的是什么。只有一部分评论会得到反馈，而糟糕的评论比优秀的评论更容易促使用户作出反应，因此绝对情绪数值存在偏差，只有排序值得信赖。每次审查的评论数也并非召回率——一个模型可以写出更多评论，却没有发现更多真实缺陷。我们将此作为适用于自身工作负载的部署信号，而不是作为能力相同的证明。“这个模型能否发现 bug？”已不再是问题。真正的问题是：一项发现有多大概率可执行、有多少误报伴随其中、模型是否知道何时_不_应评论，以及它的输出是在建立还是侵蚀开发者信任？

我们也以相同方式评估问答：先在我们自己的组织内启用 GLM，要求所有人标记错误答案，观察一周。然后逐步扩大部署，先从免费套餐开始，最后才是最大客户。

“OpenAI-compatible”是一个范围，而不是标准#

这里实际上消耗了大部分工程时间。每一家开源权重模型服务商都宣传 OpenAI-compatible API。实践中，兼容性在有趣功能开始的地方就结束了。

以下是我们遇到的问题，全部发生在生产环境中或上线前一周：

推理没有标准的线路格式。 Fireworks 在 reasoning_content 字段中返回思考输出，Together 在 reasoning 中返回，而 OpenAI 第一方甚至不会在 chat completions 中提供它。如果你运行多轮工具调用循环，还需要将推理内容回放到先前的 assistant 轮次中，而每个托管服务对这一点都有自己的预期。

推理强度映射可能悄无声息地将账单推至上限。 GLM 的聊天模板会将除字面字符串 high 之外的任何强度值都视为最高强度。我们当时发送了一个自认为表示“最低”的值，直到发现问题前，每次调用都在为最高强度推理付费。

Prompt 缓存的键在各处定义不同。 OpenAI 第一方使用专门的缓存键字段。Fireworks 和 Together 则完全忽略该字段，而是基于 user 字段设定缓存键。在发现这一点前，多轮会话在这些服务商上每次请求都无法命中缓存。请注意这种故障的表现：响应成功、输出正确、没有任何报错。如果没有按服务商区分的缓存命中指标，它看起来只会像普通的成本波动。

用量统计并不一致。 一些 OpenAI-compatible API 会将缓存 token 包含在 prompt_tokens 中。如果成本管线假设这些分类桶彼此不重叠，你会对自己重复计费，后续每个仪表盘都会出错。

结构化输出支持因托管服务而异，只有持续监控时才会显性失败。 Baseten 直接拒绝 response_format: json_schema。结果是：在七天内，336 次结构化调用中没有一次成功，导致这些流量被悄无声息地完全固定在 Fireworks 上。相关地，在一些服务商上，将工具定义与结构化输出请求一同发送会导致硬性 400 错误；在另一些服务商上，则会产生无法解析的工具调用响应。

流式用量统计的结构不同。 使用 vLLM 提供服务的 Baseten，会在每个流式数据块中报告累积 token 计数，而不是只在最后提供一次总数。如果进行朴素聚合，记录的用量会被数据块数量成倍放大。Fireworks 则会在你仅仅请求该选项时返回 400。

Prompt 无法原样迁移。 为 Claude 调优的 Prompt 会累积思维链脚手架、带有 Anthropic 风格的结构化输出指令，以及 Anthropic 形态的工具定义。在将推理作为一等 API 控制项的模型上，“逐步思考”脚手架属于过度指定，且会适得其反。我们构建了一个确定性的适配器层，按目标模型家族重构 Prompt，而不是直接透传。这里“确定性”是关键：该路径中没有模型生成的翻译，因此给定输入、目标模型和服务商，始终会产生相同请求。

这些都不是什么罕见问题，教训也不是某一家服务商正确实现了 API、其他服务商没有。边缘行为并不存在共享规范。你真正需要的是覆盖所依赖的每一种组合的兼容性测试套件——模型 × 服务商 × 流式模式 × 推理模式 × 工具使用 × 结构化输出——因为如果你没有同时测量质量和单位经济效益，这些故障中的每一个都会在数周内悄悄损害其中之一。

你买的不是模型，而是一套服务栈#

使用 Anthropic 时，模型和服务基础设施来自同一家公司。对于开源权重模型，权重是商品，服务栈才是产品。我们在 Fireworks、Together 和 Baseten 上运行同一个 GLM 5.2，它在延迟、稳定性、功能支持和价格上都表现不同。在迁移期间，Fireworks 在我们的工作负载上的平均延迟比 Opus 服务路径高约 30%——这个差距足以体现在用户可感知的响应时间上。

因此，我们不再将 GLM 视为一个服务商，而是将它当作一个资源池：

• 在同一路由层之后接入多个独立服务商，每个服务商都由熔断器封装。当某个服务商性能退化时，其熔断器打开，流量会转移至健康节点；熔断器状态会在整个服务集群中传播，以便所有节点协同响应。
• 我们最初在 Fireworks 和 Together 之间进行均匀轮询，之后因 Baseten 具备更有利于合规的静态基础设施而将其加入为第三个节点，最终又移除了 Together。随后我们发现，均匀轮换是错误的：Baseten 在费率卡的每个项目上都低于 Fireworks，大约便宜 20%，并且基线运行速度更快，因此均分流量意味着在没有获得任何可靠性收益的情况下，购买了更昂贵、更慢的请求。我们改为优先级路由：Baseten 作为主服务，Fireworks 作为故障转移。故障转移刻意不保持粘性，因此一次短暂波动只会转移一个请求，而不会迁移整个调用方。
• 人工选择优先级，仍然是对动态问题的静态回答——费率卡会变化，本月最快的服务商下个月未必仍然最快。因此，我们最终转向自适应路由：在生产环境中持续采样每个服务商的成本、可靠性和速度，并将大部分流量发送给当前在这些维度综合表现最佳的服务商，而不是等待人工发现问题后再重排列表。
• Claude 仍然作为深度兜底接入。如果整个 GLM 资源池耗尽，流量会转移到 Claude 资源池，而不是直接失败。

这套机制带来的收益，在诊断方面与可用性方面同样显著。8 月 11 日，GLM 代码审查单日发生了超过 500 次失败，房间里的第一反应是将整个工作负载切回 Claude。但问题不在模型。资源池中一个节点突发的 429 错误持续时间很短，而另一个节点明显更差的延迟，将可见影响延长到了实际故障之后很久。如果没有按服务商区分的故障转移和指标，每一次服务商事故看起来都会像模型质量事故——最终你会基于错误诊断，做出昂贵且永久的路由决策。

成本核算也必须改变。每个模型只有一家供应商时，model + tokens = cost 这套公式可行。有了资源池后，同一个模型在不同服务商处对应不同费率卡，因此每条用量记录都必须携带实际提供服务的服务商信息，否则仪表盘就会逐渐偏离账单。

而可靠性问题来自各个方向。我们曾遇到 Fireworks 因消费限额在午间暂停账户，导致一部分生产能力随之中断，直到我们在数小时内将该流量切回 Anthropic。另一方面，今年更早时候，我们也曾因 Anthropic 侧的可靠性问题，将流量从 Anthropic 直接 API 迁移至 Bedrock。这个市场中没有任何一家足够稳定到可以成为单点故障。故障转移机制并不是 GLM 专属的保险；它只是如今在任何推理服务上运行所必需的能力。

回归问题，以及它教会我们的事#

在将一个主要工作负载迁移到 GLM 约一周后，我们的延迟和质量指标出现下滑。当时有两项变更先后上线：模型切换，以及循环轮次预算增加。要厘清两者，需要真正的聚合分析，而不是抽样检查。

根本原因恰好可拆解为前文所述的行为差异：GLM 探索更多、犯错更多，因此每项任务消耗更多轮次，

TL;DR: We moved most of Unblocked's agent traffic from Claude Opus to GLM 5.2.

“Per-token” math promised 95% savings; per-task production reality delivered 68%. Getting there took blind A/B evaluation on real code reviews, a circuit-broken multi-provider serving pool, and weeks of "OpenAI-compatible" surprises.

Background: Unblocked gives agents the organizational understanding of your best engineers. Its context engine connects and reconciles knowledge across your code, conversations, issues, documentation, and production systems, then makes that understanding available wherever work happens. Agents access context through MCP/CLI/APIs and Unblocked products like Code Review and Unblocked Code apply that context directly to reviewing and writing code. The result is better software, produced with fewer tokens, fewer iterations, and less back-and-forth.

For most of the last two years, Unblocked ran on Anthropic models. Our product is a set of agentic loops: answering engineering questions, reviewing code, searching across an organization's institutional knowledge. Claude Opus sat in the outer loop of nearly all of it.

This summer we moved most of that traffic to GLM, an open-weight model served by third-party inference providers. This post covers why we did it, what we measured, what broke, and where we've landed. We're not going to describe our system internals, but the findings should transfer to anyone running agent loops at scale.

The problem#

When we audited our AI spend, one number stood out: over 80% of it came from frontier model usage in our main agent loops, almost entirely Opus.

We listed the knobs available to us:

• Model. The single biggest factor in output quality for code review, less so for Q&A.
• Turns. Less important than expected once caching is optimized for long loops.
• Tools. Variable, and growing as features ship.
• Caching. Already optimized. Aggressive prompt caching holds us above a 95% cache hit rate turn-over-turn, so there was nothing meaningful left to reclaim there.
• Prompts. Mostly optimized, and mattering less with every model generation.

Caching and prompts were tapped out. Deleting features wasn't on the table. That left the model itself, and the gap between frontier pricing and open-weight pricing had become impossible to ignore. GLM 5.2 was billed at less than a third of frontier rates, and on raw per-token pricing across serving providers the savings looked closer to 95% against Opus.

The question was whether the quality held up in production, on real workloads, judged by real users. Benchmarks weren't going to answer that.

Per-token pricing is a lie you tell yourself#

The first correction came fast. One of our engineers pulled real usage off the Fireworks dashboard: 2.35 billion tokens served for roughly 5% of what the same volume would have cost on Opus, with caching accounted for on both sides. A 95% cost reduction, on paper.

Then we ran the same code review on the same commit, same PR, same prompt, once with Opus and once with GLM. The real-world saving was 68%, not 95%.

The arithmetic is multiplicative, and only one of the two terms is printed on the rate card:

Copy

texteffective cost = token price × tokens required per task

On price alone, GLM looked like a 20× improvement. On the bill it was about 3.1×.

The gap is behavioral. GLM uses far more tokens per task than Opus does:

• It makes more tool calls. It explores more aggressively, which is sometimes good and sometimes waste.
• It makes more mistakes, and mistakes cost turns. A malformed search or an empty result means another round trip through the loop.
• It ingests larger tool results and spends more tokens reasoning about them.
• It's more talkative in general.

This isn't an anecdote from a handful of samples. In our search flow, average tokens per call rose from about 18,500 to about 24,700 (up a third) over the migration window, and GLM was only serving around 45% of those calls. The exploration overhead is visible in the fleet-wide ledger.

So the metric that matters is cost per task, not cost per token. Any team evaluating a model swap should run identical workloads end to end and compare invoices, not rate cards. A 68% reduction is still an enormous win. It's just not the win the rate card promised.

How we evaluated quality#

We didn't trust internal vibes, and we didn't trust public benchmarks. We built the comparison into the product.

For code review, we set up dual pipelines that run two models on the exact same PR, same commit, same prompt, and publish both sets of comments. End users can't tell which model produced which comment. We already recorded reaction sentiment on review comments and could attribute it per pipeline, so every reaction became a blind vote.

Sentiment measures precision: of the comments that got a reaction, what fraction were negative? Across thousands of production reviews, three models sorted cleanly:

Model	Reviews	Comments	Comments per review
Claude Opus	3,708	1,146	0.31
GLM 5.2	3,384	1,568	0.46
GPT-based alternative	722	617	0.85

On precision, Opus ranked best, GLM close behind, and the GPT-based alternative far behind despite posting nearly triple Opus's comment volume per review. GLM was simply more willing to speak. In code review that's dangerous: a reviewer that posts weak findings trains developers to ignore it, which is the worst possible shape for the tool. GLM's precision was within a tolerance we could accept for the price. The third model wasn't.

Be honest about what this measures, though. Only a fraction of comments ever draw a reaction, and a bad comment is more motivating to react to than a good one, so the absolute sentiment numbers are biased and only the ordering is trustworthy. Comments per review is not recall either — a model can write more comments without finding more real bugs. We used this as a deployment signal for our workload, not as proof of equal capability. "Can this model find bugs?" stopped being the question. The questions were: how often is a finding actionable, how many false positives ride along with it, does the model know when not to comment, and does its output build or erode developer trust?

We ran the same play for Q&A: enable GLM for our own organization first, tell everyone to flag bad answers, watch for a week. Then roll out gradually, free tier first, largest customers last.

"OpenAI-compatible" is a spectrum, not a standard#

This is where most of the engineering time actually went. Every open-weight serving provider advertises an OpenAI-compatible API. In practice, compatibility ends where the interesting features begin.

Things we hit, all in production or in the week before it:

Reasoning has no standard wire format. Fireworks returns thinking output in a reasoning_content field, Together returns it in reasoning, and first-party OpenAI doesn't put it in chat completions at all. If you run multi-turn tool-calling loops, you also need to replay reasoning content back onto prior assistant turns, and each host has its own expectations about that.

Reasoning effort mappings can silently max out your bill. GLM's chat template treated any effort value other than the literal string high as maximum effort. We had been sending a value we thought meant "minimal" and were paying for maximum-effort reasoning on every call until we caught it.

Prompt caching is keyed differently everywhere. First-party OpenAI uses a dedicated cache-key field. Fireworks and Together ignore that field entirely and key the cache off the user field. Until we discovered this, multi-turn sessions were missing cache on every request on those hosts. Note what that failure looks like: successful responses, correct output, no errors anywhere. Without per-provider cache-hit metrics it reads as ordinary cost variance.

Usage accounting is inconsistent. Some OpenAI-compatible APIs include cached tokens inside prompt_tokens. If your cost pipeline assumes disjoint buckets, you double-bill yourself and every dashboard downstream is wrong.

Structured output support varies per host, and fails loudly only if you're watching. Baseten rejected response_format: json_schema outright. The result: zero of 336 structured calls succeeded over seven days, silently pinning that traffic entirely to Fireworks. Related: sending tool definitions alongside a structured-output request caused hard 400s on some providers and unparseable tool-call responses on others.

Streaming usage stats differ in shape. Baseten, which serves on vLLM, reports cumulative token counts on every streamed chunk rather than a single final total. Aggregate naively and you multiply your recorded usage by the chunk count. Fireworks returns 400 if you even request that option.

Prompts don't transfer verbatim. Prompts tuned for Claude accumulate chain-of-thought scaffolding, Anthropic-flavored structured-output instructions, and Anthropic-shaped tool definitions. On models where reasoning is a first-class API knob, "think step by step" scaffolding is over-specified and counterproductive. We built a deterministic adapter layer that restructures prompts per target model family instead of passing them through. Deterministic is the operative word: there is no model-generated translation in that path, so a given input, target model, and provider always produce the same request.

None of these are exotic, and the lesson isn't that one provider implemented the API correctly and the others didn't. There is no shared behavior at the edges. What you actually need is a conformance suite over every combination you depend on — model × provider × streaming mode × reasoning mode × tool use × structured output — because each of these failures quietly corrupts either your quality or your unit economics for weeks if you aren't measuring both.

You're not buying a model, you're buying a serving stack#

With Anthropic, the model and the serving infrastructure come from one company. With open-weight models, the weights are a commodity and the serving stack is the product. We ran the same GLM 5.2 across Fireworks, Together, and Baseten, and it behaved differently on each in latency, stability, feature support, and price. Over our migration window, Fireworks averaged roughly 30% higher latency on our workloads than our Opus serving paths — enough of a gap to show up in user-visible response times.

So we stopped thinking of GLM as a provider and started treating it as a pool:

• Multiple independent serving providers behind one routing layer, each wrapped in a circuit breaker. When a provider degrades, its breaker opens and traffic shifts to healthy legs; breaker state propagates across our fleet so every node reacts together.
• We started with an even round-robin across Fireworks and Together, added Baseten as a third leg for its compliance-friendly static infrastructure, and eventually dropped Together. Then we realized even rotation was wrong: Baseten undercut Fireworks on every line of the rate card (roughly 20% cheaper) and ran faster at baseline, so an even split was buying more expensive, slower requests for zero reliability gain. We moved to priority routing: Baseten as primary, Fireworks as failover. Failover is deliberately not sticky, so a transient blip diverts one request instead of migrating the whole caller.
• Hand-picked priority is still a static answer to a moving question — rate cards change, and a provider that is fastest this month may not be next month. Which is why we eventually moved to adaptive routing: continuously sampling each provider's cost, reliability, and speed in production and sending the majority of traffic to whichever provider is currently winning on that blend, instead of waiting for a human to notice and re-rank the list.
• Claude remains wired in as the deep fallback. If the entire GLM pool is exhausted, traffic diverts to the Claude pool rather than failing.

The payoff from that machinery is as much diagnostic as it is uptime. On August 11, GLM code review threw more than 500 failures in a day, and the first instinct in the room was to move the whole workload back to Claude. It wasn't the model. A burst of 429s from one leg of the pool was short-lived, and the other leg's materially worse latency stretched the visible impact well past the actual outage. Without per-provider failover and per-provider metrics, every provider incident looks like a model-quality incident — and you end up making an expensive, permanent routing decision off a bad diagnosis.

Cost accounting had to change too. With one vendor per model, model + tokens = cost works. With a pool, the same model has a different rate card per provider, so every usage record has to carry the provider that actually served it, or your dashboards drift from your invoices.

And reliability cuts in every direction. We've had Fireworks suspend our account mid-day over a spending limit, taking a slice of production capability with it until we flipped that traffic back to Anthropic for a few hours. But we've also, earlier this year, moved traffic off Anthropic's direct API to Bedrock because of Anthropic-side reliability issues. Nobody in this market is stable enough to be a single point of failure. The failover machinery isn't GLM-specific insurance; it's just what running on anyone's inference looks like now.

The regression, and what it taught us#

About a week after moving a major workload to GLM, our latency and quality metrics dipped. Two changes had shipped close together: the model swap, and an increase in the loop's turn budget. Untangling them took real aggregate analysis, not spot checks.

The root cause decomposed into exactly the behavioral differences from earlier: GLM explores more, errs more, and therefore consumes more turns per task, all on serving infrastructure that was already slower per call. The turn increase permitted the behavior but didn't cause it; the model's shape did. Framed precisely, GLM's weakness here isn't intelligence — it's exploration discipline and tool-call correctness.

That distinction matters when you're debugging an agent, because "the answers got worse after the model swap" is five failures:

• the model reasoned poorly;
• the model used tools differently;
• the loop gave that behavior too much room;
• the serving provider was slow or partially incompatible;
• the cost and latency instrumentation was wrong.

Telling them apart requires model output, tool behavior, provider health, latency, and cost measured together. Any one of those signals alone will point you at the wrong culprit.

The interesting outcome was a design question we're still working through: turn budgets were tuned for a model that averaged 2.4 tool calls per answer. A model that naturally explores more may deserve a tool-call budget instead of a turn budget, one that treats parallel tool calls as a single unit so the model is incentivized to batch its exploration rather than meander through it. Budgets designed around one model's temperament don't transfer to another's.

The ledger, five weeks in#

We track unit cost per call for every agent flow, normalized per weekday, and decompose each week-over-week cost change into a volume effect (traffic moved) and a unit-cost effect (economics moved). Here's the window from mid-July, before the defaults flipped, through this week:

Flow	GLM share of calls	Unit cost per call	Change vs pre-migration
Code review	6% → 86%	$0.089 → $0.027	down 68%
Search agent	2% → 43%	$0.052 → $0.029	down 43%
Internal MCP search (never migrated)	0%	~$0.024 → ~$0.024	within a few percent

Three things worth pulling out:

• The drop is step-shaped and it holds. Code review's unit cost fell 68% in the single week the default flipped and has stayed there since. That fleet-wide number landing on the same 68% we measured in the controlled same-PR experiment is exactly what you want to see: the A/B predicted production.
• The control flow didn't move. The one flow still running entirely on Claude Sonnet and Haiku held its unit cost within a few percent across all five weeks. Whatever moved the other two lines, it wasn't seasonality or traffic mix.
• Volume grew while spend fell. In the search flow's migration week, call volume rose about 20% and per-weekday cost still fell 28%, because the unit-cost effect was more than twice the size of the volume effect. Search is only partway migrated: Opus still serves roughly a quarter of those calls, largely for compliance-constrained organizations, so the 43% is a blended number with room left in it.

One caveat on the unit: these are ledger calls, not completed reviews or answered questions. Calls are the right unit for infrastructure accounting and the wrong unit for product economics, so we watch both — cost per call, and end-to-end cost per completed task including retries, failed calls, cached input, and tool results.

Where we are#

As of August 2026:

• Code review and Q&A default to GLM 5.2. Free-tier users moved first, then nearly all customers.
• Organizations with data processing agreements, or with contractual model-vendor or region restrictions, stay on Claude. Routing enforces this at request time: for a restricted org, a GLM-targeted request is rewritten to the Claude pool and never touches a GLM provider.
• Claude remains the failover pool and the compliance pool. This was never a full divorce.
• The split isn't only about compliance. Frontier models hold the core paths, where precision and instruction-following dominate; GLM takes the high-volume leaf work, where good enough is genuinely good enough. Model choice is a per-task policy now, not an application setting.
• For our single largest code-review customer, daily cost dropped 68% the day after the switch, with sentiment holding within our acceptance band.
• Provider selection inside the GLM pool is adaptive, continuously-sampling to determine the best provider on cost/reliability/speed dimensions

The evaluation harness has become permanent infrastructure. We continuously run candidate models through the same blind-sentiment pipeline, which is how we caught that a fashionable GPT-based alternative was far worse than either incumbent before any customer saw it, and it's how we'll judge GLM 5.3 and whatever comes after. Model choice stopped being a decision we made once and became a system we operate.

What we'd tell anyone considering this#

1. Measure cost per task, not cost per token. Per-token math promised 95% savings; per-task reality delivered 68%.
2. Blind, production-grade evaluation beats benchmarks. Sentiment on real work, attributed per model, settled arguments that benchmark charts never could.
3. Budget serious engineering time for the "compatible" API surface: reasoning formats, cache keys, structured output, usage accounting. Each provider differs, and the failures are silent.
4. Treat providers as a pool with circuit breakers and priority routing from day one. You will need it in the first month.
5. Keep a frontier model wired in as fallback and for compliance-constrained customers. Partial migration is a feature, not a compromise.
6. Loop budgets encode a model's temperament. Swap the model and the budgets are wrong.

The frontier models are still better. Opus wins our precision measurements, makes fewer mistakes per task, and needs less babysitting. But for well-instrumented agent loops with tight evaluation feedback, the quality gap is now small enough, and the cost gap large enough, that defaulting to frontier models everywhere is hard to justify. The work is in the measurement and the serving infrastructure. That work turned out to be the actual migration.