我们的 AI Agent 把同一条成功的 SQL 查询执行了 254 次

在 Funnel，我们一直在尝试使用不同的语言模型来驱动我们内部的数据 Agent 平台 Reef。

Reef 运作在三层公司知识体系之上——以图谱形式表示的公司世界模型（company world model）、包含精细建模的财务、SaaS 和 CRM 指标的结构化运营数据，以及我们从中提取结构化主张（claims）的非结构化客户对话。对于一个 Agent 而言，这是一个相当复杂的探索环境。

最近，我们进行了一次随机 A/B 测试，对比了开源权重模型 GLM-5.2 与 Claude Opus 4.8。每个新的用户任务都会随机分配一个模型，并在随后的交互中保持不变，包括更深入的调研以及创建我们称之为探索（explorations）的交互式仪表盘。

初步结果看起来很有前景——在我们的观察窗口期内，GLM 平均每次模型调用耗时 7.8 秒，而 Opus 为 18.1 秒——速度快了一倍以上。

但当我们审视端到端任务执行情况时，呈现出的却是另一番景象。GLM 的工具调用次数多出了 66%，平均为 19 次，而 Opus 为 11 次。尽管单次响应更快，GLM 完成这些任务的平均耗时却反而增加了 14%。

在两次独立的运行中，GLM 把同一条成功的 SQL 查询分别执行了 113 次和 254 次。

在其中一个案例中，模型试图从一个仅返回 20 行预览的工具中获取更多数据。它没有意识到这种限制，而是反复执行相同的查询。每一次调用都成功了，但 Agent 显然没有任何实质性进展。一次成功的工具调用并不一定是一次有用的工具调用。

我们在 Kimi K2.6 和 Qwen3.8 27B 上也观察到了类似倾向；与 GLM 相比，Qwen 在工具调用上激进了一倍。

现在，人们很容易归咎于模型本身，但这忽略了全局。我们保存的部分 Agent 指令引用了实际上并不存在的 Athena 表和字段。其他问题则暴露了模糊的工具契约（tool contracts）、缺乏防重复调用的保护机制，或数据检索指令不够清晰。Opus 能够克服其中的大部分问题，但无论如何，这些依然是系统缺陷。

Agent 运行框架（Agent harness）与模型本身的智能同等重要。

不同的模型对相同的指令、工具和终止条件的理解各不相同。更换模型可能会暴露先前隐蔽的架构假设。

对于正在将开源权重模型集成到现有 Agent 平台的工程师，我建议做好以下三点：

1. 明确工具契约（Make tool contracts explicit）。 明确区分预览与完整结果，清晰地呈现分页机制，并阐明何为真正意义上的“成功响应”。
2. 约束 Agent 行为（Bound agent behaviour）。 检测重复的等价调用，实施合理的执行预算，并对照真实 Schema 校验保存的指令。
3. 对整个系统进行基准测试（Benchmark the whole system）。 衡量有价值的任务完成度、正确性、延迟、工具效率、计费成本以及输入缓存（input caching）的效果。

开源权重模型在成本、性能和控制力方面提供了令人兴奋的机遇。但要真正兑现这些收益，往往不仅仅是替换一个 API 端点那么简单。

又及：GLM-5.2 现在表现得很顺畅（事实上，我们现在用的是 GLM-5.3）。

At Funnel, we've been experimenting with different language models powering Reef , our internal data agent platform.

Reef operates across three layers of company knowledge - a company world model represented as a graph, structured operational data containing carefully modelled financial, SaaS and CRM metrics, and unstructured customer conversations from which we extract structured claims. It's a fairly sophisticated environment for an agent to navigate.

Recently, we ran a randomised A/B test comparing GLM-5.2, an open-weight model, against Claude Opus 4.8. Each new user task was randomly assigned a model, which remained the same throughout subsequent interactions, including deeper investigations and the creation of interactive dashboards we call explorations.

The initial results looked promising - across our observation window, GLM averaged 7.8 seconds per model call, compared to 18.1 seconds for Opus - more than twice as fast.

But when we looked at end-to-end task execution, a different picture emerged. GLM made 66% more tool calls, averaging 19 compared to Opus's 11. Despite faster individual responses, GLM took 14% longer to complete those tasks on average.

In two separate runs, GLM executed the same successful SQL query 113 and 254 times.

In one case, the model was attempting to retrieve more data from a tool that only exposed a 20-row preview. Rather than recognising the limitation, it repeatedly executed the same query. Every call succeeded but the agent obviously wasn't making any progress. A successful tool call is not necessarily a useful tool call.

We've observed similar tendencies with Kimi K2.6 and Qwen3.8 27B; Qwen is twice as aggressive on tool usage compared to GLM.

Now, it's tempting to blame the model, but that's missing the big picture. Some of our saved agent instructions referenced Athena tables and fields that didn't actually exist. Other problems revealed ambiguous tool contracts, missing safeguards against repetitive calls, or insufficiently explicit instructions about data retrieval. Opus was able to overcome most of these issues, but they are issues nonetheless.

Agent harnesses matter as much as model intelligence.

Different models interpret the same instructions, tools, and stopping conditions differently. Replacing a model can expose architectural assumptions that were previously invisible.

For engineers integrating open-weight models into existing agentic platforms, I'd suggest three things:

1. Make tool contracts explicit. Distinguish previews from complete results, expose pagination clearly, and communicate what a successful response actually means.
2. Bound agent behaviour. Detect repeated equivalent calls, enforce sensible execution budgets, and validate saved instructions against real schemas.
3. Benchmark the whole system. Measure useful task completion, correctness, latency, tool efficiency, billed costs and the effects of input caching.

Open-weight models offer exciting opportunities for cost, performance and control. Realising those benefits may require more than swapping out an API endpoint.

PS. GLM-5.2 is a happy little camper now (actually, we now use GLM-5.3).