用 Jules 衡量真正重要的东西
英文标题：Measuring What Matters with Jules
AI 编程智能体正从被动完成任务转向主动洞察目标。Google 团队认为，评估主动型智能体需关注其“洞察策略”——判断什么重要、是否通知开发者。他们利用真实缺陷修复历史构建评估基准，通过时间邻近和语义相似性对错误进行聚类，还原代码状态让智能体探索。初步结果显示，核心诊断逻辑有效，且增加探索轮次能显著提升复杂问题的命中率。这项工作为开发者提供了一种衡量智能体能否在目标层面提供价值的新方法，并计划扩展至公开 GitHub 数据。
11.7 KB
1.0 MB
2.2 MB
中文
AI 编程智能体正在迅速从按需完成任务的被动助手，转变为持续吸收上下文、发现新风险并在开发者提出要求之前就给出诊断洞察的主动引擎。这一演变的核心，是从定义明确的任务向目标的转变，这要求智能体主动探索代码库，发现相关之处，并给出有助于引导开发者实现更高层次目标的诊断性观察。
像 SWE-Bench 这样的公开基准测试，衡量的是智能体完成具体任务（如修复一个定义狭窄的缺陷）的能力。但目前还没有面向目标的基准测试。在我们最新的论文《主动型编码需要主动性，而非仅仅是自主性》中，我们提出，评价主动型智能体必须看其洞察策略——即判断什么重要、用什么证据支持、以及是打断开发者还是保持沉默的能力。
上图展示了一个主动型智能编码引擎的设计。上下文信息流入引擎，引擎维护着开发状态和开发者模型，发出洞察（通知、提问、草拟方案、保持沉默），并从响应中学习。
将真实缺陷修复作为“真实依据”
基于我们在 Google Labs 对持续 AI 系统的研究，我们发现，要构建能够评价主动型智能体洞察策略的评估体系，必须先建立一个“真实依据”。一种建立这种“真实依据”的方法，是沿着两个我们称之为时间邻近性和语义相似性的启发式原则，分析团队的真实缺陷修复历史。
我们的假设很简单：当工程师在短时间窗口内提交并修复多个相关缺陷时，这些缺陷通常是一个底层工程工作的症状。一系列围绕“沙箱超时错误”、“代理配置失败”和“网络隔离不稳定测试”的缺陷，都指向一个共同的、期望达成的目标，比如_“增强沙箱执行可靠性”_。单独看，每个缺陷都过于具体于任务，无法作为目标。而将它们聚在一起，就揭示了更高层次的目标。
构建并测试我们的初步评估集
为了构建初步的基准测试并验证我们的假设，我们使用了来自 Google 内部代码库的 705 个缺陷（涉及 1178 个变更列表），步骤如下：
- 对历史上相关的缺陷进行聚类，揭示开发者当时实际上正在追求的更高层次的“期望目标”。
- 将每个聚类中的各个缺陷设为“真实依据”目标，并将代码库恢复至修复前的精确状态，让智能体从人类工程师当时的状态开始。
- 允许智能体在生成最终洞察之前，进行最多三轮代码库探索（即其“探索预算”，记为 N）。
- 使用一个 LLM 作为评委，将智能体预测的洞察与我们的“真实依据”目标进行打分，范围从 1（无关）到 5（精确匹配）。
- 通过追踪智能体的平均最高分和它产生高度精确匹配的频率（Hit@K）来衡量成功率。
初步结果与经验教训
我们的测试初步结果令人兴奋，主要有两个原因。
核心诊断逻辑有效： 仅需一轮探索，智能体就能持续给出高度相关的洞察（平均得分 4.5 分，满分 5 分）。它能成功捕捉到简单工程问题的主要信号。
探索预算很重要： 复杂、多方面的问题自然更难，但为智能体提供更多调查资源是值得的。通过将探索预算从两轮增加到三轮，智能体的 Hit@5 准确率（即正确的诊断洞察出现在其前 5 推荐中的比例）从 33% 显著回升至 57%。这证明，额外的探索轮次直接帮助智能体发现了它最初遗漏的次要信号。
下一步
这些是基于初步样本得出的初步结果，我们正在多个方面积极扩展覆盖面。首先，我们计划将这一评估方法扩展到公开的 GitHub 数据（issue 和对应的 PR），以便更广泛的 AI 社区都能应用这套方法。此外，我们还在探索如何纳入更丰富的上下文流，不仅仅是代码库，还包括 issue 跟踪器、对话和设计文档等。
在此阅读完整论文，如果你有兴趣了解我们在 Google Labs 关于编程未来的更多工作，请访问 labs.google/code 关注我们。
English / 原文
AI coding agents are rapidly shifting from reactive assistants that complete tasks when prompted to proactive engines that continuously absorb context, spot emerging risks, and surface diagnostic insights before developers have to ask. At the center of this evolution is a shift from well-defined tasks to goals , which require the agent to explore the codebase, discover what is relevant, and surface diagnostic observations that help guide developers toward a higher-level objective.
Public benchmarks like SWE-Bench test an agent’s ability to complete tasks, like fixing a narrowly defined bug, but no benchmarks currently exist for goals. In our most recent paper, Agentic Coding Needs Proactivity, Not Just Autonomy, we argue that proactive agents must be graded on their insight policy—the ability to decide what matters, what evidence supports it, and whether to interrupt the developer or stay silent.
The Figure above shows the design of a proactive agentic coding engine. Context streams into an engine that maintains development state and a developer model, emits insights (notify, question, draft, stay silent), and learns from response.
Leveraging real bug fixes as “ground truth”
Based on our work on continuous AI systems at Google Labs, we’ve found that building evaluations capable of grading a proactive agent on its insight policy requires establishing a “ground truth.” One way to build this “ground truth” is to analyze a team’s real bug-fixing history along two heuristics we term temporal proximity and semantic similarity.
Our hypothesis is simple: when engineers file and fix several related bugs within a short time period, those bugs are often symptoms of a single underlying engineering effort. A cluster of bugs around "sandbox timeout errors," "broker config failures," and "network isolation flaky tests" all point toward a common aspirational goal like "Strengthen sandbox execution reliability." Individually, each bug is too task-specific to serve as a goal. Together, they reveal the higher-level objective.
Building and testing our preliminary eval set
To build our preliminary benchmark and test our hypothesis, we used 705 bugs (1,178 CLs) from internal Google codebases to:
- Cluster related historical bugs to reveal the higher-level "aspirational goals" developers were actually working toward.
- Set the individual bugs within each cluster as our "ground truth" targets and reverted the codebase to its exact pre-fix state so the agent began where the human engineer did.
- Allow the agent to investigate the codebase for up to three rounds (its "exploration budget," or N) before generating its final insights.
- Use an LLM to judge the agent’s predicted insights from 1 (irrelevant) to 5 (exact match) against our “ground truth” targets.
- Measure success by tracking the agent's average top score and how often it successfully produced a highly accurate match (Hit@K).
Preliminary results and what we learned
The preliminary results of our testing are exciting for two primary reasons.
The core diagnostic logic works: Given a single exploration round, the agent consistently identified a highly relevant insight (averaging 4.5 out of 5). It successfully captured the primary signal for straightforward engineering problems.
Exploration budgets matter: Complex, multi-faceted problems are naturally harder, but giving the agent more resources to investigate pays off. By increasing the exploration budget from two rounds to three, the agent’s Hit@5 accuracy (defined as the rate at which a correct diagnostic insight appears within its top 5 recommendations) rebounded significantly from 33% to 57%. This proves that extra passes directly help the agent uncover secondary signals it initially missed.
What’s next
These are preliminary results on an initial sample, and we are actively expanding coverage on multiple fronts. To start, we are expanding this evaluation to public GitHub data (issues and resolving PRs) to make this methodology broadly applicable to the wider AI community. We are also exploring how to ingest richer context streams like issue trackers, conversations, and design documents beyond just the codebase.
Read the full paper here and follow along with us at labs.google/code if you’re interested in learning more about our work on the future of coding at Google Labs.