摘要：一次购物对话可以通过多种路径达成同一个购物车，而任务成功率会将这些不同路径简化为单一分数。我们构建了一个确定性且可复现的电商环境，为每次试验预先设定顾客及交互轨迹参数，包括用户画像、难度、目标购物车和商品披露时间表。一名模拟消费者在被评估模型的协助下，尝试从该环境中购买目标购物车中的商品。该环境会引导模拟器的行为，并记录每一次助手操作及其发生时对应的环境状态。试验结束后，这些记录使评估者能够依据保留的证据，对对话的各个环节进行评估。例如，只有当顾客已经提及某一目标商品时，评估者才会对未能展示该目标商品的搜索行为进行扣分。我们还利用这些证据，根据助手操作与预期工具调用集合之间的差异，对工具调用施加不同的惩罚。我们的环境还会与模拟器进行双向交互：读取其输出，并在模拟器判定顾客已变得过于沮丧时停止试验；同时实时注入指令，规定何时进行探索、暂缓购买某件商品，或回顾先前的交流内容。这种交互形成了一种开放式且可验证的模拟。在对 8 个参数规模介于 200 亿至 350 亿的开放权重 Agent 进行测试时，我们为每个 Agent 进行了 160 次试验，并采用 44 项指标。所得能力画像能够区分行动不足、过度购买、缺乏依据的商品属性描述和搜索能力不佳等问题，而这些问题都会被最终成功与否的指标所掩盖。

Abstract:A shopping conversation has many routes to the same cart, and a task-success rate reduces all of them to one score. We build a deterministic and reproducible e-commerce environment that precommits each trial's customer and trajectory parameters, including the persona, difficulty, target cart, and an item reveal schedule. A simulated consumer attempts to buy a target cart from the environment with assistance from the evaluated model. The environment guides the simulator's actions and records every assistant action alongside the environment state at that point. After the trial, these records allow the evaluator to assess individual parts of the conversation against the retained evidence. For example, the evaluator penalizes a search for failing to surface a target product only when the customer has already mentioned that product. We further use this evidence to apply different penalties to tool calls depending on how the assistant's actions compare with an expected tool-call set. Our environment also interacts with the simulator bidirectionally, reading its output to stop the trial when the simulator determines that the customer has become too frustrated and injecting directives in real time that specify when to explore, defer buying an item, or recall a previous exchange. This interaction creates an open-ended and verifiable simulation. Across eight open-weight agents from 20B to 35B parameters, with 160 trials per agent and 44 metrics, the resulting capability profiles distinguish under-action, over-purchase, unsupported product attributes, and poor search, all of which terminal success obscures.