🔶 超越排行榜：真实工作流中的决策模型

尝试在哪些环节失败

上方动画展示了应用程序工作流。桑基图展示了同一组88个场景在各个评估检查点的情况：候选项创建、需求断言和路由检查。每次被评分的尝试都会在首次未通过的检查处退出；这些退出并不代表已测量的生产环境交接。

“需求匹配”是指通过针对范围、条件、日期和排序的显式断言。这些检查会在最终路由器之前运行；不会由另一个模型来评判答案。

在一项已记录的 OpenAI 失败案例中（此处为转述），请求要求统计一个三个月时间窗口内的月度总计。候选项保留了分组方式，却遗漏了日期限制，尽管其置信度为 0.60。它在最终路由器运行之前就未通过了语义检查。

超过 0.50 并不意味着该答案就是正确的。需求检查捕捉到了缺失的限制条件。

TypeSafe Jev 1.13

Jev 1.13 通过了全部264次可评分尝试，以及全部57项安全/边缘检查。中间置信度分数虽有变化，但并未改变可评分场景中的结果。

TypeSafe Jev 1.14（预览版）

Jev 1.14 在264次支持的尝试中通过了247次。14次低于置信度门槛，另有3次无法生成有效候选项。数值分桶和实体边界场景在3次运行中均发生回退。在单独进行的安全与边缘检查中，它每次运行均通过19项中的18项：重复出现的失败是遗漏了请求要求的一个分组。额外回退与需求缺失属于不同的失败模式；一次失败的尝试并不必然意味着错误答案到达了用户。

我们撰写本文时，Jev 1.14 仍处于预览阶段。我们希望这些回归问题能在其结束预览前得到解决，并会在届时重新运行该测试套件。

OpenAI Decisions

在可评分尝试中，OpenAI 有48次未生成候选项。12个候选项遗漏了条件或排序等需求；其中9个的置信度达到或超过0.50。最终路由器又因低置信度拒绝了24个候选项，最终留下180次通过。

相同的60个可评分场景在每次运行中均通过，匹配的请求返回了完全相同的记录分布。错误也同样可以重复出现。降低门槛无法修复遗漏需求的问题。

Microsoft Decision 1

在可评分尝试中，Microsoft 有57次未生成候选项。另有32次因遗漏或额外添加条件、或范围不正确而未通过语义检查；其中21次的置信度达到或超过0.50。最终路由器又因低置信度拒绝了11次，最终留下164次通过。

总通过次数分别为55次、54次和55次，彼此接近，但有10个场景在通过与失败之间发生了变化。对于相同的请求，分数和选择也发生了变化。这些追踪记录并不能确定其原因。

同时存在语义错误和低置信度的候选项，只会在语义检查环节计数，因为测试会在该环节停止。

Where attempts failed

The animation above shows the application workflow. The Sankeys show evaluation checkpoints for the same 88 scenarios: candidate creation, requirement assertions, and routing checks. Each scored attempt exits at its first failed check; these exits are not measured production handoffs.

“Requirements matched” means passing explicit assertions about scope, conditions, dates, and ordering. These checks run before the final router; another model does not judge the answer.

In one recorded OpenAI failure, paraphrased here, the request asked for monthly totals within a three-month window. The candidate kept the grouping but omitted the date restriction, despite confidence of 0.60. It failed the meaning check before the final router ran.

Clearing 0.50 did not make that answer correct. The requirement check caught the missing restriction.

TypeSafe Jev 1.13

Jev 1.13 passed all 264 scorable attempts and all 57 safety/edge checks. Intermediate confidence scores varied without changing outcomes on the scorable scenarios.

TypeSafe Jev 1.14 (preview)

Jev 1.14 passed 247 of 264 supported attempts. Fourteen fell below the confidence gate and three could not produce a valid candidate. The numeric-bucket and entity-boundary scenarios fell back in all three runs. In the separate safety and edge checks, it passed 18 of 19 each run: the repeated failure omitted a requested grouping. Extra fallbacks and missing requirements are different failure modes; a failed attempt does not necessarily mean a wrong answer reached the user.

Jev 1.14 was in preview when we wrote this post. We hope these regressions are addressed before it leaves preview and will rerun the suite when it does.

OpenAI Decisions

Among scorable attempts, OpenAI produced no candidate in 48 attempts. Twelve candidates missed requirements such as conditions or ordering; nine had confidence at or above 0.50. The final router rejected another 24 for low confidence, leaving 180 passes.

The same 60 scorable scenarios passed each run, and matched requests returned identical recorded distributions. The mistakes were repeatable too. Lowering the gate would not fix missing requirements.

Microsoft Decision 1

Among scorable attempts, Microsoft produced no candidate in 57 attempts. Another 32 failed meaning checks through missing or extra conditions or incorrect scope; 21 had confidence at or above 0.50. The final router rejected another 11 for low confidence, leaving 164 passes.

Totals were close at 55, 54, and 55 passes, but ten scenarios changed between pass and fail. Scores and choices also changed for identical requests. The traces do not establish the cause.

Candidates with both incorrect meaning and low confidence count only at the meaning check, where the test stopped.

🔶 超越排行榜：真实工作流中的决策模型

中文

尝试在哪些环节失败

TypeSafe Jev 1.13

TypeSafe Jev 1.14（预览版）

OpenAI Decisions

Microsoft Decision 1

English / 原文

Where attempts failed

TypeSafe Jev 1.13

TypeSafe Jev 1.14 (preview)

OpenAI Decisions

Microsoft Decision 1