AI News
商业产业 Hacker News AI 2026-09-24T00:10 正文 ok 1103 words

在智能体式 CAD 测试框架中对比 Sol 6 与 Opus 5.5

英文标题：🔶 Sol 6 and Opus 5.5 compared on an agentic CAD harness

Anthropic 发布 Opus 5.5 后数小时，OpenAI 也推出 GPT-6 Sol 与 GPT-6 Luna，Partforge 随即在 26 项高难度 CAD 建模任务上进行了评测。结果显示，Opus 5.5 相比 Opus 5 单任务成本约降低 40%、模型耗时缩短至约三分之一，并以微弱优势取得本轮最高综合分。GPT-6 Sol 的单任务成本基本持平，但通过更多推理和操作步骤带来了略好的建模结果，成功构建并运行的比例维持在 92%。不过，Partforge 仍将 Gemini 3.8 Flash 作为默认模型，原因在于其单个成功任务成本更低、整体可靠性更高，并且持续展示工作过程带来了更好的交互体验。
打开原文相关 3 · 机会 0
media64.8 KB media62.7 KB media35.2 KB media38.1 KB media65.4 KB media59.5 KB media68.5 KB media53.5 KB media31.9 KB media3.5 KB

中文

LLM CAD 测试框架评测

Anthropic 昨天发布了 Opus 5.5。几小时后，OpenAI 又发布了 GPT-6 Sol 和 GPT-6 Luna。又到了令人兴奋的“新模型日”——两家前沿实验室，在同一个下午。

我们长期维护着一组用于 partforge 的高难度 CAD 任务，并会让每个模型都跑一遍，因此昨晚就启动了测试。以下是返回的结果。

简短结论是：Opus 5.5 比此前的 Opus 显著更快、更便宜；GPT-6 Sol 在相同预算下提供了更多能力；Opus 5.5 是目前最接近击败我们默认模型的产品；不过，我们仍会继续将 Gemini 3.8 Flash 作为默认模型。

我们实际测试了什么

本次评测包含 26 项高难度建模任务，例如带沉孔的轴承座，或仅根据一张照片描述的零件。每个模型会对每项任务尝试三次，每次尝试都会通过三种方式检查：几何尺寸是否正确、模型为得到结果所采取的方法是否合理，以及由一对独立评审模型判断其外观是否正确。随后，我会人工逐一审查结果、查看渲染图，并在全部 26 项任务中对模型进行相互排名。

我们有意将测试集保持在较小规模。简单任务在数月前就已经无法提供有价值的信息了，所有前沿模型每次都能通过这些任务。

你可以在 partforge.ai/evals 查看完整测试过程，包括每一次尝试和每张渲染图。

Opus 5.5 有了显著提升

这是最重要的结论。在相同任务上，与两周前的 Opus 5 相比：

  • 每项任务的成本约低 40%。
  • 耗时约为原来的三分之一。Opus 5 的平均模型耗时为每项任务四分半钟，Opus 5.5 则平均为一分半钟。

Anthropic 还将标价下调了约 20%，因此节省的一部分来自价格本身，其余则是因为 Opus 5.5 为得到答案所进行的无效反复明显更少。

它也在本轮测试中取得了最佳综合得分，以微弱优势领先 Gemini。它并非我们在这些任务上测得的历史最佳模型——Fable 5.1 两周前的得分更高，但没有参加本轮测试——但它是我们会认真考虑设为默认模型的模型中表现最强的一个。

我们较难的一项评测要求模型根据一张复杂照片生成电机辐条组件，而照片中只有半个电机进入画面。在我们测试过的所有模型中，Opus 5.5 目前最接近真实几何形状。

GPT-6 Sol 悄然带来了更高性价比

Sol 很有意思，因为从表面看，几乎没有什么改变。

OpenAI 将输入价格下调了约 20%，输出价格下调了约三分之一。但 GPT-6 Sol 在每项任务中完成的工作也比 GPT-5.6 Sol 更多。它的读取量大约增加了 50%，会额外执行一两步，并在遇到困难任务时投入更多思考。

这两方面几乎恰好相互抵消。旧版 Sol 的一项高难度任务成本为 25.7 美分，新版为 25.2 美分。可以认为基本持平。

而同样约 25 美分带来的，是一个略好一些的零件。得分小幅上升，能够顺利构建并运行的尝试比例则稳定在 92%。因此，诚实的总结不是“更便宜”，而是“用同样的钱获得更多模型能力”；而这通常才是更好的取舍。

还有一个值得注意的背景：在同样的两周内，完全没有变化的 Gemini，由于我们的测试集和提示词围绕它发生了调整，在这些完全相同的任务上成本上升了 16%。在这一时间窗口内保持成本不变，其实比听起来更好。

那么，为什么我们仍在使用 Gemini？

成本。 我们关注的指标不是每项任务的价格，而是实际正确完成的每项任务的价格。Gemini 3.8 Flash 每个成功任务的成本约为 0.34 美元，GPT-6 Sol 约为 0.48 美元，Opus 5.5 则约为 0.72 美元，略高于 Gemini 的两倍。在迭代一个零件时，你不会只运行一次提示词，而是会运行 15 次，这一差距会迅速累积。

它仍然最可靠。 Opus 5.5 赢得了我们的综合得分，但 Gemini 在日常最重要的两项指标中胜出：零件在完全没有问题的情况下通过每一项检查的比例（64% 对 58%），以及能够成功构建并运行的比例。在我的人工排名中，两者差距极小，Gemini 略微领先。Opus 5.5 已经足够接近，下个月结果可能会反转，但现在还没有。

而且，即使实际上并非如此，它感觉更快。 这一点让我们感到意外。Opus 5.5 完成一项任务的总耗时比 Gemini 更短，但使用时仍然感觉更慢。

Gemini 会以大量小步骤工作：每项任务约 8 步，每步平均 17 秒。Opus 则以约 4 个较大的步骤工作，每步平均 23 秒，最慢的步骤会超过两分钟。更糟的是，Anthropic 的模型在这些时间段内经常完全不向我们发送可见的思考过程，因此没有任何内容可以展示给用户。你会盯着静止的屏幕看两分钟，然后零件突然一次性改变。我们可以用一个动态星号改善这一点，但我们认为，展示真实的思考过程以及有关当前状态的线索，能为用户体验带来真正的好处。

使用 Gemini 时，屏幕上的内容每 15 到 20 秒就会发生变化。你可以看到它在工作，也能及早发现它何时误解了你的意图并将其停止。一个在纸面指标上更快、但工作时保持沉默的模型，并不会让人感觉更快，反而会损害使用体验。

亲自试用

这不必由我们替你做决定。本文提到的所有模型都可以在 partforge 中选择，包括 Opus 5.5 以及两款新的 GPT-6 模型。在设置中开启高级模型，打开聊天框旁的模型选择器即可切换。

如果你使用自己的 Anthropic、OpenAI 或 OpenRouter API key，这些运行将使用你的 key 并计入你的账单，我们完全不会对它们进行计量。

下次有人发布新东西时，我们会再次运行这项测试。按照目前的发布速度来看，这件事可能已经发生了。

关于上述数字，有一点必须坦诚说明：我们对比的两轮测试相隔两周，而且两次测试之间我们自己的测试框架发生了变化。完全没有变化的 Gemini 在第二轮测试中成本高了 16%，速度也更慢。因此，请将前后对比数据视为总体趋势，而不是精确测量值。新一轮测试内部的正面对比数据则可以直接比较。

English / 原文

LLM CAD Harness Evals

Anthropic released Opus 5.5 yesterday. A few hours later OpenAI released GPT-6 Sol and GPT-6 Luna. Yay 'new model day!' Two frontier labs, one afternoon.

We have a standing set of hard CAD tasks we run every model through for partforge, so we started a run yesterday evening. This is what came back.

The short version: Opus 5.5 is dramatically faster and cheaper than the Opus before it, GPT-6 Sol gives you more for the same money, Opus 5.5 is the closest anything has come to beating our default, and we are still shipping Gemini 3.8 Flash as the default anyway.

What we actually test

This eval was 26 hard modeling tasks. Things like a bearing seat with a counterbore, or a part described only by a photo. Each model tries each task three times, and every attempt is checked three ways: does the geometry measure correctly, did the model work sensibly to get there, and does it look right to a pair of independent reviewer models. Then a human (I) go through the results by hand, look at the renders, and rank the models against each other on every one of the 26 tasks.

It is a small set on purpose. Easy tasks stopped telling us anything useful months ago. All the frontier models pass them every time.

You can read the whole run, every attempt and every render, at partforge.ai/evals.

Opus 5.5 got dramatically better

This is the headline. On the same tasks, compared to Opus 5 two weeks ago:

  • About 40% cheaper per task.
  • Roughly a third of the time. Opus 5 averaged four and a half minutes of model time per task. Opus 5.5 averaged a minute and a half.

Anthropic also cut the list price about 20%, so some of that is the price tag and the rest is Opus 5.5 simply doing less thrashing to get to an answer.

It also posted the best overall score in this run, narrowly ahead of Gemini. It is not the best we have ever measured on these tasks (Fable 5.1 scored higher two weeks ago, and was not in this run), but it is the strongest showing from a model we would consider making the default.

One of our harder evals is one that asks the model to generate a motor spoke assembly based on complicated photo with only half the motor in the frame. Opus 5.5 got the closest to the actual geometry yet in all the models we've tested.

GPT-6 Sol quietly got better value

Sol is the interesting one, because on the surface almost nothing changed.

OpenAI cut the price about 20% on the way in and a third on the way out. But GPT-6 Sol also does more work per task than GPT-5.6 Sol did. It reads about half as much again, takes an extra step or two, and thinks a little harder on the tasks it finds difficult.

Those two cancel out almost exactly. A hard task cost us 25.7 cents on the old Sol and 25.2 cents on the new one. Call it the same.

What you get for that same quarter is a slightly better part. The score nudged up, and the share of attempts that built and ran cleanly held steady at 92%. So the honest summary is not "cheaper", it is "more model for your money", which is usually the better trade anyway.

Worth one piece of context: over the same two weeks, Gemini, which did not change at all, got 16% more expensive on these exact tasks as our test set and prompts moved around it. Holding flat in that window is better than it sounds.

So why are we still on Gemini?

Cost. The number we care about is not price per task, it is price per task that actually came out right. Gemini 3.8 Flash costs about $0.34 per solved task. GPT-6 Sol is about $0.48. Opus 5.5 is about $0.72, a bit over double Gemini. When you are iterating on a part, you do not run one prompt, you run fifteen, and that gap compounds fast.

It is still the most reliable. Opus 5.5 wins our overall score, but Gemini wins the two measures that matter most day to day: how often a part passes every single check with nothing wrong (64% versus 58%), and how often it builds and runs at all. In my own ranking pass the two were a hair apart, with Gemini barely ahead. Opus 5.5 is close enough that this could flip next month. It has not flipped yet.

And it feels faster, even when it is not. This one surprised us. Opus 5.5 finishes a task in less total time than Gemini does. It still feels slower to use.

Gemini works in lots of small steps, around eight per task, averaging 17 seconds each. Opus works in about four big ones, averaging 23 seconds each, and its slowest steps run past two minutes. Worse, Anthropic's models often send us no visible thinking at all during those stretches, so there is nothing to show you. You sit watching a still screen for two minutes and then the part changes all at once. We could improve that with an animating asterisk but we think showing real thinking and clues to what's going on is a real benefit to the user experience.

With Gemini, something on screen moves every fifteen or twenty seconds. You can see it working, and you can tell early when it has misunderstood you and stop it. A model that is quicker on paper but silent while it works does not feel quick and hurts the experience.

Try them yourself

None of this has to be our call. Every model here is selectable in partforge, including Opus 5.5 and both new GPT-6 models. Turn on advanced models in the settings, open the model picker next to the chat box and switch.

If you bring your own API key from Anthropic, OpenAI or OpenRouter, those runs go on your key and your bill, and we do not meter them at all.

We will run this again the next time somebody ships something. Which, at the current rate probably has already happened.

One honest caveat on the numbers above: the two runs we are comparing are two weeks apart, and our own test harness changed between them. Gemini, which did not change at all, came out 16% pricier and slower on the second run. So treat the before and after figures as a general trend rather than exact measurements. The head to head numbers within the new run are directly comparable.