前沿模型能否看破“诱饵”？Elorian 发布 DecoyCount 基准测试视觉推理能力

本月，Elorian 推出了其首款视觉推理模型 Glimpse。该模型被宣传为“一种全新类别的视觉思维模型：一个能够根据所见内容对物理世界进行推理的系统”。

那么，在视觉测试中，Glimpse 与前沿大模型以及世界模型（World Models）相比表现如何？Elorian 联合发布的全新基准测试 DecoyCount 提供了一些答案。

我们专访了其首席执行官 Andrew Dai，就该基准展开探讨，并聊到了 Elorian 在拥挤的模型竞争格局中的定位。

DecoyCount 测试结果

Elorian 于今年 4 月结束隐身状态，完成了由 Striker Venture Partners、Menlo Ventures 和 Altimeter Capital 联合领投的 5500 万美元种子轮融资。如今，Elorian 与 AI 数据公司 Mercor 合作推出了 DecoyCount 基准，该基准旨在测试语言模型究竟是在真实计数物体，还是依赖视觉捷径与替代性启发规则。

根据随同发布的一篇论文，前沿模型在 DecoyCount 上的准确率均低于 50%——得分最高的是 GPT-6 Astra，准确率为 43.17%。虽然论文中并未提及 Elorian 的新模型，但 Dai 告诉我们，Glimpse 在内部测试中取得了 50.48% 的准确率。

针对这些前沿模型，论文得出的结论是：“更强的通用推理能力本身并不能直接泛化为对齐的视觉推理能力。”

该基准的运作机制：“DecoyCount 揭示了跨七种机制的可解释计数失败情况。”

为了确保与前沿模型的对比具有公平性，我们询问了论文中的评估图像是否从 Glimpse 的训练数据中进行了剔除？

“评估所用的图像和文本均未参与我们的模型训练，我们的数据污染检查在已检查的训练数据中未发现任何重叠，”Dai 通过电子邮件回复道。“在没有任何提示线索的情况下，Glimpse 在 1660 个 DecoyCount 测试项上取得了 50.48% 的成绩，成为参与对比中唯一突破 50% 的模型。 表现最接近的是 GPT-6 Astra，得分 43.17%，落后约 7 个百分点；Gemini-3.8-flash 得分为 32.71%，Qwen3.8-max 得分为 28.31%，而 Claude-Fable-5.1 的成绩为 25.22%。”

Dai 确实指出，Glimpse 的结果“来自于模型自身的单次运行，未借助 Agent 或工具框架支持”，而论文中的六个模型均是在其报告设置下三次运行取的平均值。

前沿模型在 DecoyCount 基准上各运行 3 次的测试结果。

那么，DecoyCount 究竟揭示了哪些现有模型基准所遗漏的内容？Dai 表示，它将视觉与推理结合起来进行共同测试——而这正是大多数基准所忽略的。

“如果一个人看到镜子前放着一个苹果，我们知道那里只有一个苹果，另一个是镜像倒影。但模型可能会完全漏掉这个苹果（视觉问题），或者把苹果和它的倒影算成两个（推理问题）。目前大多数测试此类错误的基准通常只检查最终数字，因此这两种错误会被同等对待。如果你想解决根本问题，却分不清到底是视觉感知问题还是判断推理问题，这就会带来阻碍。”

DecoyCount 论文中的前沿模型对比

什么是视觉思维？

Andrew Dai 并不是 AI 领域的后来者。早在 2015 年在 Google 工作期间，他就合著了题为《Semi-supervised sequence learning》的论文。在这篇论文中，Dai 与当时的同事 Quoc Le 展示了后来被大语言模型广泛采用的先预训练后微调（pretrain-then-fine-tune）方法的早期版本。在随后的十年里，Dai 在 Google DeepMind 参与了 GLaM MoE、PaLM 2 以及后来的 Gemini 的研发工作。

Dai 直到今年才离开 Google 并联合创立了 Elorian。他的部分初衷是拓展视觉模型的能力边界。他向 Latent Space 解释称，他的方法建立在人类心理学中一个被称为**视觉思维（visual thinking）**的概念之上。

他解释道，当大多数人“思考空间问题或视觉问题时，他们会通过在脑海中生成图像和编辑图像来对问题进行可视化”。

他举了一个买沙发的例子：你会在脑海中构想它摆在客厅里会是什么样子。

概念性视觉推理循环

此外还有在脑海中操作图像；例如，如果你在数桌子周围坐了多少人，你可能会在脑海中“把他们一个一个划掉”，Dai 说道。

这正是 Elorian 试图在其模型中实现的目标：教会模型具备“将图像生成与编辑同语言一道，作为推理轨迹一部分”的视觉思维。

关键在于，对于视觉模型而言，图像生成不仅仅是一种输出结果——生成或编辑后的图像本身就是思维链的一部分。

Elorian 与世界模型有何不同？

视觉推理仍处于发展阶段，但今年在以视觉为核心组件的**“世界模型（world models）”**领域已经出现了一些令人瞩目的成果。其中最突出的两个代表是 World Labs 9 月发布的 Atlas（该公司随后被 AMD 收购）以及 Google 刚发布的 Genie 4。

当 World Labs 宣布推出 Atlas 时，它被描述为一种“用于空间智能的全能世界模型”。但 Atlas 所实现的能力——本质上是一个世界的 3D 模拟——与视觉推理并不相同。

根据 Dai 的说法，世界模型实现的是“在世界中的直接行动”，而 Elorian 则更侧重于他所谓的“间接行动”。他的意思是利用视觉推理为决策或工作流提供信息（Glimpse 的作用），而不是构建支持导航与物理交互的模拟环境（Atlas 的作用）。

举个例子，你可能会利用视觉推理来判断是否应当因违规而驳回用户上传的视频。“要做到这一点，你并不需要 3D 渲染，”Dai 说。

视觉推理的应用场景

Elorian 的其他商业机会包括机器人技术、工程及防务领域的应用。Dai 补充道，建筑师需要定期审查建筑规范和合规性——这正是视觉推理模型的理想任务。

在两个机器人基准测试 WGO Bench 和 ERQA 中，Elorian 声称其表现达到或超过了前沿模型，且使用成本显著更低。

回到将计算诱饵物体作为衡量视觉模型成功标准的话题上，Dai 认为，对模型所见之物进行推理，对其在现实世界应用中的准确性与有效性至关重要。

他以仓库机器人清点箱内物品为例说明了这一点。

“一个常见的 12 格托盘可能只装了 10 个零件，而一个标注 24 件装的标签并不意味着箱子里还剩 24 件物品，”Dai 说道。“如果一个模型依赖于它‘预期看到的内容’，它在现实环境中就会遇到许多问题；而 Glimpse 旨在逐一解析图像，而不是草率得出结论。我们设计 Glimpse 的目的，就是让计数建立在可见证据之上，而不是依赖那些容易让其他模型误入歧途的基础线索。”

为何 Elorian 认为 AGI 离不开视觉推理

Elorian 甚至认为，视觉推理是实现 AGI（通用人工智能）的基础要求。正如 Glimpse 发布博文中所写：

“我们相信，从底层出发，跨系统、数据、模型架构和训练全栈构建原生多模态智能，能够解锁物理世界中的 AGI。”

Dai 承认“人类与机器之间仍然存在巨大鸿沟”。但他认为视觉推理是弥合这一鸿沟的关键。

**“我认为，应当将视觉推理视为人类智能推理过程中的一个根本步骤，”**他说道。“归根结底，如果我们想打造出类似于 AGI 的系统，我们就绝不能绕过视觉。”

This month Elorian launched its first visual reasoning model , Glimpse. It’s being promoted as “a new category of visual thinking model: a system that reasons about the physical world from what it sees.”

So how does Glimpse compare to both frontier models and world models in vision tests?A new benchmark called DecoyCount , co-released by Elorian,**** provides some answers.

We quizzed CEO Andrew Dai on the benchmark, and also talked about Elorian’s positioning in a crowded models landscape.

DecoyCount results

Elorian emerged from stealth in April with $55 million in seed funding, co-led by Striker Venture Partners, Menlo Ventures and Altimeter Capital.**** Now, in partnership with AI data company Mercor, Elorian is announcing the DecoyCount benchmark, which tests whether language models genuinely count objects or rely on visual shortcuts and substitution heuristics.

According to a paper accompanying the release, frontier models scored below 50% accuracy on DecoyCount — the highest was GPT-6 Astra, with 43.17%. Elorian’s new model isn’t mentioned in the paper, but Dai told us that Glimpse scored 50.48% in an internal test.

Regarding the frontier models, the paper concludes that “stronger general reasoning does not by itself generalize to aligned visual reasoning.”

How the benchmark works: “DecoyCount exposes interpretable counting failures across seven mechanisms.”

To make sure it was a like-for-like comparison with the frontier models, we asked whether the paper’s evaluation images were withheld from Glimpse’s training?

“The evaluation images and text were withheld from our training, and our contamination check found no overlap in the training data checked,” Dai replied, via email. “Without any cues, Glimpse scored 50.48% on the 1,660 DecoyCount items, making it the only model in the comparison to clear 50%. The closest was GPT-6 Astra at 43.17%, about 7 points behind, and Gemini-3.8-flash scored 32.71% while Qwen3.8-max scored 28.31%, and Claude-Fable-5.1 came in at 25.22%.”

Dai did note that Glimpse’s results “come from a single run of the model on its own, without an agent or tool harness,” whereas the six models from the paper were each averaged across three runs under their reported settings.

DecoyCount benchmark results for frontier models over 3 runs each.

But what exactly is DecoyCount revealing that existing model benchmarks miss? According to Dai, it tests both vision andreasoning together — which most benchmarks elide.

“If a person sees an apple in front of a mirror, we know that there’s just one apple and it’s being reflected. But a model might miss the apple entirely (a vision problem) or count the apple and its reflection as two (a reasoning problem). But most benchmarks that test for this error only check the final number, so both mistakes are treated the same way, which is an issue if you want to fix the root problem and can’t tell if it’s an issue of vision or judgement. ”

Frontier Model Comparison in DecoyCount paper

What is visual thinking exactly?

Andrew Dai isn’t a Johnny-come-lately on the AI scene. In 2015, while working at Google, he co-authored a paper entitled Semi-supervised sequence learning. In this paper, Dai and his then colleague Quoc Le demonstrated an early version of the pretrain-then-fine-tune approach later adopted in large language models. Over the decade that followed, Dai worked for Google DeepMind on GLaM MoE, PaLM 2 and then Gemini.

Dai only left Google this year, in order to co-found Elorian. Part of his motivation was to push the boundaries of what vision models can do. His approach, he explained to Latent Space, is based on a concept in human psychology called visual thinking.

When most people “think about spatial problems or visual problems,” he explained, “they visualize the problem [by] generating images and editing images in their mind.”

He gave an example of buying a sofa and mentally picturing what it will look like in your living room.

A conceptual visual reasoning loop

There’s also manipulating an image in your mind ; for example if you’re counting people around a table, you might mentally “cross them off one by one,” Dai said.

This is what Elorian aims to do in its models: teach them visual thinking “that involves generation and editing images as part of the reasoning trace, along with language.”

The key point is that image generation isn’t just an output for a vision model — the generated or edited image is part of a chain of reasoning.

How is Elorian different from World Models?

Visual reasoning is still a work in progress, but there have been some impressive releases this year in “world models” — where vision is a key component. Two that stand out are World Labs’ Atlas in September (the company was subsequently acquired by AMD) and Google’s just released Genie 4.

When World Labs announced Atlas, it was described as an “omni world model for spatial intelligence.” But what Atlas enables — basically a 3D simulation of a world — isn’t the same as visual reasoning.

According to Dai, world models enable “direct action on a world,” whereas Elorian is more focused on what he calls “indirect action.” By that he means using visual reasoning to inform a decision or workflow (what Glimpse does) , rather than building simulated environments that allow navigation and manipulation (what Atlas does).

As an example, you might use visual reasoning to decide whether to reject a user-contributed video for violating a policy. “You don't need a 3D rendering to do that,” said Dai.

Applications for visual reasoning

Other business opportunities for Elorian include robotics, engineering, and defense applications. Dai added that architects need to regularly check building codes and compliance — an ideal task for a vision reasoning model.

In two robotics benchmarks , WGO Bench and ERQA, Elorian claims to rate equal or higher than frontier models , and at a much cheaper price:

Getting back to counting decoys as one measure of success for vision models, Dai argues that reasoning about what a model sees is crucial to its accuracy and effectiveness in real-world applications.

He gives an example of a warehouse robot counting items in a box.

“A familiar 12-slot tray might only hold 10 parts, and a 24-pack label doesn’t mean 24 items are still in the box,” said Dai. “A model relying on what it expects to see will face many issues in real-world situations, and Glimpse is designed to work through the image instead of jumping to a quick conclusion. We’ve designed Glimpse to ground the count in visible evidence as opposed to basic cues that could lead other models astray.”

Why Elorian thinks AGI requires visual reasoning

Elorian goes so far as to claim that visual reasoning is a foundational requirement for AGI (artificial general intelligence). From the Glimpse launch post:

“We believe building the full stack across systems, data, model architecture, and training — from the ground up for multimodal intelligence — can unlock AGI in the physical world.”

Dai acknowledges there’s “still a huge gap between humans and machines.” But he thinks visual reasoning is the key to closing that gap.

“I think the way to think about visual reasoning is as a fundamental step of reasoning in human intelligence,” he said. “And ultimately, if we want to build anything that's like AGI, we can't just skip visual.”