摘要：随着视觉语言模型（VLM）逐步走向物理部署，研究重心目前仍停留在基于主体视角消费级视频所评估的行动导向型具身智能（Embodied AI）上。这忽视了一类广泛存在的物理 AI（Physical AI）：基础设施 AI（Infrastructure AI），后者依赖固定摄像头获取开环洞察（例如安全监控和运营日志记录）。我们推出了 VANTAGE-Bench，一个专门用于衡量这一“基础设施 AI 鸿沟”（Infrastructure AI Gap）的基准测试。它涵盖三大运营领域（物流、交通和智慧空间），统一了涵盖语义、空间、时间以及时空能力的图像与视频评估，并突破了单选题形式，扩展至包括密集描述（dense captioning）和时空定位（spatio-temporal grounding）在内的八项任务形态。该基准针对单目标跟踪（Single Object Tracking）引入了单遍轨迹协议（single-pass trajectory protocol），并且据我们所知，这是首次在固定机位基础设施视频上进行此类评估并与专业跟踪器（specialist trackers）对比评分。标注涵盖 3,346 个媒体资产的三种形式：3,342 个视频任务标注、4,281 个图像定位标注以及 27,404 个检测框。

在对 17 个模型进行零样本（zero-shot）评估后，我们发现相较于以消费级视频为中心的基准，模型表现出的不足是局部的而非普遍的。事件验证（Event verification）、指称表达（referring expressions）和时序定位（temporal localization）在各个模型参数规模下普遍下降约 9 到 24 个百分点；而视频问答（Video QA）与 VideoMME 的差距保持在 5.3 个百分点以内，2D 空间指向（2D spatial pointing）相较于 BLINK 也未出现明显滑坡。从绝对指标来看，时间维度（temporal pillar）是最薄弱的环节：没有系统的时序定位 mIoU 超过 55.7，密集视频描述的 SODA_c 也没有超过 37.3。在跟踪任务上，前沿模型在短周期内与专业跟踪器的差距在 5 个百分点以内，但随着时限拉长，差距逐渐扩大。开源模型在 2D 目标定位上完全领先，这表明无论模型规模还是专有访问权限都无法单独解释这一现象。数据集、评测框架与排行榜见：此链接

Abstract:As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes.
Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: this https URL