摘要：聚合异构视觉语言模型（VLM）可以提升多模态推理能力，但无论是单个模型的置信度，还是聚合后答案的置信度，都无法在系统层面上衡量可靠性。我们提出了 CUSP（基于语义意见池化的集体不确定性，Collective Uncertainty through Semantic Opinion Pooling），这是一种免训练的不确定性量化框架，可将多个 VLM 的响应映射至共享的语义响应空间，并汇总为统一的池化语义意见（pooled semantic opinion），进而提供两个互补的系统级信号：集体不确定性（即池化意见的离散程度）以及詹森-香农散度（Jensen-Shannon divergence, JSD，即模型间意见的冲突程度）。在这一池化语义意见体系下，未归一化的集体熵可精确分解为各模型单体语义熵的均值与 JSD 之和，从而将总离散度与模型冲突彻底解耦。由于既不需要 token logits，也不需要校准标签，CUSP 同样适用于开源权重模型与商业 API 闭源 VLM。在静态多 VLM 集成实验中，集体不确定性在小模型区间表现最为突出（预测错误检测 AUROC 为 0.764，拒绝回答 AUARC 为 0.889），相比多数投票（majority voting）与朴素选择（naive selection）等不确定性基线高出 4.7 到 15.8 个百分点，且随着集成规模扩大，优势进一步拉大；而在所评估的商业模型区间中，JSD 的表现最为强劲（0.819 AUROC，0.910 AUARC），对硬答案（hard-answer）模型冲突的排序 AUROC 高达 0.982。此外，池化预测本身的准确率也比单模型平均水平提高了 5.6 到 13.0 个百分点。在多步多 Agent 系统的完整执行轨迹中，子 Agent 的集体不确定性对系统故障的排序显著高出随机水平（0.619 AUROC），并在所有评估信号中取得了最佳的拒绝回答排序表现（0.699 AUARC）。

Abstract:Aggregating heterogeneous vision-language models (VLMs) can improve multimodal reasoning, but neither an individual model's confidence nor that of the aggregated answer measures reliability at the system level. We present CUSP (Collective Uncertainty through Semantic Opinion Pooling), a training-free uncertainty quantification framework that maps multiple VLM responses to a shared semantic response space, pools them into a pooled semantic opinion, and reports two complementary system-level signals: collective uncertainty, the dispersion of the pooled opinion, and Jensen-Shannon divergence (JSD), the conflict among the model-level opinions. Within this pooled semantic opinion, the unnormalized collective entropy decomposes exactly into the mean of the models' individual semantic entropies and the JSD, separating total dispersion from model conflict. Requiring neither token logits nor calibration labels, CUSP applies to open-weight and commercial VLMs alike. In static multi-VLM ensembles, collective uncertainty is the strongest signal in the small-model regime (0.764 AUROC for prediction-error detection, 0.889 AUARC for abstention), outperforming uncertainty baselines majority voting and naive selection by 4.7 to 15.8 points and widening its margin as the ensemble grows; JSD is strongest in the evaluated commercial regime (0.819 AUROC, 0.910 AUARC) and ranks hard-answer model conflict with AUROC up to 0.982. The pooled prediction also improves accuracy over the average single model by 5.6 to 13.0 points. Over the full trajectory of a multi-step, multi-agent system, subagent collective uncertainty ranks system failures above chance (0.619 AUROC) and gives the best abstention ordering among the evaluated signals (0.699 AUARC).