论文研究 arXiv cs.CL Wed, 19 Aug 2026 正文 ok 261 words

多模态大语言模型中的不确定性感知决策

英文标题:Uncertainty-Aware Decision Making in Multimodal Large Language Models

这篇综述围绕多模态大语言模型(MLLM)的不确定性感知能力,提出以“决策”为中心的分析框架:不确定性来源产生可观测信号,信号需经校准或风险控制,并最终影响系统行动。论文梳理了 token 与 logit 不确定性、语义分歧、扰动稳定性、视觉 grounding、置信度表达、保形预测、选择性回答、拒答、澄清、检索、自检和升级处理等方法。作者强调,不确定性不应仅被评估为一个置信度数值,而应根据其是否能改善模型在证据不足、模态冲突、分布偏移或高风险场景中的行为来衡量。对开发者而言,该综述为构建更可靠的多模态 Agent、评测不确定性校准及设计拒答或升级策略提供了系统参考。
打开原文相关 4 · 机会 0
media2.1 KB media461 B media612 B media380 B media9.1 KB media32.7 KB media22.8 KB media24.2 KB media34.7 KB

中文

[提交于 2026 年 8 月 17 日]

标题:多模态大语言模型中的不确定性感知决策

查看由 Abderrahmene Boudiaf 与另外 2 位作者撰写的论文《多模态大语言模型中的不确定性感知决策》PDF

查看 PDF HTML(实验版)

摘要:多模态大语言模型(MLLM)日益需要回答其正确性依赖于视觉、文本、时间、声学、文档、图表或具身证据的问题。因此,它们的失败并不只是语言层面的失败。一个流畅的回答可能掩盖输入质量不佳、感知错误、grounding 薄弱、模态间冲突、推理不稳定、分布偏移,或所提供证据无法回答该问题等情况。本综述围绕一个以决策为中心的框架组织有关不确定性感知 MLLM 的研究:不确定性来源会产生可观测信号,这些信号必须针对风险进行校准或控制,而经校准的不确定性应当决定系统采取何种行动。我们回顾了关于 token 和 logit 不确定性、语义分歧、扰动不稳定性、grounding 与归因分数、口头表达的置信度、验证器与评判器分数、保形预测、选择性回答、拒答、澄清、检索、自检和升级处理的研究。本文的核心论点是,不确定性不应仅作为置信度数值来评估;更应评估其是否能在多模态证据不足、相互冲突、发生偏移或风险较高时改善系统行为。我们将本综述与仅面向文本的不确定性和拒答综述、广义 MLLM 综述、MLLM 幻觉综述以及面向安全的综述进行对比定位。最后,我们总结了来源感知分解、行动感知基准、分布偏移下的校准、黑盒不确定性估计、更广泛的模态覆盖、可复现报告以及以人为中心的不确定性沟通等开放问题。

提交历史

来自:Abderrahmene Boudiaf [

查看邮箱

]

[v1]

2026 年 8 月 17 日,周一,19:43:21 UTC(7,765 KB)

English / 原文

[Submitted on 17 Aug 2026]

Title:Uncertainty-Aware Decision Making in Multimodal Large Language Models

View a PDF of the paper titled Uncertainty-Aware Decision Making in Multimodal Large Language Models, by Abderrahmene Boudiaf and 2 other authors

View PDF HTML (experimental)

Abstract:Multimodal large language models (MLLMs) increasingly answer questions whose correctness depends on visual, textual, temporal, acoustic, document, chart, or embodied evidence. Their failures are therefore not only linguistic. A fluent answer may conceal poor input quality, a perceptual error, weak grounding, conflict between modalities, unstable reasoning, distribution shift, or a question that is not answerable from the supplied evidence. This survey organizes the literature on uncertainty-aware MLLMs around a decision-centered framework: uncertainty sources give rise to observable signals, signals must be calibrated or controlled for risk, and calibrated uncertainty should determine the system action. We review work on token and logit uncertainty, semantic disagreement, perturbation instability, grounding and attribution scores, verbalized confidence, verifier and judge scores, conformal prediction, selective answering, abstention, clarification, retrieval, self-checking, and escalation. The central argument is that uncertainty should not be evaluated only as a confidence number; it should be evaluated by whether it improves behavior under insufficient, conflicting, shifted, or high-risk multimodal evidence. We position this survey against text-only uncertainty and abstention surveys, broad MLLM surveys, MLLM hallucination surveys, and safety-oriented reviews. We conclude with open problems in source-aware decomposition, action-aware benchmarks, calibration under shift, black-box uncertainty estimation, broader modality coverage, reproducible reporting, and human-centered uncertainty communication.

Submission history

From: Abderrahmene Boudiaf [

view email

]

[v1]

Mon, 17 Aug 2026 19:43:21 UTC (7,765 KB)