文本、像素,还是两者兼用?评估多模态文档问答的输入表示
英文标题:Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
该研究系统比较了页面图像、提取文本及二者结合三种输入表示对多模态文档问答(Document QA)系统的影响。在固定提示词、评审模型和评分流程的条件下,研究覆盖 4 个商业模型端点、2 个语料库,以及金标准证据页和完整文档两种上下文设置。结果显示,在文档未超出图像预算时,页面图像在两个语料库、所有文档长度下均具备更高准确率,但延迟和成本会随文档长度明显增长;文本与图像各自擅长回答不同问题。仅根据问题文本进行路由的轻量级 TF-IDF 路由器,相比始终使用文本准确率提升 2.6 个点,并较始终使用视觉输入将中位延迟降低 30%,为开发者在准确率、成本和响应速度之间进行动态权衡提供了思路。
640 B
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 9 月 18 日]
标题:文本、像素,还是两者兼用?评估多模态文档问答的输入表示
查看由 Nikhil Reddy Pottanigari 及其他 4 位作者撰写的论文《文本、像素,还是两者兼用?评估多模态文档问答的输入表示》的 PDF
摘要:每个文档问答系统都始于一个很少被单独研究的选择:是向模型输入页面图像、提取出的文本,还是两者同时输入。我们将这一选择单独剥离出来,在 4 个商业模型端点、2 个语料库以及 2 种上下文设置(金标准证据页和完整文档)中,保持提示词、评审模型和评分流程不变进行比较。对于能够纳入图像预算的文档,页面图像在两个语料库的所有文档长度下均在准确率上领先;但这一优势伴随着不断增加的延迟和成本溢价:随着文档变长,文本处理延迟大致保持平稳,而图像处理延迟则持续上升。文本和图像也会在不同问题上失败:在所报告的各个单元中,19--25% 的样本恰好只有一种表示能够答对,因此两者无法相互替代。利用这种互补性,一个仅读取问题文本的轻量级 TF-IDF 路由器,在按文档划分且彼此独立的留出测试集上,相比始终使用文本的策略提升了 2.6 个点;相较始终使用视觉输入,中位延迟则降低了 30%。
提交历史
提交者:Nikhil Reddy Pottanigari [
]
[v1]
2026 年 9 月 18 日,星期五,22:26:49 UTC(2,071 KB)
English / 原文
[Submitted on 18 Sep 2026]
Title:Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
View a PDF of the paper titled Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA, by Nikhil Reddy Pottanigari and 4 other authors
Abstract:Every document QA system begins with a choice that is rarely studied on its own: whether to feed the model page images, extracted text, or both. We isolate this choice, holding the prompt, judge, and scoring pipeline fixed, across four commercial model endpoints, two corpora, and two context regimes (gold evidence pages and the full document). On documents that fit the image budget, page images lead on accuracy at every document length on both corpora, but this advantage carries a growing latency and cost premium: text latency stays roughly flat as documents lengthen while image latency rises steadily. Text and images also fail on different questions, with exactly one representation correct on 19--25% of items across the reported cells, so neither subsumes the other. Exploiting this complementarity, a lightweight TF-IDF router that reads only the question text gains 2.6 points over always-text while cutting median latency 30% relative to always-vision, on a document-disjoint held-out split.
Submission history
From: Nikhil Reddy Pottanigari [
]
[v1]
Fri, 18 Sep 2026 22:26:49 UTC (2,071 KB)