问题在规划而非解码器:诊断并修复推理增强型文生图的组合性失败
英文标题:The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation
论文研究推理增强型文生图模型在处理组合式提示词时出错的根源,发现主要瓶颈在生成前的文本规划,而非图像解码器。实验显示,GoT-R1 的解码器能够以 94% 的比例忠实实现规划中的空间关系,但规划器对同一语义布局会因措辞不同而表现悬殊,例如“左侧”准确率为 98%,而“右侧”仅为 54%。通过符号验证、最小化原地修复、仅重写边界框几何信息或直接替换规划,研究分别获得最高 +13.3 个百分点的提升,且无需重新训练模型。作者还发布了规划忠实度评估协议、全部规划数据及 1.2 万张生成图像。
640 B
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 8 月 22 日]
标题:问题在规划而非解码器:诊断并修复推理增强型文生图的组合性失败
查看 Ashritha Gonuguntla 所著论文《问题在规划而非解码器:诊断并修复推理增强型文生图的组合性失败》的 PDF
摘要:GoT-R1 等推理增强型文生图模型会在生成图像 token 之前输出一份显式文本规划——其中包括对象名称、属性和边界框。当这类模型未能完成一个组合式提示词时,究竟是规划出了问题,还是规划正确但解码器未能忠实执行?由于规划可由机器读取,因此可在解码前进行编辑,这使两种问题能够被区分开来。我们首先验证测量标准。在模型自身推理链中交换两个边界框,能够显著翻转生成的布局:基于检测器的准确率从 0.75 降至 0.48(p<1e-3),而一个被广泛使用的、基于 VQA 的空间指标却上升。一项由五名评审者参与的人类研究显示,在 81% 的样本上,人类判断与检测器一致;而与 VQA 判定器一致的比例仅为 57%。因此,所有空间结果均采用几何评分。在可靠的测量下,解码器是忠实的执行者:94% 的生成布局实现了规划中的关系,并且即使重新排列规划中对象片段的顺序,对象与边界框的绑定依然保持。规划器才是瓶颈。它会因依赖措辞的原因写出错误关系——对于语义相同的布局,“左侧”的准确率为 98%,而“右侧”仅为 54%;我们通过提及顺序控制实验分离出这一栅格顺序偏差——并且会生成杂乱的几何布局,而解码器会忠实地将其复现。因此,编辑规划即可在无需重新训练的情况下修复图像:采用带重采样的符号验证可提升 +5.0 个百分点(p<1e-3),最小化原地修复可提升 +6.0 个百分点(p=.02),仅重写边界框几何信息可提升 +10.7 个百分点(p<1e-4),而直接替换规划可提升 +13.3 个百分点(p=1e-4)。这些提升不受规划文本风格及其在规划器下似然的影响,但取决于其几何信息。因此,只要规划在内部保持一致,模块化的规划器—解码器设计就是可行的:边界框与文本之间的矛盾会导致对象重复和身份融合。我们发布了规划忠实度评估协议、全部规划数据以及 1.2 万张生成图像。
提交历史
来自:Ashritha Gonuguntla [
]
[v1]
2026 年 8 月 22 日,星期六,01:34:23 UTC(1,302 KB)
English / 原文
[Submitted on 22 Aug 2026]
Title:The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation
View a PDF of the paper titled The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation, by Ashritha Gonuguntla
Abstract:Reasoning-augmented text-to-image models such as GoT-R1 emit an explicit textual plan - object names, attributes, and bounding boxes - before generating image tokens. When such a model fails a compositional prompt, is the plan wrong, or is the plan right and the decoder unfaithful? Because the plan is machine-readable it can be edited before decoding, which makes the two separable. We first validate the ruler. Swapping the two bounding boxes inside the model's own chain demonstrably flips the generated layout: detector-based accuracy falls 0.75 -> 0.48 (p<1e-3), while a widely used VQA-based spatial metric rises. A five-rater human study agrees with the detector on 81% of items and with the VQA judge on 57%. All spatial results therefore use geometric scoring. Under sound measurement the decoder is a faithful executor: 94% of generated layouts realize the planned relation, and object-box binding survives reordering of the plan's object segments. The planner is the bottleneck. It writes wrong relations for phrasing-dependent reasons - 98% accuracy on "left" against 54% on "right" for semantically identical layouts, a raster-order bias we isolate with a mention-order control - and cluttered geometry that the decoder faithfully reproduces. Editing the plan therefore fixes the image without retraining: symbolic verification with resampling gives +5.0 points (p<1e-3), minimal in-place repair +6.0 (p=.02), rewriting only box geometry +10.7 (p<1e-4), and replacing the plan outright +13.3 (p=1e-4). Gains are indifferent to the plan's prose style and to its likelihood under the planner, but not to its geometry. Modular planner-decoder designs are therefore viable, provided the plan is internally consistent: box-text contradictions induce object duplication and identity fusion. We release the plan-fidelity evaluation protocol, all plans, and 12k generated images.
Submission history
From: Ashritha Gonuguntla [
]
[v1]
Sat, 22 Aug 2026 01:34:23 UTC (1,302 KB)