摘要：对比式视觉语言模型通过对齐匹配的图像—文本对来学习共享嵌入空间，但其表征之间仍存在模态鸿沟。既有研究报告称，修改这一鸿沟会产生不同效果：缩小鸿沟能够改善零样本分类和跨模态对齐，而移除与鸿沟相关的结构则可能降低图文检索性能。本文为这些依赖任务的效应提供了统一的几何学解释。在 CLIP 和 SigLIP 编码器中，我们发现，一个单一主导方向可捕获图像—文本均值分离平方范数的 94.4% 至 99.9%，表明均值分离分量近似为秩一。对相似度分数的分解进一步揭示了三个任务特定的作用。在零样本分类中，查询侧固定的鸿沟偏移减法与添加类别偏置完全等价。在标准跨模态检索中，投影移除鸿沟方向并对残差重新归一化，会丢弃候选项特异的范数信息，从而引入乘法性的排序失真；一个由几何推导出的指数与网格搜索最优值高度一致（Spearman rho = 0.93），并可在部分设置中恢复性能，尽管其收益的迁移并不均匀。在混合模态检索中，鸿沟方向会按模态对候选项进行排序；与随机方向或非鸿沟对照相比，移除该方向能够改善跨模态排序。移除后的残余语义结构界定了秩一解释的适用边界。总体而言，这些结果解释了为何修改鸿沟会在不同下游设置中提升、降低或恢复性能。通过阐明鸿沟修改在何时以及为何会改变模型行为，该研究为在已评估的下游任务中、基于相似度的视觉语言系统选择鸿沟干预措施提供了原则性依据。

Abstract:Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing gap-related structure can degrade image-text retrieval. In this paper, we provide a unified geometric explanation for these task-dependent effects. Across CLIP and SigLIP encoders, we find that a single dominant direction captures 94.4-99.9% of the squared norm of the image-text mean separation, revealing that the mean-separation component is approximately rank-one. A decomposition of the similarity score then identifies three task-specific roles. In zero-shot classification, query-side fixed gap-offset subtraction is exactly equivalent to an additive class bias. In standard cross-modal retrieval, projecting out the gap direction and renormalising residuals discards candidate-specific norm information, inducing a multiplicative ranking distortion; a geometry-derived exponent tracks the grid-search optimum (Spearman rho = 0.93) and restores performance in some settings, although the gains transfer unevenly. In mixed-modal retrieval, the gap direction sorts candidates by modality; its removal can improve cross-modal ranking, unlike random or non-gap controls. Residual semantic structure after removal defines the limits of the rank-one account. Together, these results explain why gap modification can improve, degrade, or restore performance across downstream settings. By clarifying when and why gap modification changes model behavior, this account provides a principled basis for selecting gap interventions in similarity-based vision-language systems across evaluated downstream tasks.