摘要：人们普遍观察到，在线强化学习（RL）在广泛的性能指标上能够产生优于离线方法的策略。尤其是，经过 RL 训练的策略表现出更强的分布外（OOD）行为，而仅采用模仿学习方法训练的模型通常难以应对这种情况。近期一项研究提出了一个侧重 OOD 的基准测试，并报告称，经过 RL 训练的视觉-语言-动作（VLA）策略相比通过监督微调（SFT）训练的对应模型，具有显著更好的 OOD 性能，以及略优的分布内（IND）性能。在本工作中，我们研究混合离线—在线训练能否结合两种方法的优势。具体而言，我们研究了通过离线数据或离线训练的参考策略进行正则化的 RL 方法。我们在该 OOD 基准上评估这些方法，并将其与纯离线训练和标准 RL 进行比较。结果表明，尽管离线训练本身只能实现有限的 OOD 性能，但将离线监督纳入 RL 后，能够在显著提高训练效率的同时保留强大的 OOD 能力。具体来说，这些引导式方法仅需约一半的训练预算，即可达到接近标准 RL 的性能。混合方法并未形成速度与 OOD 性能之间的权衡，而是在获得效率提升的同时保持了较强的 OOD 能力。项目页面：此 HTTPS URL

Abstract:It is commonly observed that online reinforcement learning (RL) produces better-performing strategies than offline methods across a broad range of performance measures. In particular, RL-trained policies exhibit stronger out-of-distribution (OOD) behavior, where models trained only with imitation learning approaches often struggle. A recent study introduced an OOD-focused benchmark and reported that RL-trained vision-language-action (VLA) policies achieve noticeably better OOD performance and slightly better in-distribution (IND) performance than their counterparts trained with supervised fine-tuning (SFT). In this work, we investigate whether hybrid offline-online training can combine the advantages of both approaches. Specifically, we study RL methods regularized by offline supervision via either offline data or an offline-trained reference policy. We evaluate these approaches on the OOD benchmark and compare them with both offline-only training and standard RL. Our results show that although offline training achieves limited OOD performance by itself, incorporating offline supervision into RL preserves strong OOD capability while substantially improving training efficiency. In particular, the guided methods reach performance close to that of standard RL while requiring roughly half of the training budget. Rather than producing a trade-off between speed and OOD performance, the hybrid approach retains strong OOD capability while achieving this efficiency gain. Project page: this https URL