摘要：我们在环境动态未知且无合适奖励函数可用的情况下，解决如何安全地训练智能体策略并部署良好且安全策略的问题。在安全关键环境的背景下，我们认为传统强化学习不切实际，转而借助人类输入这一资源。我们提出 DROPJ，一种以人为中心的安全训练与部署方法。我们首先从已有的真实世界轨迹数据集中学习一个世界模型（一个可学习的模拟器）。随后，由人类在该学习到的模拟器中玩游戏，以提取多条信息丰富的模拟轨迹。我们从这些轨迹中采样成对的模拟轨迹片段，并向人类征询他们对这些片段的偏好，以及做出选择的理由（即理由说明）。接着，我们根据这些带理由的偏好训练一个奖励模型，并将其与世界模型相结合，通过模型预测控制直接部署智能体。通过真实用户实验，我们发现，相比其他策略，由用户生成信息丰富的模拟轨迹能显著降低训练过程中的计算成本，并同时提升部署阶段的性能。在学习模拟器内进行训练的背景下，我们证明使用偏好而非其他类型的反馈能大幅改善部署性能。我们进一步证明，伴随偏好的安全理由可以显著增强安全性，或在部署过程中优先处理用户指定的、与这些理由相关联的安全要素。

Abstract:We address the problem of safely training an agent policy and deploying a good and safe policy, in settings where the environment dynamics are unknown and no suitable reward function is available. In the context of safety-critical environments, we consider traditional reinforcement learning impractical and resort to the resource of human input. We introduce DROPJ, a human-centred method for both safe training and deployment. We first learn a world model (a learned simulator) from a dataset of prior real-world trajectories. A human then plays the game in this learned simulator to extract several informative simulated trajectories. From these, we sample pairs of simulated trajectory segments and elicit from a human their preference over these segments, as well as a reason (justification) for their choice. We then train a reward model from these justified preferences and use it, together with the world model, to directly deploy the agent using model predictive control. Running real-user experiments, we find that generating informative simulated trajectories from a user significantly reduces the computational cost during training compared to other strategies, and can also improve the performance during deployment. In the context of training within a learned simulator, we show that the use of preferences rather than other types of feedback substantially improves the performance during deployment. We further demonstrate that safety justifications accompanying preferences can significantly enhance safety or prioritise user-prescribed aspects of safety associated with them during deployment.