摘要：大型语言模型（LLM）日益介入对人权是否得到实现以及如何实现的法律判断。然而，目前尚不存在用于评估其能否正确进行人权法推理的评测基准。为此，我们报告了开发 HumRightsBench 的工作：这是一项稳健且可扩展的方法论，用于构建首个经专家验证、基于情境的基准，以评估扎根于国际人权法义务结构的推理能力。我们对法律推理的 IRAC 框架进行了调整，使其更适合人权工作独特的推理模式：以 P（“提出救济方案”）替代 C（“法律结论”），形成 IRAP，并用其组织评测启发式方法。我们还制作了一系列试点真实情境，旨在涵盖现实世界人权议题的多个维度；这些情境由来自世界各地的人权律师和专业人士进行标注。最终，我们发现，模型在各项法律推理任务上的准确率差异很大：模型总体表现范围为 0.339 至 0.577，各任务的最小值—最大值范围为 0.025 至 0.774。这强烈表明，在 AI 评测科学这一新兴子领域发展的关键时期，HumRightsBench 能够成为推动其发展的有效工具。

Abstract:Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.