摘要：在推荐系统中，评估自然语言解释对于建立用户信任、提高系统透明度与增强用户参与度至关重要。然而，现有的人工评估方法虽然准确，但在现代推荐平台所需的大规模场景下资源消耗巨大且难以落地；同时，采用单智能体（Single-Agent）大语言模型（LLM）的自动化方法普遍存在提示词敏感（Prompt Sensitivity）和输出不一致的问题。为应对这些挑战，我们提出了 MAREval——一个用于评估推荐解释的结构化多智能体（Multi-Agent）LLM 框架。MAREval 协同编排了以下机制：(i) 规划 Agent（Planner Agent），采用创新的辩论链（Chain of Debate, CoD）提示策略协调各 Agent 角色并执行逻辑一致的评估规划；(ii) 调解 Agent（Moderator Agent），通过缓解提示词漂移（Prompt Drift）来规范讨论流程；(iii) 仲裁 Agent（Arbitrator Agent），用于汇总多轮评估结果；以及 (iv) 蒙特卡洛采样（Monte Carlo Sampling）方法，以提升系统鲁棒性及与人类判断的对齐度。我们在公开推荐数据集（TopicalChat）和专有推荐数据集上进行了全面评估，结果表明 MAREval 明显优于当前的 SOTA 基线模型。在公开基准与专有电商数据集上的广泛实验显示，相较于强大的单智能体和多智能体基线，MAREval 显著提升了与人类判断的一致性；稳定性分析表明其在重复试验中的方差大幅降低。在严格的大规模人工标注准入门槛（Human-Annotation Gate）测试中，MAREval 达到了以往评估工具未能达到的生产环境质量阈值，且在线 A/B 测试表明其在用户参与度和收入指标上均带来了统计学上显著的提升。这些结果确立了 MAREval 作为真实业务系统中针对推荐解释进行可扩展、高可靠且符合人类偏好评估的有效解决方案。

Abstract:Evaluating natural language explanations in recommender systems is essential for fostering user trust, transparency, and engagement. However, existing evaluation approaches like human evaluations, while accurate, are resource-intensive and impractical at the scale required by modern recommendation platforms. Also, automated methods using single-agent LLMs suffer from prompt sensitivity and inconsistent outputs. To address these challenges, we propose MAREval, a structured multi-agent framework for evaluating recommendation explanations using large language models. MAREval orchestrates (i) a planner agent that uses a novel Chain of Debate (CoD) prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans; (ii) a moderator agent that regulates discussions by mitigating prompt drift; and (iii) an arbitrator agent that aggregates outputs from multiple evaluation rounds and (iv) a Monte Carlo sampling method, improving robustness and alignment with human judgment. We conduct comprehensive evaluations on both public (TopicalChat) and proprietary recommendation datasets, demonstrating that MAREval outperforms state-of-the-art baselines. Comprehensive experiments on a public benchmark and a proprietary e-commerce dataset show that MAREval improves alignment with human judgments over strong singleand multi-agent baselines. Stability analyses indicate substantially lower variability across repeated trials. In a large human-annotation gate, MAREval meets production quality thresholds where prior evaluators fall short, and online A/B testing demonstrates statistically significant improvements in engagement and revenue metrics. These results establish MAREval as a scalable and reliable solution for human-aligned evaluation of recommendation explanations in real-world systems