评测大语言模型的个性化能力
英文标题:Benchmarking the Personalization Capabilities of Large Language Models
论文提出以经典“说服”意义上的个性化为核心,评估大语言模型生成的信息能否促使第三方接收者采取预期行动。研究发布了 SDR-Bench:涵盖 22 个行业、约 200 家企业的 6,279 条客户成功案例,并通过时间约束模拟避免未来数据泄漏。测试发现,前沿 LLM 与深度研究 Agent 的个性化表现存在一致的平台期;在一个《财富》100 强科技公司样本中,没有模型能够在统计意义上区分成功与失败的触达内容。与 12 名职业销售代表开展的实地部署表明,48% 的模型生成内容被评为可立即使用,资深专家评分的一致性达到 Pearson 0.82。
483 B
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 5 月 23 日]
标题:评测大语言模型的个性化能力
查看 Ashutosh Srivastava 及另外 6 位作者撰写的论文《评测大语言模型的个性化能力》PDF。
摘要:个性化是指在发送者、渠道和时间保持不变的前提下,通过改变信息来促使特定接收者采取行动。它在心理学和营销领域有着悠久传统,通常被视为发送者与接收者具有独立目标的双边问题。大语言模型能够根据所推断的接收者状态生成连续变化的信息版本,从而消除了经典检索与排序方法中受限于固定内容库存的约束;这也带来了一个问题:当前模型在经典意义上的个性化表现究竟如何。现有 LLM 个性化基准测试衡量的是发送者一侧的适应能力,即接收者就是模型所服务的同一用户。而双边问题——生成的信息是否能促使第三方采取预期行动——此前仅通过 A/B 测试和小规模人工研究进行考察,无法在需要时针对新模型重新运行。我们将 Kamenica 和 Gentzkow(2011)的贝叶斯说服框架适配至生成式 Agent,并将该框架应用于销售场景;在这一场景中,接收者的行动通常会与促成行动的触达内容一并被记录。我们发布了 SDR-Bench,这是一个公开语料库,包含跨越 22 个行业、约 200 家企业的 6,279 条客户成功案例,并通过时间受限的模拟机制防止未来数据泄漏。在前沿 LLM 和深度研究 Agent 的测试中,我们观察到一致的个性化性能平台期;在一个《财富》100 强科技公司样本中,没有任何模型能够在统计意义上区分成功与失败的触达内容。一项由 12 名职业销售代表参与的实地部署验证了该框架:48% 的模型生成内容被评为可立即使用,资深专家之间的一致性达到 Pearson 0.82。我们公开发布 SDR-Arena 和 SDR-Bench,以支持对大规模生成式个性化开展可复现研究。
提交历史
来自:Ashutosh Srivatava [
]
[v1]
2026 年 5 月 23 日(星期六)18:16:54 UTC(7,253 KB)
English / 原文
[Submitted on 23 May 2026]
Title:Benchmarking the Personalization Capabilities of Large Language Models
View a PDF of the paper titled Benchmarking the Personalization Capabilities of Large Language Models, by Ashutosh Srivastava and 6 other authors
Abstract:Personalization, the act of varying a message to induce action from a specific receiver while keeping sender, channel, and time fixed, has a long tradition in psychology and marketing as a two-party problem in which sender and receiver have independent objectives. Large language models remove the bounded-inventory constraint of classical retrieval-and-ranking approaches by generating a continuum of message variants conditioned on inferred receiver state, raising the question of how well current models perform personalization in the classical sense. Existing LLM personalization benchmarks measure sender-side adaptation, in which the receiver is the same user the model is serving. The two-party question, whether a generated message induces its intended action in a third party, has been investigated only through A/B tests and small-scale human studies that cannot be re-run against a new model on demand. We adapt the Bayesian Persuasion framework of Kamenica and Gentzkow (2011) to generative agents and instantiate the formulation in sales, where receiver actions are routinely logged against the outreach that induced them. We release SDR-Bench, a public corpus of 6,279 customer success stories spanning 22 industries and approximately 200 enterprises, served through a temporally constrained simulation that prevents future-data leakage. Across frontier LLMs and deep-research agents, we observe a consistent personalization plateau and on a Fortune 100 tech cohort no model statistically separates successful from unsuccessful outreach. A field deployment with 12 professional sales representatives validates the framework, with 48 percent of model-generated content rated immediately useful and senior-expert agreement at Pearson 0.82. We release SDR-Arena and SDR-Bench publicly to support reproducible study of generative personalization at scale.
Submission history
From: Ashutosh Srivatava [
]
[v1]
Sat, 23 May 2026 18:16:54 UTC (7,253 KB)