哪个答案是17岁少年写的?六款前沿语言模型在作者年龄错误归因、置信度失调与非矫正性自我解释中的行为案例研究
英文标题:Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models
研究人员针对 Claude Fable 5、Gemini 3.1 Pro、GPT-5.6 Sol、Grok 4.5、GPT-4o 和 DeepSeek V4 等六款前沿语言模型展开了作者年龄推断能力的结构化案例研究。实验通过 1,081 次 API 会话评估了模型将手写试卷答案与 17 岁、28 岁和 45 岁三位作者匹配的表现;结果显示,半数模型表现接近随机猜测(1/3),却给出了 73% 至 88% 的极高置信度,且 17 岁少年的正式措辞在高达 90% 的情况下被误判为成人。五款模型的自述置信度与实际准确率完全无关甚至呈负相关,且模型在知晓真相后虽能头头是道地自我剖析“刻板印象”,但这种解释能力并不能反映或修正其原本的判断缺陷。该研究揭示了前沿大模型在文本属性推断、置信度校准与事后反思机制上的深层系统性偏差,提醒开发者在将 LLM 应用于作者画像、评分及司法文本分析时须保持审慎。
612 B
380 B
24.2 KB
34.7 KB
中文
[提交于 2026 年 10 月 22 日]
标题:哪个答案是17岁少年写的?六款前沿语言模型在作者年龄错误归因、置信度失调与非矫正性自我解释中的行为案例研究
查看论文 PDF 版本:《Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models》,作者:Brianne Lee
摘要:三名学生手写回答了同一道试题:“什么是幸福?你幸福吗?”。采集时记录的学生年龄分别为 17 岁、28 岁和 45 岁。其中 17 岁学生的回答措辞谨慎、表达正式。在被要求将每个答案与对应的年龄和性别标签进行匹配时,绝大多数前沿语言模型都极其自信地将其归因于成年人。本文针对这一失败现象展开了结构化案例研究,测试了六款前沿模型:Claude Fable 5、Gemini 3.1 Pro、GPT-5.6 Sol、Grok 4.5、GPT-4o (2024) 和 DeepSeek V4。我们在六种操纵了情境框架、披露格式和受众意识提示的实验条件下,共运行了 1,081 次 API 会话,并对 3,225 次模型响应进行了评分。六款模型中有三款在匹配任务中的表现接近或等于随机概率(1/3),但其报告的平均置信度却高达 73% 至 88%。最年轻作者的答案在高达 90% 的测试轮次中被错误归因为成年人。移除多项选择的辅助框架后,准确率进一步崩溃:在要求进行开放式年龄估计时,三款模型预测偏差从未进入真实年龄 3 年的范围内。机构情境框架(例如标明“社区大学生物学期末考试”)显著降低了表现最佳模型的准确率;而中性标签则没有这种负面影响。单条受众意识提示能显著改善某一款模型的表现(+21 个百分点),并将另一款模型推至上限满分,但对表现最弱的三款模型没有可检测到的效果。在六款模型中,有五款模型的自述置信度与正确性无关甚至呈负相关。在揭晓真实答案后,某款模型在 114 次解释中有 68 次准确指出了自身的刻板印象,但它在盲测条件下的准确率却是所有模型中最低的,同时置信度最高:这种条理清晰的事后诊断并不能对应或反映其判断本身的质量。本研究是一项基于三道题目、单一主题、单语言的案例研究,旨在剖析并隔离其作用机制,而非用于评估总体的偏见发生率。相关数据、代码和模型原始输出均已公开披露。
提交历史
来自:Brianne Lee [
]
[v1]
2026 年 10 月 22 日 星期四 21:45:19 UTC (18 KB)
English / 原文
[Submitted on 22 Oct 2026]
Title:Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models
View a PDF of the paper titled Which Answer Did the Seventeen-Year-Old Write? A Behavioral Case Study of Author-Age Misattribution, Miscalibrated Confidence, and Non-Corrective Self-Explanation in Six Frontier Language Models, by Brianne Lee
Abstract:Three students answered the same exam prompt by hand: "What is happiness? Are you happy?" Their ages, recorded at collection, are 17, 28, and 45. The 17-year-old's answer is hedged and formally phrased. Asked to match each answer to an age and gender label, most frontier language models confidently assign it to an adult. This paper reports a structured case study of that failure across six models: Claude Fable 5, Gemini 3.1 Pro, GPT-5.6 Sol, Grok 4.5, GPT-4o (2024), and DeepSeek V4. We ran 1,081 API sessions and scored 3,225 responses across six experimental conditions that manipulate framing, disclosure format, and audience-awareness instruction. Three of six models perform at or near chance (1/3) on the matching task while reporting 73-88% mean confidence. The youngest writer's answer is misattributed to an adult in up to 90% of trials. Removing the multiple-choice scaffold collapses accuracy further: three models never place within three years of the true age once open-ended estimation is required. Institutional framing ("a community-college biology final") significantly degrades the best-performing model's accuracy; a neutral label does not. A single audience-awareness instruction significantly improves one model (+21 points) and moves another to ceiling, with no detectable effect on the three weakest models. Stated confidence is uncorrelated or inversely correlated with correctness for five of six models. After ground truth is revealed, one model names its own stereotyping in 68 of 114 explanations, yet shows the lowest accuracy and highest confidence of any model under blind presentation: articulate retrospective diagnosis that does not track the quality of its judgments. This is a three-item, single-topic, single-language case study meant to isolate a mechanism, not to estimate a population bias rate. Data, code, and raw model outputs are publicly released.
Submission history
From: Brianne Lee [
]
[v1]
Thu, 22 Oct 2026 21:45:19 UTC (18 KB)