arXiv:2607.10590v1 Announce Type: new Abstract: We investigate how annotator demographic attributes, supplied as prompt cues, shape the alignment between large language model (LLM) predictions and human annotations across five tasks. Using five open-source LLMs, we systematically vary the number an

中文全文翻译正在补齐，可先查看下方英文原文。

[Submitted on 12 Jul 2026]

Title:Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement

View a PDF of the paper titled Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement, by Mahammed Kamruzzaman and 2 other authors

View PDF

Abstract:We investigate how annotator demographic attributes, supplied as prompt cues, shape the alignment between large language model (LLM) predictions and human annotations across five tasks. Using five open-source LLMs, we systematically vary the number and composition of demographic components in the prompt, spanning every combination from single-attribute through full-attribute configurations. Our experiments reveal three principal findings. First, alignment consistently peaks with one to three high-signal attributes and degrades under the full attribute set, establishing a clear over-specification threshold. Second, the overall magnitude of demographic influence on human annotations does not predict which attributes improve LLM alignment; instead, both the learnability and the directional coherence of each attribute's annotation signal need to be considered jointly. Third, neuron probing reveals that specialized activation correlates with alignment gains only under coherent annotation signals, and that activation volume alone does not imply steerability. Together, these results demonstrate that demographic prompting is not a monolithic intervention: its utility is highly context-dependent, shaped by attribute signal quality, task characteristics, and model architecture.

Submission history

From: Mahammed Kamruzzaman [

view email

]

[v1]

Sun, 12 Jul 2026 06:00:51 UTC (9,109 KB)