是什么驱动了 LLM 自我反思?武装冲突预测中不确定性路由的受控消融研究
英文标题:What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting
这项研究通过六种受控实验条件,拆分并检验 LLM 自我反思中的证据暴露、诊断脚手架、分类体系词汇和行动路由四个组成部分。结果显示,结构化诊断问题与非结构化反思的效果几乎没有差异,而仅提供完整不确定性分类体系词汇也无法带来显著增益。相比之下,按类型进行行动路由可将 F1 从 0.296 提升至 0.379;相对单次作答基线的总体提升为 0.101,且 Bootstrap 置信区间显示该提升显著。研究还在 GPT-4o 上复现了这一结论,表明对开发者而言,构建元认知预测 Agent 时应优先设计与不确定性类型对应的后续行动,而非单纯增加反思问题或分类术语。
640 B
612 B
380 B
9.1 KB
32.7 KB
22.8 KB
24.2 KB
34.7 KB
中文
[提交于 2026 年 5 月 29 日]
标题:是什么驱动了 LLM 自我反思?武装冲突预测中不确定性路由的受控消融研究
查看 Poli Nemkova 及另外 1 位作者撰写的论文《是什么驱动了 LLM 自我反思?武装冲突预测中不确定性路由的受控消融研究》的 PDF
摘要:人们普遍认为,自我反思能够改善 LLM 的推理能力,但究竟是哪个组成部分带来了这种增益,仍缺乏充分理解。我们提出了一项包含六种条件的受控消融实验,分离 LLM 自我反思的四个组成部分:证据暴露、诊断脚手架、分类体系词汇和行动路由。两项精确的零结果共同指向单一机制。第一,结构化诊断问题相比非结构化反思并未增加可测量的价值($\text{F1} = 0.296$ 对 $0.297$,$p = 1.000$,95% CI $[-0.041, +0.040]$)。第二,在将行动空间压缩为单一通用行动的同时呈现完整不确定性分类体系,同样没有增加价值($\Delta\text{F1} = +0.008$,95% CI 重叠),这排除了分类体系词汇是作用机制的可能性。按类型进行的行动路由带来了稳定的方向性增益($\text{F1} = 0.379$ 对 $0.296$);控制分类体系词汇影响后的保守估计为 $\Delta\text{F1} = +0.075$,而相对单次作答基线的总体增益经 Bootstrap CI 证实具有显著性($\Delta\text{F1} = +0.101$,95% CI $[+0.020, +0.185]$)。词汇—路由分解在 GPT-4o 上得到了复现:与通用反思相比,分类体系词汇没有显著价值($p = 0.773$),而行动路由带来了显著增益($p = 0.025$),证实该机制跨模型骨干依然成立。增益主要集中于结构上新颖的冲突:在缅甸($\text{F1}: 0.000 \rightarrow 0.353$)和乌克兰($0.167 \rightarrow 0.500$)案例中,仅词汇条件的表现并未超过通用反思,而行动路由打破了退化先验。这些发现表明,对于元认知 LLM 预测 Agent 而言,按类型进行的行动路由——而非诊断脚手架或分类体系词汇——是一项有前景的设计原则;同时也说明有必要在更多冲突类型上开展大规模评估。
提交历史
来自:Poli Apollinaire Nemkova [
]
[v1]
2026 年 5 月 29 日,星期五,03:48:38 UTC(32 KB)
English / 原文
[Submitted on 29 May 2026]
Title:What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting
View a PDF of the paper titled What Drives LLM Self-Reflection? A Controlled Ablation of Uncertainty Routing in Armed Conflict Forecasting, by Poli Nemkova and 1 other authors
Abstract:Self-reflection is widely assumed to improve LLM reasoning, yet which component drives the gain remains poorly understood. We present a controlled six-condition ablation isolating four components of LLM self-reflection: evidence exposure, diagnostic scaffolding, taxonomy vocabulary, and action routing. Two precise null results converge on a single mechanism. First, structured diagnostic questions add no measurable value over unstructured reflection ($\text{F1} = 0.296$ vs $0.297$, $p = 1.000$, 95% CI $[-0.041, +0.040]$). Second, presenting the full uncertainty taxonomy while collapsing the action space to a single generic action also adds no value ($\Delta\text{F1} = +0.008$, overlapping 95% CIs), ruling out taxonomy vocabulary as the mechanism. Typed action routing provides consistent directional gains ($\text{F1} = 0.379$ vs $0.296$); the conservative estimate controlling for taxonomy vocabulary is $\Delta\text{F1} = +0.075$, and the overall gain over the single-shot baseline is significant by bootstrap CI ($\Delta\text{F1} = +0.101$, 95% CI $[+0.020, +0.185]$). The vocabulary-routing decomposition replicates on GPT-4o: taxonomy vocabulary adds no significant value over generic reflection ($p = 0.773$), while action routing provides significant gains ($p = 0.025$), confirming the mechanism holds across backbones. Gains concentrate on structurally novel conflicts: in Myanmar ($\text{F1}: 0.000 \rightarrow 0.353$) and Ukraine ($0.167 \rightarrow 0.500$), the vocabulary-only condition recovers no more than generic reflection while action routing breaks the degenerate prior. These findings identify typed action routing -- not diagnostic scaffolding or taxonomy vocabulary -- as a promising design principle for metacognitive LLM forecasting agents, while motivating larger-scale evaluation across conflict typologies.
Submission history
From: Poli Apollinaire Nemkova [
]
[v1]
Fri, 29 May 2026 03:48:38 UTC (32 KB)