[提交于 2026 年 10 月 19 日]

标题：流畅却易错：大语言模型幻觉与事实可靠性基准测评（Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models）

查看由 Osama Shabih 及其他 3 位作者撰写的论文《Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models》PDF 版

查看 PDF HTML (实验性版本)

摘要：大语言模型（LLM）凭借在推理、文本生成和知识检索等方面的卓越能力，显著推动了自然语言处理及人工智能应用的发展。然而，大语言模型仍易受到两大核心挑战的困扰：幻觉（即生成事实错误或捏造的内容）以及响应偏差（指系统性偏离标准真实响应）。这些局限性引发了学术界和科研环境对其可靠性与可信度的严重担忧。本研究基于涵盖基础（Basic）、解释（Explanation）、高级（Advanced）和陷阱（Tricky）4 个复杂度层级的 70 个精心设计的学术提示词（Prompt，于 2025 年 2 月在各特定模型版本上进行测试），利用多种指标对 10 款前沿大语言模型——ChatGPT、Gemini、Claude 3.5 Sonnet、LLaMA 3、Mistral、Grok、Perplexity、DeepSeek V3、Falcon 和 Gemma 进行了全面对比分析。每份模型回答均由 3 位评审员独立依据 5 项指标进行评估：准确率（Accuracy）、精确率（Precision）、召回率（Recall）、响应偏差指数（Response Deviation Index）和幻觉指数（Hallucination Index）。结果显示，Perplexity 和 DeepSeek V3 取得了最高的综合准确率；Gemini 展现了强大的召回率，并在所有 5 项指标中呈现出均衡的性能轮廓；LLaMA 3 则通过较为保守的生成机制展现了强大的抗幻觉能力。在所有模型中，高级类提示词最具挑战性，而陷阱类提示词由于其二元（对/错）结构处理起来相对更可靠。Falcon 实现了完美的精确率和零幻觉，但代价是极低的召回率和最高的响应偏差。研究表明，生成回答的完整性与事实可靠性之间存在根本性的权衡取舍，且随着提示词复杂度的提高，这一矛盾愈发显著。

提交历史

来自：Shahab Saquib Sohail 博士 [

查看电子邮件

]

[v1] 2026 年 10 月 19 日 星期一 16:54:21 UTC (3,045 KB)

[Submitted on 19 Oct 2026]

Title:Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models

View a PDF of the paper titled Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models, by Osama Shabih and 3 other authors

View PDF HTML (experimental)

Abstract:Large Language Models (LLMs) have significantly advanced natural language processing and artificial intelligence applications through their capabilities in reasoning, text generation and knowledge retrieval. However, LLMs are still vulnerable to two important challenges: hallucination, generating factually incorrect or fabricated content, and response deviation, meaning systematic deviations from ground-truth responses. These limitations raise serious concerns regarding reliability and trustworthiness in academic and research environments. This study offers a thorough comparative analysis using a variety of metrics of ten advanced LLMs ChatGPT, Gemini, Claude3.5Sonnet, LLaMA3, Mistral, Grok, Perplexity, DeepSeekV3, Falcon, and Gemma based on a benchmark consisting of 70 carefully designed academic prompts across four complexity levels: Basic, Explanation, Advanced, and Tricky, evaluated in February 2025 on specific model versions. Each response was independently evaluated by three reviewers using five metrics: Accuracy, Precision, Recall, Response Deviation Index, and Hallucination Index. Perplexity and DeepSeekV3 achieved the highest overall accuracy, while Gemini demonstrated strong recall and a well-balanced performance profile across all five metrics. LLaMA3 showed strong resistance against hallucinations through a conservative method of generation. Advanced-level prompts proved most challenging across all models, while Tricky prompts were handled more reliably owing to their binary true/false structure. Falcon achieved perfect precision and zero hallucinations, but at the cost of the lowest recall and the highest response deviation. The results show a fundamental trade-off between response completeness and factual reliability, intensified by prompt complexity.

Submission history

From: Shahab Saquib Sohail PhD [

view email

]

[v1]

Mon, 19 Oct 2026 16:54:21 UTC (3,045 KB)