评估科学应用中的语言模型：以惯性约束聚变（ICF）为例
英文标题：Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
本研究以高专业度的惯性约束聚变（ICF）为测试场景，构建了包含 250 道题目的专家级科学评测基准，全面考察定性物理推理、数值计算、符号推导与公式构建。评测对比了主流开源与闭源前沿大模型以及经领域预训练与微调的专用模型 Prospero；结果显示专用模型 Prospero 相比基础模型提升了 10.7 个百分点，前沿通用模型虽综合得分最高，但在数值计算等关键物理环节仍存在严重错误。此外，检索增强（RAG）能有效辅助较小模型，但对前沿模型未带来统计学显著增益，研究建议在投入高昂成本进行科学领域专属微调前，应优先评估现有前沿通用能力与检索增强的实际效果。
2.0 KB
612 B
380 B
24.2 KB
34.7 KB
中文
[提交于 2026 年 10 月 27 日]
标题：评估科学应用中的语言模型：以惯性约束聚变（ICF）为例
作者：P. B. Radha, D. D. Meyerhofer, M. Bhattarai, B. Cheng, L. Diaz, C. L. Rousculp, A. G. Seaton, G. Gupta, P. Gurumurthy, V. Gutta, A. Jhunjhunwala, A. Vem, P. Cook, G. Rowe
查看 P. B. Radha 及其他 13 位作者撰写的论文《评估科学应用中的语言模型：以惯性约束聚变为例》（Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case）PDF 版本
摘要：推动语言模型在专业科学领域的应用，需要对其科学语言表达流畅度、物理推理能力以及定量分析水平进行系统性评估。我们以惯性约束聚变（Inertial Confinement Fusion, ICF）作为测试范例，构建了一个由专家精心策划的基准数据集，包含 250 道涵盖定性推理、数值计算、符号推导以及方程构建的专业问题。模型生成的回答结合了确定性数值与符号验证机制，并辅助以语言模型裁判（LLM-as-a-judge）进行综合打分。我们对比了多种开箱即用的闭源和开源大语言模型，以及通过结合领域自适应预训练与监督微调针对 ICF 进行专门优化的专用模型 Prospero。同时，我们还开展了初步的检索辅助（Retrieval-assisted）评测。结果表明，Prospero 的综合得分相较于其基础模型提升了 10.7 个百分点，且在所有题型类别中均实现了性能增长。前沿模型（Frontier models）获得了最高的综合评分，但关键物理错误仍然普遍存在，尤其是在数值计算方面。我们指出，虽然通过领域特定适配可以提高模型在该领域的表现，但仅靠基准评测分数的提升并不能确立其在科学研究中的可靠性。检索增强提升了较小规模模型的性能，但对前沿模型未产生统计学上显著的改善。两项科学压力测试进一步展现了模型的实用潜能，同时也突显出由领域专家评估物理假设与模型有效性的必要性。基于这些发现，我们建议在投入大量资源进行深度领域专门化研发之前，应首先充分评估模型现有的通用领域能力以及检索增强带来的实际收益。
提交历史
来自：Radha Bahukutumbi [查看邮箱]
[v1] 2026 年 10 月 27 日 星期二 22:21:10 UTC (10,697 KB)
English / 原文
[Submitted on 27 Oct 2026]
Title:Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case
Authors:P. B. Radha
,
,
,
,
,
,
,
,
,
,
,
,
,
View a PDF of the paper titled Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case, by P. B. Radha and 13 other authors
Abstract:Advancing language models for specialized scientific applications requires evaluations of scientific fluency, physical reasoning, and quantitative analyses. Using inertial confinement fusion (ICF) as a test case, we have developed an expert-curated benchmark of 250 questions spanning qualitative reasoning, numerical calculation, symbolic derivation, and equation formulation. Responses have been evaluated using deterministic numerical and symbolic checks together with language-model judges. We compared a variety of out-of-the-box closed-source and open-source large language models with Prospero, a model specialized through combined domain-adaptive pretraining and supervised fine-tuning for ICF. We also conducted initial retrieval-assisted evaluations. Prospero improved its aggregate score by 10.7 percentage points over its base model, with gains across all question types. Frontier models achieved the highest aggregate scores, but critical physics errors persisted, particularly in numerical calculations. We note that domain-specific performance can improve through specialization, but benchmark gains alone do not establish scientific reliability. Retrieval improved performance for a smaller model but yielded no statistically significant improvement for a frontier model. Two scientific stress tests further illustrate useful capabilities and the need for expert assessment of physical assumptions and model validity. Based on these findings, we recommend evaluating existing domain capabilities and retrieval benefits before committing to substantial additional specialization.
Submission history
From: Radha Bahukutumbi [
]
[v1]
Tue, 27 Oct 2026 22:21:10 UTC (10,697 KB)