摘要：norregel 评测语言模型在挪威成文法、行政法规和议会决议中指定日期下的数值信息（挪威克朗金额、百分比、国民保险基本数额 (G) 的倍数以及年龄限制）。这 916 个测试条目全部机械提取自 Lovdata 的开放数据，且每一项都附带有对应法规条文的带时效修订历史，因此模型的回答可被分类为：正确（correct）、过时（outdated）、错误（wrong）或无法解析（uninterpretable）。我们报告了针对相同条目开展的第一次与第二次预注册评测运行。在第一次运行中（评测了 5 个模型，仅在本地分析完成后公开），假设 H1（即在模型知识截止日期之后生效的数值，出现过时回答的频率更高）在实质上无法进行检验，且假设 H2 与 H3 未得出明确结论；Opus 5.5 与 Sonnet 4.6 在 880 个主要测试条目中的正确率分别为 46.7% 和 40.2%。第二次运行在进行任何 API 调用前便完成了预注册与公开发布，共对 8 个模型进行了分析。在其汇总的主要检验中，按照预注册的判定规则支持了假设 H1（$d = 17.8$ 个百分点，95% 置信区间 $[7.7, 28.9]$，$p = 0.0001$）。但在预注册的敏感性分析 S8 中（该分析仅将“与标准答案不同且最接近最近生效的数值”算作过时），$d$ 为 3.9 $[-3.6, 12.7]$（$p = 0.1754$），该区间包含了 0。尽管 H1 在其他所有预注册敏感性分析中均得到支持，但鉴于 S8 是唯一给两组均设定单一目标值的分析，我们并不认为对 H1 的支持具有稳健性。过时答案中等于该最新数值的比例（H2）为 56.1% $[46.6, 65.0]$，该区间包含 0.5；而关于新挪威语（Nynorsk）正确率低于书面挪威语（Bokmål）的汇总检验（H3）则无明确结论。包括 Haiku 4.5 在内的若干模型给出的回答大部分属于无法解析。测试题目与标准答案暂未公开；条目 ID、每个回答的分类结果、代码与评测结果已发布在 Zenodo 上。

Abstract:norregel asks language models for numeric values in Norwegian statutes, regulations and parliamentary decisions (NOK amounts, percentages, multiples of the National Insurance basic amount (G) and age limits), each at a stated date. The 916 items were extracted mechanically from Lovdata's open data, and each carries the dated history of its provision, so an answer can be classed as correct, outdated, wrong or uninterpretable. We report a first and a second preregistered run on the same items. In the first run (five models, listed publicly only after local analysis), H1, that outdated answers are more frequent for values that took effect after a model's cutoff, could not be tested in substance, and H2 and H3 were inconclusive; Opus 5.5 and Sonnet 4.6 answered 46.7% and 40.2% of the 880 primary items correctly. The second run was preregistered and published before any of its calls, and eight models were analysed. Its pooled primary test supported H1 under the preregistered decision rule (d = 17.8 percentage points, 95% interval [7.7, 28.9], p = 0.0001). In the preregistered sensitivity analysis S8, which counts as outdated only the most recent value that differs from the key, d was 3.9 [-3.6, 12.7] (p = 0.1754), an interval that contains 0. H1 was supported in every other preregistered sensitivity analysis, but S8 is the only one that gives both groups one target value, so we do not regard the support for H1 as robust. The share of outdated answers equal to that most recent value (H2) was 56.1% [46.6, 65.0], an interval that contains 0.5, and the pooled test of a lower share correct in Nynorsk than in Bokmål (H3) was inconclusive. Several models, among them Haiku 4.5, gave mostly uninterpretable answers. Items and keys are withheld; item IDs, per-answer classes, code and results are on Zenodo.