随笔 · 2026 年 7 月 30 日

你的模型早已知道答案

Francesco Moramarco • Elman 联合创始人

检验 AI 模型的最佳方式，是让它解决一个答案已知的真实世界问题；但这也是最容易作弊的测试。AI 模型读过互联网上的大量内容，因此，在你提问之前，它可能早已见过你的问题、答案，或者两者都见过。此时，高分掩盖了两种截然不同的情况：一种是模型通过推理得出答案，另一种则是模型在复述曾经读到的内容。区分二者，就是污染问题；它潜藏在该领域所依赖的相当一部分基准测试之下。

假设你想知道，一个模型能否可靠地预测某种药物是否会在临床试验中成功。与其等待数年以获得新的试验结果，不如测试那些已经结束的试验：选取进入后期试验的药物项目，隐藏每个项目是否成功的结果，并让模型进行预测。表面上看，这是一项干净的测试：快速，并且以真实结果评分；但事实并非如此：值得测试的试验往往恰恰是那些知名试验，其结果会反复出现在模型训练数据所包含的教材、综述、新闻和专利中。当模型判断一种药物将会成功时，评分无法区分模型究竟是通过推理得出这一结论，还是认出了药物名称并回忆起发生过什么。

临床试验是我们的领域，但这一陷阱具有普遍性：它会出现在任何由已解决事件构建的基准测试中，无论是金融、法律、预测，还是其他任何结果已经公开的领域。基准之所以从公开历史中构建，原因很充分：真实结果比任何专家能够编造的测试案例都更好，因为它反映了真实的生物学机制，以及药物在现实中失败的各种方式，而不仅仅是出题者想到要问什么。这些结果是我们希望据以评分的黄金标准，而它们的公开性也恰恰使其发生泄漏。那么，我们能做什么？

答案进入模型的三种路径

答案可以通过三条路径进入模型。以下划分是我们组织这一问题的方式，并非一套已命名的分类体系，不过每条路径都有既有研究基础。

图 1。 三种泄漏方式并列展示。每个面板中的红色标记，都是模型本应预测的答案。输入泄漏在测试时通过模型为该案例读取的文档进入模型。基准泄漏是指你自己的测试文件被纳入训练数据。结果泄漏则是模型在训练期间从论文、新闻和专利中学到公开结果。三者进入的都是同一个答案；区别只在于进入路径不同。

输入泄漏发生在测试时，来自模型为眼前案例读取的文档：无论是你将这些文档输入给模型，还是模型通过搜索工具自行获取。任何写于决策日期之后的文档，都可能公布模型本应预测的结果；这无异于让模型阅读未来的信息（Kaufman et al., 2012）。

基准泄漏发生在模型发布时间晚于你的基准测试时：模型可能已经在基准集本身上训练过，从而记住问题和答案。这是基准测试随时间推移而失效的最常见方式。Balloccu 及其同事追踪发现，在 GPT-3.5 和 GPT-4 发布后一年内，超过 400 万个测试样本泄漏进了这些模型（Balloccu et al., 2024）；而当其他研究者从头改写一组小学数学题时，多个模型家族在这一新版本上的得分显著降低（GSM1k, Zhang et al., 2024）。

结果泄漏发生在结果本身属于公开事实、且模型已在相关内容上训练过的情况下：一项完成的试验、一次监管批准，或一种有效药物的结果，会在数千篇论文和申报文件中反复出现；模型会在训练期间吸收这些信息，无论是在预训练中，还是在随后进行的微调中，且完全不需要接触你的特定基准测试。模型掌握该结果的方式，与掌握其他被广泛报道的事实并无不同，例如一场战争结束的年份。因此，你可以编写一套模型从未见过的全新测试集，并对提供给模型的每项输入进行日期盲化处理，但模型仍然会知道这款知名药物取得了成功。将基准保密同样无济于事，因为答案从一开始就不在你的文件中；它早已是模型对世界的一般知识的一部分。

等待新的结果有效，但结果可能要十年后才会出现

防止任何类型泄漏的最干净方法，是完全停止在历史事件上测试，只对尚未发生的事件进行模型评分，这样根本没有可泄漏的答案。实时基准测试会按照固定周期更新问题（LiveBench, White et al., 2024），而 ForecastBench 等预测基准会对测试时尚未得到解决的事件进行模型评分（ForecastBench, Karger et al., 2024）。CT Open 是一个实时平台，在临床试验结果公开前对模型评分：每轮挑战开启前就要提交预测；一条经专家标注验证的自动化流水线，会搜索每项结果最早的公开提及，因此，只有在作出预测时答案尚未在任何地方出现的试验才会被计入（Wang et al., 2026；ct-open.net）。试验结果全年陆续发布，CT Open 也相应地每年评分四次。我们关注的决策则更慢：一项耗资 10 亿美元的投入，通常会在确定结果的读出数据出现前约十年作出。因此，若把这类决策放进实时基准测试，你的第一个分数要到 2036 年才会出现。

因此，我们仍然受制于历史，以及随之而来的结果泄漏。这里有八种值得了解的方法。每种方法要么预防泄漏，要么检测泄漏。但更重要的划分是：一种方法针对三类泄漏中的哪一类。该领域大多数工作瞄准基准泄漏；但对于由真实世界结果构成的基准而言，最重要的是针对结果泄漏的三种方法。

图 2。 八种方法各自如何发挥作用，按其针对的泄漏类型分组。两种预防方法会在泄漏发生前阻止它；六种检测方法则在发生后将其发现。对于由已解决结果构成的基准，最下方一行的三种结果泄漏方法最为重要。

八种方法：按各自处理的泄漏类型分组

输入泄漏：从源头切断。

1. 按日期限制的工具（预防）：为模型提供搜索和检索工具，并拒绝任何发布于决策日期之后的内容，例如设置截止日期限制的网页搜索。预测系统已经采用这一方法，只允许检索在问题决策日期之前发布的新闻（Halawi et al., 2024）。这是唯一能够封堵输入泄漏的做法；由于它从源头起效，之后便没有任何东西需要检测。其效果取决于来源所携带日期的可靠性，因此在实践中，它依赖具有可靠发布日期的学术来源，而非开放网络。

基准泄漏：模型是否见过你的文件。

2. 新鲜或私有测试集（预防）：这是此处最干净的做法：测试模型未曾训练过的案例，或者让测试集保持未发布状态，使任何模型都无法在其上训练。一些较新的基准现在正采用这一方案，收集在每个模型训练截止日期之后发布的新问题，以确保模型不可能在测试集上训练过（LiveCodeBench, Jain et al., 2024）。

针对这类泄漏的另外三种方法，试图在事后进行检测。它们构成了该领域发展最成熟的一部分；然而对于一个基于公开结果的基准，它们却是八种方法中最没有用的，因为模型完全可能在从未见过你提出问题的特定文件的情况下，就已经知道结果。

3. 成员推断（检测）：通过检查模型对每个词的置信度——即序列中每个 token 的对数概率——来询问模型是否见过某段特定文本，因为模型在训练过的文本上会表现出异常的流畅性（Min-K%, Shi et al., 2023）。对我们而言，它有两方面弱点：单个文件几乎不会改变一个在数万亿词语料上训练的模型的统计特征，因此信号接近随机；而且，该方法需要模型逐词的概率数据，并非每个提供商都会公开这些数据（Anthropic 的 API 不返回这些数据，因而不得不采用基于采样的替代方案，例如 SaMIA, Kaneko et al., 2024）。无论如何，它只能告诉你模型是否读过你的文件，而不能告诉你模型是否知道答案。

4. 扰动（检测）：在保持正确答案不变的情况下改写基准问题，再检查准确率是否下降；因为显著下降意味着模型是在匹配记忆中的措辞，而非进行推理。GSM-Symbolic 通过替换数学题中的数字来实现这一点，并识别出那些只是看似掌握了题目的模型（Mirzadeh et al., 2024）。这种方法适用于改写后的问题仍有可推导答案的场景，例如上述数学题。但真实试验没有任何可供推导的内容：改变药物或患者群体，就描述了一项从未开展过的试验，因此没有人知道它会以何种结果结束。

5. 水印和金丝雀（检测）：在发布前向基准中植入隐藏标记，因此，如果模型日后复现该标记，就必然曾在该文件上训练；经典案例是埋藏在 BIG-bench 中的金丝雀字符串。和保持基准私有一样，这种方法只能保护尚未发布的基准；对于任何模型训练之前就已公开多年的结果，它毫无作用。

结果泄漏：模型是否已经知道结果。

最后三种方法对于由已经得到结果的案例构成的基准尤为重要，因为它们直接针对结果泄漏，而不是追问模型是否恰好见过某个特定文件。

6. 截止日期差距（检测）：比较模型在其训练截止日期之前已作出决定的案例，与训练截止日期之后作出决定的案例上的准确率。截止日期前表现强劲、截止日期后接近随机的模式，是记忆结果的特征，因为真正的能力不应在意案例位于某个日期的哪一侧。研究人员已经在金融预测以及与发布日期相关的知识上采用这种划分方法（Lopez-Lira et al., 2025；Roberts et al., 2023）。它是一种诊断方法，而不是一种设计选择；因此，它不同于仅对截止日期后的案例评分，也不同于对输入进行日期盲化处理。只有当两部分基准采用完全相同的构建方式时，这种方法才有效；否则，你测量到的可能是格式变化而非记忆。

7. 识别测试（检测）：通过三种变体判断模型是在依赖识别而非证据：不给任何支持材料、直接询问结果；隐藏实体名称但保留事实；或者只给出名称、不提供其他信息。真正进行推理的模型，应当在名称被隐藏时保持稳定、在证据被移除时无能为力；而依赖知名名称的模型则恰恰相反。在一项关于肿瘤学靶点的研究中，以这种方式隐藏名称后，模型约六分之一的首选结果发生改变，而它对已验证靶点的召回率并未变化（Cuccarese, 2026）。该论文同时提醒，这并不意味着盲化后的答案更好，因为基因名称本身携带真实的生物学信息；但如果不进行盲化，就无法判断答案中有多少来自页面上的数据，又有多少来自模型对名称的记忆。该数字来自四种适应症，每种仅运行一次。

8. 推理评分（检测）：忽略最终答案，转而评判其背后的论证；检查每项主张是否都建立在决策日期之前已存在的来源之上，因为仅仅记住结果的模型可以说出结果，却无法利用当时可得的证据构建带有日期的论证。我们自己的方法属于这一类，即 Receipts Test：我们要求模型出示其依据，利用决策日期时可获得的证据，给出一套可核查且带有日期的论证，并统计一个 corr

Essay · 30 July 2026

Your Model Already Knows the Answer

Francesco Moramarco • Co-founder, Elman

The best test of an AI model is a real world problem with a known answer. It is also the easiest test to cheat. An AI model has read much of the internet, so it may have met your question, the answer or both long before you asked. A high score then hides two very different things: a model that reasoned its way to the answer, and one that is repeating something it already read. Telling those apart is the contamination problem , and it sits under a surprising share of the benchmarks the field relies on.

Imagine you want to know how reliably a model can predict whether a drug will succeed in the clinic. Rather than wait years for fresh trials, you test on ones that have already finished: take drug programmes that reached late-stage trials, hide whether each worked, and ask the model to predict. On paper it looks like a clean test, fast and scored on real outcomes, but it is not: the trials worth testing are often the famous ones , and their outcomes recur across the textbooks, reviews, news and patents the model trained on. When the model says a drug will succeed, the score cannot separate a model that reasoned its way there from one that recognised the name and recalled what happened.

Clinical trials are our field, but the trap is general: it springs on any benchmark built from resolved events, in finance, law, forecasting or anywhere the outcome is already public. These benchmarks are built from public history for a good reason: a real outcome is a better test case than any expert can invent, because it reflects true biology and the real ways a drug fails, not just what a question-writer thought to ask. Those outcomes are the gold standard we want to score against, and being public is exactly what makes them leak. So, what can we do?

The three ways the answer gets in

The answer can reach the model by three routes. This split is our own way of organising the problem rather than a named taxonomy, though each route has an established basis.

Figure 1. The three leaks, side by side. In each panel the red mark is the answer the model is meant to predict. The input leak reaches the model at test time, inside a document it reads for the case. The benchmark leak is your own test file, pulled into the training data. The outcome leak is the public result, learned from papers, news and patents during training. It is the same answer in all three; only the route in is different.

Input leak happens at test time, through the documents the model reads for the case in front of it, whether you feed those documents in or the model fetches them itself with a search tool. Any document written after the decision date can announce the very outcome the model is meant to predict, which is as good as letting it read the future (Kaufman et al., 2012).

Benchmark leak happens when a model is released after your benchmark and the model might have been trained on the benchmark set itself, memorising both the questions and answers. This is the most common way in which a benchmark decays over time. Balloccu and colleagues traced more than four million test samples leaking into GPT-3.5 and GPT-4 within a year of their release (Balloccu et al., 2024), and when other researchers rewrote a set of grade-school maths problems from scratch, several model families scored markedly lower on the fresh version (GSM1k, Zhang et al., 2024).

Outcome leak happens when the outcome is itself a public fact and the model has trained on it: a completed trial, a regulatory approval or a drug that worked is repeated across thousands of papers and filings, and the model absorbs it during training, whether in pre-training or in the later fine-tuning that follows, with no contact of any kind with your particular benchmark. The model comes to hold the result in the same way that it holds any other widely reported fact, such as the year in which a war ended. As a consequence, you can write a completely fresh test set that the model has never seen and date-blind every input you give it, and the model will still know that this well-known drug succeeded. Keeping the benchmark secret does not help either, because the answer was never contained in your file in the first place; it was already part of the model’s general knowledge of the world.

Waiting for fresh outcomes works, until the outcome is a decade away

The cleanest way to prevent any type of leak is to stop testing on history altogether and to score the model only on events that have not yet happened, so that there is simply no answer available to leak. Live benchmarks refresh their questions on a fixed schedule (LiveBench, White et al., 2024), and forecasting benchmarks like ForecastBench score models on events that have not yet resolved at the time of the test (ForecastBench, Karger et al., 2024). CT Open is a live platform that scores models on clinical trials before their results are public: predictions go in before each challenge opens, and an automated pipeline, validated against expert annotation, searches out the earliest public mention of every outcome, so a trial counts only if its answer was nowhere to be found when the prediction was made (Wang et al., 2026; ct-open.net). Trial results land all year round, and CT Open scores four times a year to match. The decision we care about is a slower one: the commitment that costs a billion dollars is taken about a decade before the readout that settles it, so put that decision to a live benchmark and your first score arrives in 2036.

So we are stuck with history, and with the outcome leak that comes with it. That leaves eight methods worth knowing. Each one either prevents or detects a leak. The split that matters more, though, is which of the three leaks a method tackles. Most of the field’s work targets the benchmark leak, but for a benchmark built from real world outcomes, the methods that matter most are the three aimed at the outcome leak.

Figure 2. How each of the eight methods works, grouped by the leak it tackles. The two prevention methods stop a leak before it happens; the six detection methods catch one after. For a benchmark of resolved outcomes, the three outcome-leak methods on the bottom row carry the most weight.

The eight methods, grouped by the leak each one works on

Input leak: shut it at the source.

1. Date-gated tools prevent give the model a search-and-retrieval tool that refuses anything published after the decision date, for example a web search with a before-date limit. Forecasting systems already work this way, restricting retrieval to news published before the question’s decision date (Halawi et al., 2024). This is the one move that closes the input leak, and because it works at the source there is nothing left to detect afterwards. It is only as good as the dates its sources carry, so in practice it leans on scholarly sources with reliable publication dates rather than the open web.

Benchmark leak: whether the model saw your file.

2. Fresh or private test set prevent is the clean move here: test on cases no model has been trained on, or keep the set unpublished so that none can be. A number of newer benchmarks now take exactly this route, collecting fresh problems released after each model’s training cutoff so the test set cannot have been trained on (LiveCodeBench, Jain et al., 2024).

The other three methods for this leak try to detect it after the fact. They make up the most developed part of the field, and yet for a benchmark of public outcomes they are the least useful of the eight, because a model can know an outcome perfectly well without ever having seen the specific file in which you posed the question.

3. Membership inference detect asks whether the model has seen a specific piece of text, by checking how confident it is on each word (log probability of each token in the sequence), since a model is unusually fluent on text it trained on (Min-K%, Shi et al., 2023). For us it is weak on two counts: a single file barely moves the statistics of a model trained on trillions of words, so the signal is close to chance, and the method needs the model’s word-by-word probabilities, which not every provider exposes (Anthropic’s API returns none, forcing sampling-based workarounds like SaMIA, Kaneko et al., 2024). Either way it tells you only whether the model read your file, not whether it knows the answer.

4. Perturbation detect rephrases the benchmark question while keeping its true answer, then checks whether accuracy drops, since a large drop means the model was matching remembered wording rather than reasoning. GSM-Symbolic does this by swapping the numbers in maths problems and catches models that had only appeared to master them (Mirzadeh et al., 2024). It works where the reworded question still has an answer you can work out, as with those maths problems. But a real trial gives you nothing to work out: change the drug or the patient group and you have described a trial that was never run, so nobody knows how it would have ended.

5. Watermarks and canaries detect plant a hidden marker in the benchmark before release, so that a model which later reproduces it must have trained on the file; the classic is the canary string buried in BIG-bench. Like keeping a benchmark private, this only protects a benchmark you have not published yet, and does nothing about outcomes that were public for years before any model trained.

Outcome leak: whether the model already knows the result.

These last three are the ones that matter for a benchmark built from cases that have already resolved, because they aim at the outcome leak directly rather than at whether the model happened to see a particular file.

6. The cutoff gap detect compares the model’s accuracy on cases decided before its training cutoff with cases decided after it. Strong before and near-chance after is a signature of memorised outcomes, since real skill should not care which side of a date a case falls on. Researchers have run this split on financial forecasts and on knowledge tied to publication dates (Lopez-Lira et al., 2025; Roberts et al., 2023). It is a diagnostic, not a design choice, and so is different from only scoring post-cutoff cases or from date-blinding the inputs; it works only if both halves are built identically, or you measure a change in format instead of memory.

7. Recognition tests detect ask whether the model is running on recognition rather than evidence, in three variants: ask for the outcome with no supporting material, hide the entity’s name but keep the facts, or give only the name and nothing else. A model that truly reasons should be steady when the name is hidden and helpless when the evidence is removed, while a model riding on a famous name does the reverse. Blinding the name this way, in a study of oncology targets, changed about one in six of the model’s top picks while its recovery of validated targets was unchanged (Cuccarese, 2026); the paper’s caveat is that this does not make the blinded answers better, because a gene name carries real biology, but that without blinding you cannot tell how much of an answer came from the data on the page and how much from the model’s memory of the name. That figure comes from four indications with a single run each.

8. Grading the reasoning detect ignores the final answer and judges the argument behind it, checking that every claim rests on a source that existed before the decision date, because a model that only memorised the outcome can state it but cannot build a dated case for it. Our own method sits here, the Receipts Test: we ask the model to show its receipts, a dated and checkable case using only evidence available at the decision date, count a correct answer it cannot defend as a miss, track the gap between how often it is right and how often it can prove it, and let it answer “not yet knowable” instead of guessing. Grading reasoning traces against dated sources is established (process supervision, Lightman et al., 2023), and TimeSPEC breaks a prediction into atomic claims and reports how much of the decision rested on post-cutoff information (Zhang et al., 2026).

No method can prove a model is completely clean

No method can prove that a model is completely clean. Prevention comes closest: a date-gated tool is built to keep post-dated documents out, and a test set no model has trained on cannot leak through your file, so those two leaks can be designed out rather than merely caught. The outcome leak is the one you cannot close. There is no way to demonstrate that the weights gave no help at all, and you can only catch the cases where they plainly did. Each method also has its own blind spot, and even the two prevention moves are not airtight: date-gating is only as reliable as the publication dates it trusts, and a fresh test set does nothing about an outcome the model already absorbed from the wider literature. This is not only our experience. An independent effort to train a language model on pre-1931 text alone, with temporal leakage named as its most important objective and a purpose-built anachronism filter enforcing it, still produced a model that knows the Second World War happened (Talkie, Levine et al., 2026). A decontamination filter leaks even when keeping the data clean is the whole point. Membership inference is weak at scale, and watermarks protect the file rather than the public outcome. Perturbation depends on an answer that can be regenerated, which a real trial does not provide, and the cutoff gap collapses into a measurement of formatting if the question format is allowed to drift across the boundary. Blinding the name flags a shift that genuine biology would also produce, and grading the reasoning can be defeated by a fluent argument constructed backwards from an answer the model already knew. For these reasons we do not rely on any single method, but run several of them together and pay particular attention to the cases where they disagree, which is what turns a number one cannot trust into a number one can defend. For a benchmark built from resolved outcomes, the three methods worth their weight are the ones aimed at the outcome leak.

We are building a benchmark that must survive every leak

The benchmark we are building for drug-development outcomes has to survive every one of these leaks at once, because the cases that are most worth scoring are precisely the famous ones that a model is most likely to have memorised, and waiting a decade for fresh outcomes is not a realistic option.

We are not alone here. CT Open avoids the outcome leak by going live (Wang et al., 2026). TimeSPEC grades the trace (Zhang et al., 2026). MemGuard-Alpha filters memorised signals out of a financial backtest (Roy and Roy, 2026). All three score whether the prediction was right; none scores why the trial turned out as it did. A verdict is not something a drug developer can act on. The mechanism is, because the mechanism is what you would change: the target, the cohort, the biomarker you select patients on. So that is what we are scoring, on a bet we would rather test than assert, that a model can memorise a verdict far more easily than a mechanism it never had to reason about.

If you build or evaluate models on historical data, in drug discovery or anywhere else, tell us what catches memorisation and what slips through, especially where it contradicts what we have written here. And if you hold outcome data with accurate dates attached, that is the raw material a benchmark of this kind is made of. We would love to hear from you!

Discuss this on Hacker News →

Get the next post

New writing from Elman on drug-development AI and what we can prove about a model. No more than one email a month.

One click to unsubscribe, any time. We store only your email, nothing else. See our privacy note.

Thanks. Check your inbox and click the link to confirm your subscription.

Francesco Moramarco

Co-founder at Elman Bio · francesco@elman.ai

Thanks to Moustafa Khedr for review.