🔶 Auto-autoresearch：在 Karpathy 的 NanoChat 基准上自我改进的 Agent

如果有人突然给了你一大批研究生，你会如何组织他们高效开展研究？这是我们在 rekursiv.ai 一直思考的问题。

AI Agent 在许多方面都像“天才实习生”：它们拥有天才级别的知识，却不知道什么最重要。我们没有直接正面解决这个问题，而是思考：“如果能让这些天才实习生自己找出最优策略，会怎样？”

继 ARC-AGI 和 数独 项目之后，我们给 AI 科学家团队布置了一项新挑战：在单张 B200、5 分钟训练预算内改进一个语言模型。如基准对比所示，最新配方在 Karpathy 的 NanoChat 基准上、跨 10 个随机种子的平均结果达到 0.887791 每字节比特数（BPB）。

每字节比特数（BPB）↓ · 越低越好

图 2. NanoChat 基准：rekursiv.ai 的实验结果，取 10 个随机种子的平均值，并与已报告的参考分数并列比较。我们的每次运行均在单张 B200 上使用 300 秒（5 分钟）的计费训练时间。BPB 越低越好。

在 3 天内，Agent 发起了 6,164 次实验。它们探索了广泛的思路，包括训练数据、模型架构、分词器以及定制 Triton 内核的改进。这 3 天并不只是在改进 NanoChat，系统也在学习如何做研究：修订指令、修复交接流程，并改变 Agent 测试和建立在彼此想法之上的方式。我们明确要求系统设计新策略，以弥补自身弱点，并在每一轮迭代中更快地开展研究。

这在很大程度上得益于 Trackinizer：这是我们开源的定制图数据库，用于追踪信念、实验和分析的谱系。随着时间推移，AI 研究团队将丰富的实验想法与结果编织进这张图中，最终产出最终结果。

最终配方已在 GitHub 上开源。欢迎你在 GitHub 上探索我们的工具，或加入我们的 Discord 进一步讨论。

为什么选择 NanoChat？

NanoChat 为我们提供了短实验周期，同时训练流程中几乎每个部分都可以调整。更快的内核可以在预算内容纳更多更新；不同的架构可以从每次更新中学到更多；新的分词器可以改变模型上下文中可容纳的文本量。团队必须在完整训练循环中测试每一个想法，才能判断它是否有效。

我们在元框架（meta-harness）之上，以多轮研究活动的方式处理该问题，每一轮研究持续 2 小时。当一轮结束后，系统会使用前几轮提出的新框架重新启动。研究团队还会将重要发现和实验结果传递给下一批团队，以便并行探索。图 3a 展示了各轮活动取得的进展，图 3b 则展示了按总实际耗时计算的进展。

图 3a. 我们的研究活动随时间变化的散点图。每个点代表 NanoChat 上的一次 5 分钟实验，线条代表每个活动的帕累托前沿。颜色渐变直观表示实际耗时，展示重启后的活动如何从此前活动停止的位置继续推进。

图 3b. 相同运行按完成时间排序。早期，AI 科学家发起大量实验，以探索一个巨大的思路空间。随着时间推移，知识图谱积累了大量失败实验的记录，快速取得进展变得更加困难。此后发生了阶段转换：如图中单位时间实验数量逐渐下降所示，Agent 决定在发起新实验前投入更多时间思考更具新颖性的想法。

我们发现，尽管最终评估挑战确实需要 B200 硬件，但 Hopper GPU 同样可以很好地完成研究。因此，在转向成本更高的硬件之前，我们决定使用 H200 代理完成绝大多数研究活动。我们先在一个 H200 集群上进行了两天研究，整个过程跨越两天半，随后再转移到单张 B200 上进行半天训练。

为后人栽下核桃与梨树（前人种树，后人乘凉）

在 rekursiv.ai，我们的另一句格言是“速通科学”。我们努力为 Agent 构建最高效的协作框架，使其能够在彼此结果的基础上持续爬坡优化。分享一项发现只是开始。下一位科学家还需要理解测试，并决定该改动什么。图 4 中的三元组时间线展示了一个例子：

从一次失败的三元组实验到六项新活动

记住 token 对已经有所帮助。记住 token 三元组会更有帮助吗？2026 年 9 月 7 日 · UTC

1 · 失败设计 2 · 不同实现 3 · 代码交接 4 · 证据就绪 5 · 独立认定 6 · 共享起点

1 · 一种投影设计失败

团队通过一个狭窄的可学习投影，为 token 三元组加入记忆能力，希望在不使用全宽表的情况下延续此前二元组带来的收益。第一次运行在随机种子 42 上表现变差，尽管其完成的更新次数几乎与父实验相同。证据工作者保留了实现和负面结果：这否定的是经过测试的设计，而不是所有三元组记忆方案。

BPB（越低越好）：0.9751，对比父实验 0.9686

2 · 负责人提出另一项测试

负责人将该结果与此前一次投影二元组失败结果一并分析。也许额外上下文确实有用，但狭窄投影阻碍了效果？下一个提议移除该投影，并在一个经过新正则化的父实验上、以新的规模加入全宽三元组。这是一套修订后的配方，而不是仅针对投影的消融实验；此前的失败仍然是证据的一部分。

新父实验平均 BPB：0.96343 · 候选方案尚未运行

3 · 经测试的代码跨越角色边界

证据工作者准备实现，并检查其因果查找、零初始化和优化器归属。负责人接受经测试的代码；集成者随后将其适配进共享软件包并提交实验。交接传递的是源代码和检查结果，而不只是一个想法。通过 CPU 检查并不等同于获得 GPU 质量结果。

候选 BPB：尚未测量

4 · 三个匹配随机种子均有提升

在一次有希望的首轮运行后，团队保留结果，并针对完全相同的父实验再测试两个随机种子。三个种子全部提升。证据工作者将源代码、配置、任务回执和各随机种子结果核对并汇总为一个面板，供其他 Agent 检查。证据已经就绪，但独立认定仍在等待。

平均 BPB：0.96343 → 0.95987 · 3 个种子全部提升

5 · 协调者进行独立认定

协调者独立检查原始三元组实验并接受该结果，但不会将这一决定追溯记为证据就绪的时间。该配方现在可以进入下一次交接。模型改进有匹配运行作为支持；但先前接触过同行摘要以及一次运营层面的干预，仍限制了我们对“哪种团队组织方式导致该发现”的主张。

经认定的平均 BPB：0.95987

6 · 六项活动共享起始配方

在 04:48:22 至 04:49:31 之间，六项活动从共享配方启动，其中包含其设置、匹配结果和失败历史。每次运行都重新初始化权重；不会转移已训练的检查点。下一批研究者可以追问如何在这个起点上进一步改进，而不必重复同样的搜索。选择和准备这一起点，是协调者在整个过程中的贡献。

继承配方的平均 BPB：0.95987 · 并非六项新结果

图 4. 在不到 90 分钟内，一种修订后的三元组设计成为六项活动的起点。该时间线追踪了失败实验、修订实现、评估、独立审查，以及向下一波研究的交接。

交接首先要找到正确的实验。在 Trackinizer 中，科学家可以搜索其节点，并沿着边追溯相关假设、实现、证据和后续工作。图 5 展示了这些活动之间的连接。下一位研究者可以追溯某项实验为何运行，以及其结果支持什么结论。

• 问题
• 工件
• 实验
• 信念
• 代码变更

图 5. Agent 团队构建的知识图谱子集。探索它们的想法、实验和发现之间的联系。

然后，交接必须可运行，而不只是可阅读。

为了促进可靠的机器学习科学，我们构建了 Priml：一个可组合的 ML A/B 测试框架，提供可复用的训练和评估组件。它完全由 Configgle 驱动，后者允许实验继承一个具名父配置，并修改相关组件或设置。二者结合，使人类和 AI 研究者能够在共同实验基础上继续工作，而不是重新搭建彼此的训练流水线。我们希望下一位科学家能够找到证据、复现实验设置，并测试一项改动，而无需重建其背后的工作。

在三元组案例中，修订组件保留了其周围的父配方；团队检查了实现，并比较了匹配的随机种子。共享基础并没有消除这些检查的必要性。随后，协调者将审查过的代码、配置和发现打包，用于六项新活动，将继承的进展与每个团队的新贡献区分开来。

AI 元协调者还改变了活动的运行方式。在一次审查中，一名 AI 审查者发现，工程师被要求将结果返回给会话可能已经结束的研究人员。协调者修订提示词，要求将迟到的结果通过主管和持久化记录进行路由。研究历史成为下一团队指令的输入，而不只是其训练配方的一部分。

交接内部：角色、控制、回执与证据

多个团队将工作分配给首席科学家、集成者和证据工作者。首席科学家选择实验并记录预期改动。集成者负责共享实现和 GPU 提交。证据工作者检查实现和结果，然后报告它们支持什么结论。Agent 通过其活动对应的 Trackinizer 房间交换消息，标明实验名称并确认交接。一条结果消息可以指向已存储的审计记录，供下一个 Agent 检查。独立研究者则自行履行相同职能。

记录包含的不只是分数：

• 研究意图会标明父配方、拟议改动、预测、控制条件，以及当实验失败时 Agent 将采取什么行动。
• 实现和证据工件记录源版本、检查、任务回执、原始日志引用、各随机种子分数和时间信息。它们将初步结果与经过独立审查的结果分开保存。
• 实验记录总结配置和结果。信念记录陈述结论，并链接支持或反驳这些结论的证据。
• 活动交接汇集当前配方、失败方向、开放问题和仍等待审查的结果。研究计划和提示词修订与科学工作一同存储。

这些记录也指导了指令的修订。当一位研究者错误地排除了自定义组件后，后续简报明确了许可范围。其他变更则澄清了截止时间决策、运行时检查和发布所有权。我们可以看到新指令解决了哪些问题，但仅凭这一点还无法说明它们是否带来了更好的研究。

指令发生了什么变化，以及之后发生了什么

下一轮所需的不只是获胜配方，也需要关于研究人员可以尝试什么的更清晰指令。在早期的二元组搜索中，一位研究者错误地认为它可以修改配置，但不能添加组件。在得到明确纠正后，它在数分钟内提交了一个哈希二元组实验。首个面板的 BPB 为 0.97990，而父实验为 0.98543；后续实验达到 0.97110。此后的简报明确了这一许可。

• 截止时间变成了一项决策，而非自动停机。 一些 Agent 将第 105 分钟理解为硬性停止点。修订后的指令要求评估暂存、训练、评估和记录能否在实际截止时间前完成。研究者随后接受了 4 个按时完成的任务。它们的结果没有改进现有最佳方案；收益是完成了有用的测试，而不是已证明的分数提升。
• 补充任务指令改变了具体行动。 提示词不再只是要求 GPU 保持忙碌，而是要求在长时间等待前，先检查是否有已审查且可运行的工作。一项活动还组合了兼容的记录操作：观察到 43 条多标签命令行，而其两个对照组分别为 0 条和 3 条。它接受了 106 个任务，而对照组为 99 个和 92 个，但并未取得最佳最终结果。
• 运行时追踪捕捉到了无效实验。 修订后的指令要求从设置工厂一路追踪设置，经过覆盖逻辑直到消费该设置的操作。一位研究者发现了重复的优化器归属，以及请求的迭代次数与实际使用的系数不匹配的问题。它还推翻了一项错误否决。新发布的 4 项实验全部输给现有最佳方案：更好的检查并不一定会立即降低 BPB。
• 发布获得了明确责任人。 指定发布者必须进行解析检查、原子发布，并读回确切的 ev

If someone suddenly gave you large team of grad students, how would you organize them to do research effectively? This question has been on our minds at rekursiv.ai.

AI agents function in many similar ways to “intern savants”: they possess genuis-level knowledge, but they don’t know what’s important. Instead of approaching the problem head on, we pondered: “what if we could get our intern savants to figure out the best strategy for us?”.

After our ARC-AGI and Sudoku campaigns, we gave our team of AI Scientists a new challenge: improve a language model within a five-minute training budget on a single B200. The latest recipe reaches 0.887791 mean bits per byte (BPB) across 10 seeds on Karpathy’s NanoChat benchmark, as shown in the benchmark comparison.

Bits per byte (BPB) ↓ · lower is better

Figure 2. NanoChat benchmark: rekursiv.ai's experimental result averaged across 10 seeds, alongside reported reference scores. Each of our runs used 300 seconds (five minutes) of charged training on one B200. Lower BPB is better.

In three days, the agents launched 6,164 experiments. They explored a wide variety of ideas, spanning improvements to training data, architecture, tokenizers, and custom triton kernels. Those three days weren’t spent only improving NanoChat. The system was also learning how to do research: revising instructions, repairing handoffs, and changing how agents tested and built on one another’s ideas. We gave explicit directives to our system to devise new strategies to cover for its weaknesses and do research faster on each iteration.

This was enabled in large part because of Trackinizer, our open-source custom graph database that tracks the lineage of beliefs, and experiments, and analyses. Over time, the AI research team wove their rich tapestry of experimental ideas and results into the graph to produce their final results.

The final recipe is open source on GitHub. We encourage you to explore our tools on GitHub, or join our Discord for further discussion.

Why NanoChat?

NanoChat gave us a short experiment cycle with room to change almost every part of training. A faster kernel could fit more updates into the budget, while a different architecture could learn more from each update. A new tokenizer could change how much text fit into the model’s context. The team had to test each idea in the full training loop to find out whether it helped.

We approached the problem using waves of campaigns on top of our meta-harness, each research wave lasting 2 hours long. When a wave completed, it would restart with a new harness proposed by the previous waves. Research teams also passed important findings and experimental results to the next teams to explore in parallel. Figure 3a shows the progress made across the waves, while Figure 3b shows the total wall-clock progress.

Figure 3a. A scatterplot of our research campaigns over time. A point represents each 5-minute experiment on Nanochat, and lines represent the pareto frontier of each campaign. The color gradient visually indicates wall-clock time, showing how restarted campaigns pick up where previous campaigns left off.

Figure 3b. The same runs ordered by completion time. Early on, the AI Scientists launched many experiments to explore a large space of ideas. As time went on, the knowledge graph accumulated a repository of failed experiments, making it more difficult to make quick progress. A phase shift happened, and the agents decided to spend more time to think of more novel ideas before launching new experiments, as shown by the tapering off of experiments per unit time in the plot.

We found that, while the challenge did require B200 hardware for the final evaluation, Hopper GPUs could work just as well. Therefore, we set out to use H200s as a proxy for the majority of the research campaigns before transferring over to the more expensive hardware. Two days were spent on a cluster of H200s over two and a half days, then transferred another half day training on a single B200.

Walnuts and pears you plant for your heirs (前人種樹，後人乘涼)

At rekursiv.ai, another motto we have is to “speedrun science.” We strive to build the most efficient harness for our agents to collaborate and hill-climb on each other’s results. Sharing a finding is only the beginning. The next scientist needs to understand the test and choose what to change. The trigram timeline in Figure 4 shows one example:

From a failed trigram to six new campaigns

Remembering token pairs already helped. Would remembering triples help more? 7 September 2026 · UTC

1 · Failed design 2 · Different implementation 3 · Code handoff 4 · Evidence ready 5 · Independent qualification 6 · Shared starting point

1 · One projected design fails

The team adds memory for token triples through a narrow learned projection, hoping to extend the earlier bigram gain without a full-width table. The first run loses on seed 42, despite completing almost as many updates as its parent. The evidence worker keeps both the implementation and the negative result: this rejects the tested design, not all trigram memory.

BPB (lower is better): 0.9751 vs parent 0.9686

2 · The lead proposes a different test

The lead reads this result alongside an earlier projected-bigram failure. Perhaps the extra context is useful, but the narrow projection is getting in its way? The next proposal removes that projection and adds full-width trigrams on a new regularized parent, at a new scale. This is a revised recipe, not a projection-only ablation; the earlier failure remains part of the evidence.

New parent mean BPB: 0.96343 · candidate not yet run

3 · Tested code crosses the role boundary

The evidence worker prepares the implementation and checks its causal lookup, zero initialization, and optimizer ownership. The lead accepts the tested code; the integrator then adapts it into the shared package and submits the experiment. The handoff carries source and checks, not just an idea. Passing CPU checks is not a GPU quality result.

Candidate BPB: not yet measured

4 · All three matched seeds improve

After a promising first run, the team retains its result and tests two more seeds against the exact parent. All three improve. The evidence worker reconciles the source, configuration, job receipts, and per-seed results into a panel that another agent can inspect. The evidence is ready, but independent qualification is still pending.

Mean BPB: 0.96343 → 0.95987 · 3 of 3 improve

5 · The coordinator independently qualifies

The coordinator independently checks the original triplet and accepts the result, without backdating that decision to when the evidence became ready. The recipe can now enter the next handoff. The model improvement is supported by matched runs; prior peer-summary exposure and an operational intervention still limit claims about which team organization caused the discovery.

Qualified mean BPB: 0.95987

6 · Six campaigns share the starting recipe

Between 04:48:22 and 04:49:31, six campaigns launch from the shared recipe, with its settings, matched results, and failed history. Every run initializes weights fresh; no trained checkpoint is transferred. The next researchers can ask what improves on this starting point instead of repeating the same search. Choosing and preparing that starting point is the coordinator’s part of the story.

Inherited recipe mean BPB: 0.95987 · not six new results

Figure 4. A revised trigram design became the starting point for six campaigns in less than ninety minutes. The timeline follows the failed experiment, revised implementation, evaluation, independent review, and handoff to the next wave.

The handoff starts with finding the right experiment. In Trackinizer, a scientist can search for its node and follow the edges to related hypotheses, implementations, evidence, and later work. Figure 5 shows these connections across the campaigns. The next researcher can trace why an experiment was run and what its result supports.

• Issue
• Artifact
• Experiment
• Belief
• Code change

Figure 5. A subset of the knowledge graph built by the agent teams. Explore the connections between their ideas, experiments, and findings.

Then the handoff needs to be runnable, not just readable.

To facilitate reliable ML science, we built Priml, a composable ML A/B test harness with reusable training and evaluation components. It’s all powered by Configgle, which lets an experiment inherit a named parent configuration and change the relevant components or settings. Together, they let human and AI researchers build on a common experimental base rather than reconstruct one another’s training pipelines. We want the next scientist to be able to find the evidence, reproduce the setup, and test a change without rebuilding the work behind it.

In the trigram story, the revised component preserved the parent recipe around it; the team checked the implementation and compared matching seeds. A shared base did not remove the need for those checks. The coordinator then packaged the reviewed code, configurations, and findings for six new campaigns, separating inherited progress from each team’s new contribution.

The AI meta-coordinator also changed how the campaigns worked. In one review, an AI reviewer found that engineers were told to return results to researchers whose sessions might already have ended. The coordinator revised the prompts to route late results through the supervisor and durable records. Research history became input to the next team’s instructions, not just its training recipe.

Inside the handoff: roles, controls, receipts, and evidence

Several teams divided the work among a lead scientist, an integrator, and an evidence worker. The lead chose experiments and wrote down the intended change. The integrator owned the shared implementation and GPU submissions. The evidence worker checked implementations and results, then reported what they supported. Agents exchanged messages through a Trackinizer room for their campaign, naming the experiment and acknowledging handoffs. A result message could point to a stored audit that the next agent could inspect. Solo researchers performed the same functions themselves.

The records contained more than scores:

• Research intents named the parent recipe, proposed change, prediction, controls, and what the agent would do if the experiment failed.
• Implementation and evidence artifacts recorded source versions, checks, job receipts, raw-log references, per-seed scores, and timing. They kept preliminary results separate from independently reviewed results.
• Experiment records summarized configurations and outcomes. Beliefs stated the conclusions, with links to supporting or contradictory evidence.
• Campaign handoffs collected the current recipe, failed directions, open questions, and results still waiting for review. Study plans and prompt revisions were stored alongside the scientific work.

Those records also guided revisions to the instructions. When one researcher mistakenly ruled out custom components, later briefs made the permission explicit. Other changes clarified deadline decisions, runtime checks, and publication ownership. We could see which problems the new instructions addressed, but that alone did not tell us whether they produced better research.

What changed in the instructions, and what happened afterward

The next wave needed more than the winning recipe. It also needed clearer instructions about what researchers were allowed to try. Earlier in the bigram search, one researcher mistakenly believed it could change configuration but not add components. After an explicit correction, it submitted a hashed-bigram experiment within minutes. The first panel scored 0.97990 BPB against a 0.98543 parent; later experiments reached 0.97110. Subsequent briefs made the permission clear.

• A deadline became a decision, not an automatic shutdown. Some agents interpreted minute 105 as a hard stop. The revised instruction asked whether staging, training, evaluation and recording could finish before the actual deadline. The researcher then admitted four jobs that finished on time. Their results did not improve its incumbent; the benefit was completing useful tests, not a demonstrated score gain.
• Refill instructions changed concrete actions. Instead of merely asking for busy GPUs, prompts required checking for work that had been reviewed and was ready to run before a long wait. One campaign also composed compatible recording operations: 43 observed multi-label command lines, versus zero and three in its comparators. It accepted 106 jobs versus 99 and 92, but did not have the best endpoint.
• Runtime tracing caught ineffective experiments. Revised instructions required following a setting from its factory through overrides to the consuming operation. A researcher caught duplicate optimizer ownership and a requested iteration count that did not match the coefficients actually used. It also overturned a false veto. The four newly released experiments all lost to the incumbent: better checking need not immediately lower BPB.
• Publication gained an explicit owner. The designated publisher had to parse-check, atomically publish and read back the exact evidence revision. Moving that role from integrator to analyst coincided with a shorter median publication delay, 187 to 142 seconds. The analyst-publication campaign nevertheless submitted fewer jobs and ended with a worse score.
• Recovery instructions preserved the original operation. Prompts added receipt-first checks and a single owner for stalled submissions. That helped recover blocked work without duplicate submissions, but it did not guarantee timely coordinator service. A later review gap left 35 completed panels unqualified.

Keep humans in the loop

Improving the instructions did not remove the need for human oversight. We set study priorities, including team size and role organization, and could pause the search. In a review of 48 launched campaigns , 35 completed panels were found waiting for qualification after a service interruption. The same records that enabled the trigram handoff exposed a limit: evidence needs someone responsible for checking and passing it on. The coordinator’s own review and handoff process had to be part of what we improved.

Team size, context refresh, and inherited progress

The trigram team accepted 65 jobs , while a solo researcher accepted 112 in the same wave. The team nevertheless found the stronger recipe. In a later comparison, a solo researcher had the best incremental improvement and final score. Neither job count nor team size told us who would make the next useful discovery.

• A two-researcher team reached the original cohort’s 0.95610 qualified endpoint, while several larger teams cost more without consistently reaching better endpoints.
• Fifteen of sixteen researchers refreshed every thirty minutes still compacted. Persistent researchers also continued producing useful experiments after compaction.
• In one retrieval-policy cohort, inherited-recipe availability explained 97.79–100% of control-relative improvement area. Earlier review was not the same thing as discovering a better recipe.

Two early branches starting from the same packet reached qualified scores of 0.965700 and 0.985567. Learned bigram memory opened a larger gain than nearby optimizer refinements. With only two runs per model, different provider stacks, and human interventions, this did not establish a general ranking of models or team structures.

Where completed work waited before the next team could use it

Completed experiments could still wait a long time before other teams could use them. One candidate’s last required run finished 49.7 minutes before qualification. The timestamps did not separate active review from publication or service wait, so we recorded completion and qualification as different events.

GPU utilization told only part of this story. Median accepted-run turnaround was about 6.00–6.37 minutes across one set of workflow comparisons, while allocated GPU fractions ranged from 8.80% to 68.14%. We did not measure whether a useful, valid candidate was ready in every gap, so we could not count all unallocated time as avoidable idle time.

All of this comes back to a question: What should we test next, and what would make us change our mind? Keeping that question useful gets harder as experiments accumulate and researchers hand work to one another. The meta-campaign worked on those handoffs and instructions. In NanoChat, the decisions looked like this.

Behind 0.88779: the good, the bad, and the beautiful

There was no single discovery that made the final recipe. Across data, capacity, memory, and execution, the team had to decide which observation deserved another test and which result called for a different approach. The map opens the evidence behind those decisions, including what failed.

How the recipe came together

Four places the search changed the recipe. Choose a topic to explore what worked, what failed, and what surprised us.

Combined recipe · tested end to end Useful changes had to work together inside the training budget.

Figure 6. Topics in the search, not shares of the BPB gain. Equal-sized cards group related experiments; not every successful experiment became part of the final recipe. Stopped branches refer to the tested designs.

Data: the biggest lever was not a kernel

The B200 ran more updates but scored worse. Comparing learning curves, the scientist suspected repetition of the small training corpus, though it first explored wider models. We prioritized testing more data. The coordinator prepared extra shards; the scientist assigned a worker to exclude the held-out shard from training. With backend, width, and batch fixed, the paired comparison doubled training supply from seven to fourteen shards and took BPB from 0.96140 to 0.92088. Before asking how to fit even more updates into the budget, we needed to check what those updates were learning from.

Explore data: more text, filtering reversals, and the tokenizer

The good : more useful training material. The seven-to-fourteen-shard comparison changed data supply and exposure while keeping the backend, width, and batch fixed. Simply increasing the shard count would have included validation shard 7, so the worker added explicit training-shard selection and checks that training and validation shards did not overlap. The gain supported trying more data; it did not isolate repetition from corpus composition and exposure. Donor-original text also beat the rewritten alternative in the tested final-hardware comparison.

The bad : a filter did not travel well. Light filtering helped on an H200 search recipe but hurt after transfer to the B200 recipe. This was a reason to retest the treatment, not evidence that filtering always helps or always hurts.

The beautiful : the input pipeline was part of the model search. More training material delivered a large gain without a new architecture. A new tokenizer also changed how much text fit in a context, so its score could not be read as a clean tokenizer-algorithm comparison.

Data choices in the final recipe:

• A Unigram16k tokenizer. Trained and integrated a 16,384-token Unigram model, replacing the inherited tokenizer .
• A smaller training packing buffer. Reduced the training document-packing buffer from 1,000 to 256 documents, while keeping evaluation packing separately configured.
• The donor-original training-data condition. Replaced the preceding training corpus with the tested donor-original corpus and integrated its frozen prepared rows into the winning recipe.

In the historical tokenizer panels, substitutions changed evaluation byte coverage as well as training inputs. Those panels should not be treated as comparisons over identical text exposure.

Capacity: where it went mattered as much as how much

In five minutes, a bigger model has to earn back its cost. When uniform growth did not help, the next question was where capacity would pay. Moving feed-forward capacity toward the final global-attention layer helped where uniform growth had not.

Starting shallow and adding depth during training offered another way to spend the budget. But completed variants lost to the full-depth control: the transition itself took training time and changed the learning trajectory. The comparison had to include the transition, not just the model before and after growth.

Explore capacity: placement, growth costs, and combinations

The good : put capacity where it pays. Redistributing existing feed-forward channels toward the final global-attention layer improved results. Combining that allocation with an earlier momentum ramp beat both standalone changes on every seed in its three-seed panel.

The bad : growing a model was not free. Completed depth-growth variants lost to the full-depth control. The transition took training time and changed the learning trajectory. Subtracting its duration afterward could not recover the validation score of a run that never happened.

The beautiful : two wins need not add up to be worth combining. RoPE frequency and learning-rate-tail changes beat both standalone recipes, but delivered less than their additive prediction. The useful question was whether the combination won, not whether each ingredient kept its original gain.

Capacity and training choices in the final recipe:

• FFN learning rates adjusted for hidden width. Scaled feed-forward input-projection learning rates by sqrt(4 × model_width / hidden_width) to accommodate unequal layer sizes.
• Higher learning rates for both FFN projections. Applied a 1.25× multiplier to feed-forward input and output matrix learning rates.
• 256-token local attention. Reduced local attention windows to 256 tokens, retaining full-context attention in selected layers.
• A 96-sequence training batch. Selected 96 × 2,048 = 196,608 tokens per optimizer update, following local hardware and quality experiments.
• Batch-adjusted embedding learning rates. Applied square-root batch scaling to token embeddings, the output head, value embeddings, and n-gram tables when moving from 64 to 96 sequences.

Matched combinations: architecture and schedules

Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.

Memory: a failed idea could still have a future

Learned n-gram memory adds lookup tables keyed by short token sequences, giving the model another way to use local context . Introducing and scaling bigram memory lowered one campaign’s mean from 0.98840 to 0.96913 BPB. Yet the first projected trigram made things worse. Was remembering triples a bad idea, or was this a bad way to use them?

The team tried a full-width design on a regularized parent, with a different scale. All three matched seeds improved. That kept the direction open without making the earlier failure disappear, or proving which design change mattered.

Explore memory: the failed projection, the retry, and sparse updates

The good : simple memory made a large difference. Bigram memory’s 0.98840 → 0.96913 comparison covered both its introduction and scaling. Later, full-width trigram memory on a regularized bigram parent improved all three matched seeds, from 0.96343 to 0.95987 mean BPB.

The bad : the first projected trigram lost. It scored 0.9751 against a 0.9686 parent on the tested seed. That rejected implementation stayed in the record. It did not establish that trigram memory could never help.

The beautiful : revisiting the failure meant changing the experiment. The successful retry changed projection, regularization, and scale together; it did not isolate projection as the cause. The team could build on the failed test without pretending that the new one answered exactly the same question.

Making the memory practical in the final recipe:

• Sparse n-gram parameter updates. Implemented RMSProp kernels that update only embedding rows touched by the current batch, avoiding full-table parameter and gradient processing.
• Fused n-gram value mixing. Wrote custom kernels that combine hashed-table lookups, gates, and attention-value additions, replacing separate embedding and tensor operations.
• FP32 n-gram gradient accumulators. Routed embedding-table gradients into persistent FP32 buffers, retaining higher-precision accumulation before optimizer updates.
• One FP32 optimizer moment per embedding row. Replaced elementwise RMSProp second moments with a row-wise mean of squared gradients, greatly reducing optimizer-state size .
• Eager moment decay with sparse updates. Advanced inactive rows’ second moments every update while keeping parameter updates sparse, replacing the earlier lazy-decay implementation.
• Selective gradient-buffer clearing. Added touched-row bitmaps and a one-warp clearing kernel that zeros marked gradient rows instead of writing across entire accumulator tables.
• One-warp sparse optimizer launches. Tuned the sparse RMSProp implementation to use one warp per program in both its moment sweep and active-row update kernels.

Sparse updates saved full-table work, but optimizer moments still needed attention between visits to a row. The initial sparse version improved scores despite known arithmetic differences: a performance win was not a proof of numerical equivalence.

Matched combinations: memory and its training rule

Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.

Execution: a faster kernel could still lose the race

Before spending a training run on an optimization, the team could ask a cheaper question: does it change the work the machine actually does? One worker proposed rewriting a square as a multiplication to help the compiler fuse operations. The coordinator checked: the compiler already made that simplification. The worker withdrew the recommendation to screen it as a speed treatment and turned to inspecting the generated code.

Even a measured operator speedup needed another check. FP8’s isolated gains excluded quantization; the full training loop still had to benefit.

Explore execution: full-run wins, misleading speedups, and arithmetic

The good : optimize the work the recipe actually does. Saved-LSE loss backward reduced retained intermediate state; using the softcap’s known bound removed a maximum reduction. Their combination improved a three-seed mean from 0.89053 to 0.88979 on its eager-moment parent. This is a historical comparison, not the final ten-seed headline result.

The bad : faster parts were not enough.

The masked optimizer took 0.58–0.62 ms against its parent’s roughly 0.24 ms in the small probe. That justified declining this implementation, not rejecting sparse optimizers as a direction. Other apparent speedups needed checks beyond the isolated operation:

• Optimizer capture and grouped updates reduced launches but completed fewer updates and scored worse. Gradient-buffer and scalar-staging changes were also part of the treatment.
• Forward/backward capture reduced host overhead, yet the full step stayed slower. Removing private-pool memory inflation did not fix it; the remaining slowdown was not isolated.
• FP8’s isolated 1.12–2.05× throughput gains excluded quantization. They did not establish a faster training loop or better BPB.

The beautiful : a speed change could also be a numerical change. Fused operations changed arithmetic ordering as well as execution cost. The combined recipes could improve without proving that their output was numerically equivalent or that extra updates explained the whole gain.

From the research room · check the generated code

9 September · excerpts from the session logs

Scientist: “Root’s note stands: inductor already lowers pow(2) to mul, so state what your fuse descendant actually changes in generated code.”

Worker: “I read the ‘pow’ token in the kernel name as proof of a pow instruction when it is only proof of a pow node in the graph.”

In the same reply: “It should NOT be screened as a speed treatment.”

The proposal already allowed for a small win or none. The correction changed what the team would check next; it did not prove that every proposed rewrite had zero effect.

Source: hosted Trackinizer AgentSession 19495, records 1570, 1861, 1939, and 1989; 9 September 2026, 05:53–06:23 UTC.

Kernel and loss choices in the final recipe:

• Saved-LSE cross-entropy backward. Saved per-token log-sum-exp values and recomputed probabilities during backward, reducing the loss function’s retained intermediate state .
• Known-bound loss stabilization. Used the softcap’s known upper bound to stabilize exponentials, eliminating the usual per-token maximum reduction.
• Custom QK/RoPE kernels for our model. Implemented normalization and rotation using paired rotary halves, targeting redundant loads.
• Single-rounding loss gradients. Combined probability subtraction and upstream-gradient scaling in FP32, rounding once when storing the final gradient.

Matched combinations: execution and arithmetic

Three-seed means against the named comparator. These overlapping panels come from different stages of the search; their gains are not additive. Hardware, training budgets, and evaluation contexts vary across panels; these comparisons are not slices of the final score.

A useful result also changed what was worth testing next. Once final-layer feed-forward allocation and an earlier momentum ramp had each helped, the team tested them together. The combination beat both standalone changes on every seed in its three-seed panel. Other combinations gained less than the sum of their parts. The next experiment had to test the combination, not assume the isolated wins would add up.

What we learned

The reasoning behind these experiments is familiar: check the data when more training stops helping, test where capacity pays, revisit a failed implementation, and measure the whole loop. Following through means implementing each idea, comparing the results, reviewing the evidence, and leaving work that someone else can use. NanoChat gave us a concrete test of that research system.

That is our vision for rekursiv.ai: AI scientists that accumulate knowledge, run the next experiment, and help us improve how research gets done. Speedrunning science means shortening the path from an idea to a result we can trust, then making that result useful to whoever comes next.

We are building the tools for that loop in the open. Try Trackinizer, Priml, and Configgle in your own research. Bring us a hard problem, a missing capability, or an experiment that failed. Join our Discord and let’s together build what the next scientist will inherit.

References