从零开始构建推理模型：11小时 Python 与 PyTorch 实战视频课程

在过去两年里，我已经在 Ahead of AI 上撰写了大量关于推理模型的内容，包括最近一篇关于控制当代 LLM 中推理工作量水平机制的文章。推理模型的重要性如今依然不减。事实上，目前几乎所有当代前沿 LLM 要么本身就是推理模型，要么提供了专门的推理模式。

对于那些想了解推理模型底层运作机制（包括推理扩展、强化学习和模型蒸馏）的读者，我近期编写了《从零开始构建推理模型》（Build Reasoning Model (From Scratch)）一书。书中完全使用 PyTorch 从零实现了这些方法——根据我的经验，这仍然是理解和学习事物真实底层机制的最佳途径。

过去几个月里，我还抽时间录制了配套的讲座视频并与大家分享。在下文中，你将找到 8 个从零构建推理模型的代码实战视频（总计 11 小时内容）。

我们将从一个预训练 LLM 出发，使用 Python 和 PyTorch 实现评估和改进它的各种方法。如果你喜欢我去年推出的《从零开始编写 LLM 代码》（Coding LLMs from the Ground Up）课程，那么这门课本质上就是它的续篇（不过即使没有学过前置课程，你也可以直接观看）。

我目前正在休一个短暂的假期，下个月将带来更多精彩的 LLM 研究综述。在此期间，祝大家动手实操编码愉快！

---

作为一段简短的补充说明，由于经常有人问我推理模型与当今 AI Agent 之间到底是什么关系，在正式开始视频前，我先在此做个简短解析。

推理模型本质上是对传统 LLM 的一种升级——即通过带有可验证奖励的强化学习（以及可选的推理扩展技术）增加更多后训练（Post-training），使其更具解决高难度问题的能力。传统上，推理模型主要针对数学和代码任务进行开发，但这种能力同样可以泛化到其他问题领域。

随后，推理模型充当了 Agent 框架（Agent Harness）内部的“引擎”。这里的 Agent 框架是指围绕 LLM/推理模型构建的软件或工具体系，用于解决更具挑战性的综合问题（如更大规模的编码任务、旅行预订以及各类个人助手任务）。框架的核心职责是进行复杂的上下文管理并赋予 LLM 调用工具的能力。

图 1：传统 LLM、推理模型与 Agent 框架的引擎类比。图源来自我的《编程 Agent 的核心组件》一文。

在上图所示的比喻中，模型是“引擎”，而 Agent 框架则添加了让我们能够将其用于各种复杂场景的整套“机械结构”（相比于简单的聊天机器人界面的重大升级）。Agent 既可以使用传统 LLM，也可以使用推理模型，但推理模型往往是更常见的选择，因为其能力更强。（正如前文所述，大多数当前一代 LLM 都已经是具备可变推理工作量设置的推理模型。）

归根结底，推理模型作为核心“引擎”，依然是理解现代 LLM Agent 的关键一环。

上述主题的配套参考资源包括：

• 《从零构建大语言模型》（Build a Large Language Model (From Scratch)）图书与视频
• 《从零构建推理模型》（Build A Reasoning Model (From Scratch)）图书（以及下方的配套视频）
• Substack 上的《编程 Agent 的核心组件》一文

---

下面的视频重点讲解如何开发能够驱动此类系统的推理模型。

“从零构建推理模型”视频系列围绕一个小型的预训练 Qwen3 模型展开，我们将使用 Python 和 PyTorch 手动实现各项推理方法。从预训练模型开始，可以让我们把精力集中在最有趣的部分：模型评估、推理时扩展（Inference-time Scaling）、强化学习和模型蒸馏。

关于前置要求，我建议学习者熟悉 Python 和 PyTorch。我将推理材料设计为独立内容，因此推荐阅读之前的《从零构建大语言模型》，但这并不是硬性先决条件。

在算力资源方面，前期的文本生成和评估示例完全可以在 CPU 上运行。对于后期的实验，尤其是训练阶段，配备 GPU 会让实践更高效。第一期视频详细介绍了环境搭建及相关建议；代码仓库中也提供了硬件与环境配置建议。

图 2：涵盖主题总览。

这些视频是我撰写的《从零构建推理模型》（Build a Reasoning Model (From Scratch)）一书的配套教程。如果你喜欢更详尽的文字解析、希望按自己的节奏深入阅读并完成课后练习，建议结合本书一起学习。

为了获得最深入的学习体验，我的建议是：先阅读章节，跟随配套视频编写对应代码，最后尝试完成练习题。

该书目前已在 Manning 和 Amazon 发售，且印刷版为全彩印刷！

图 3：《从零构建推理模型》实体书随机摘录页面。

非常感谢购买本书或将其推荐给朋友的每一位读者，你们的支持是我持续创作这些内容的动力。

在进入下方核心视频之前的最后一个简短提示：如果你想快速了解概念层面的精简概览，我也推荐收看我在 LinkedIn Learning 上的 AI 推理模型课程（通过该链接访问可免费学习 24 小时，无需订阅 LinkedIn Premium）。

该课程时长约 90 分钟，讲解了推理模型的工作原理，涵盖了推理时扩展、强化学习和蒸馏的核心思想。如果你希望在动手完整实现以下代码之前先快速领会主要概念，这是一个很好的可选前瞻内容。

---

接下来进入本文的核心部分，以下是主要的八个实战讲座视频：

视频 1

第一期视频首先讲解了传统 LLM、推理模型与 Agent 之间的关系。随后，视频的核心主题是配置编程环境。这包括使用 uv 安装 Python 和 PyTorch 依赖项等。我还探讨了硬件选择以及常见的配置问题。

涵盖主题

• 传统 LLM、推理模型与 Agent 框架之间的关系
• 通过代码实现进行学习的益处
• 本视频系列的范围
• 安装 Python 依赖
• 选择硬件及检查 CUDA 或 Apple Silicon 支持
• 在 VS Code 中使用远程 GPU 及环境故障排查

观看视频 1 · 环境配置指南

---

视频 2

在后续视频中添加推理方法之前，我们需要一个可运行的基座模型。我们不需要从头训练它（因为那需要写另一整本书，详见《从零构建大语言模型》），而是直接从预训练模型入手。

在这里，我们加载 Qwen3 0.6B 模型的预训练权重。然后，通过流式文本生成器实现文本生成流程，并添加 KV 缓存（KV Cache）和模型编译（Model Compilation）以提高文本生成效率。

这些实现为我们提供了基础基线与核心组件，供后续视频进行评估与改进。

涵盖主题

• 将文本编码为 Token 并反向解码为文本
• 下载 Qwen3 权重并将模型加载到选定的计算设备上
• 梳理张量形状（Tensor Shapes）、Logits 和 Next-token 预测
• 实现贪婪解码（Greedy Decoding）和流式文本生成
• 处理序列结束（End-of-sequence）Token 并测量生成速度
• 添加 KV 缓存以复用注意力机制中的 Key 和 Value
• 将 KV 缓存与 torch.compile 结合并对比硬件性能

观看视频 2 · 第 2 章代码

---

视频 3

在本讲中，我们将一步步构建一个数学答案验证器（Verifier），从答案提取、标准化到检查数学等价性。随后，我们使用它在 MATH-500 数据集上评估模型。

我们使用该验证器来评估基座模型在 MATH-500 评估数据集上的性能表现。

此外，该验证器在系列后续课程中还将承担第二项重任：为可验证奖励强化学习（RLVR）提供奖励信号。

涵盖主题

• 验证器简介及其在 LLM 评估方法中的定位（其他方法在本书附录中涵盖）
• 复用答案验证作为 RLVR 的奖励信号（详见第 6 章）
• 验证器工作流：包括提取最终答案、标准化答案以及检查数学等价性
• 构建答案评分器和 MATH-500 评估循环
• 在 MATH-500 上对比基座模型与推理模型

观看视频 3 · 第 3 章代码

---

视频 4

现在，我们探索在保持模型权重不变的情况下改进答案的方法。但在实现具体的推理扩展技术之前，我们首先要实现从同一个 Prompt 生成多样化答案的方法（这也是 ChatGPT 等商业 LLM 聊天机器人的标准做法）。

具体而言，我们实现了温度缩放（Temperature Scaling）、多项式采样（Multinomial Sampling）和 Top-p 过滤来生成多样化响应。随后，我们通过自洽性（Self-Consistency）方法整合多个响应，对其最终答案进行多数投票（Majority Voting）。

我们还深入探讨了答案准确率与生成更多候选答案所需的额外计算量之间的权衡取舍。

涵盖主题

• 对比额外算力在训练阶段与推理阶段对模型性能带来的提升
• 通过思维链（Chain-of-Thought）Prompt 实现推理扩展的入门导引
• 实现温度缩放与多项式采样以生成多样化答案
• 添加 Top-p 过滤以控制多样化答案的质量
• 通过自洽性（主流推理扩展技术之一）整合候选答案
• 评估 MATH-500 准确率与额外推理算力成本之间的关系

观看视频 4 · 第 4 章代码

---

视频 5

在上一视频的基础上，我们实现了评分函数来对比同一 Prompt 下生成的多个答案。我们还可以使用这些评分方法在自洽性中解决平票问题（前一讲已涵盖）。

这里涉及的评分方法包括计算 Token 和序列的对数概率（Log Probabilities，这在后续的强化学习章节中也将至关重要）。随后，我们实现了一个自我修正循环（Self-Refinement Loop），模型在该循环中对答案提出批评意见并尝试修改。

评估环节让我们能够检验这些额外步骤对模型的具体效用。

涵盖主题

• 使用评分函数比较多个答案并打破平局
• 实现简单的启发式评分器
• 计算 Token 概率和序列似然度（对后续讲座亦非常关键）
• 使用对数概率以提高数值稳定性
• 构建用于生成批评意见与修改答案的自我修正循环
• 在 MATH-500 上评估自我修正的效果

观看视频 5 · 第 5 章代码

---

视频 6

在本讲中，我们将进阶实现名为“可验证奖励强化学习”（RLVR）的训练技术。我们首先实现简化版的群体相对策略优化（GRPO），生成成组的响应，使用我们的验证器对其评分，并在训练过程中使用它们的相对优势/奖励来更新模型权重。

涵盖主题

• 生成推理轨迹并理解答案长度与解题能力之间的关系
• 探讨并对比两项关键的 LLM 后训练方法：基于人类反馈的强化学习（RLHF）与基于可验证奖励的强化学习（RLVR）
• 将验证器评分转化为奖励与相对优势（Relative Advantages）
• 计算序列对数概率与简化版 GRPO 损失函数
• 组装包含日志记录和检查点（Checkpoint）保存的完整训练循环
• 评估检查点并检验训练稳定性与显存占用需求

观看视频 6 · 第 6 章代码

---

视频 7 与后续分析

我们在上一讲中已经完成了训练循环的搭建，现在可以回顾该流程并对训练运行进行更细致的分析。

具体而言，本视频加入了更多用于观察和分析的训练指标，包括熵（Entropy）和优势统计量、裁剪后的策略比率（Clipped Policy Ratios）、KL 散度损失项数值以及格式化奖励。

接着，我们通过审查训练日志和评估结果，深入了解这些新增指标的作用，并探讨裁剪策略比率和 KL 损失项等机制是否真正发挥了积极作用。

涵盖主题

• 运行更长周期的训练实验并解读训练日志
• 追踪优势值与熵以理解训练行为特征
• 在策略损失中实现裁剪（Clipping）并分析其实际影响
• 引入 KL 损失项并深入探讨奖励破解（Reward Hack）现象

I have written a lot about reasoning models here over the last two years on Ahead of AI, including a recent article on mechanisms to control reasoning effort levels in contemporary LLMs. Reasoning models are still as relevant as ever. In fact, almost all current-generation LLMs today are reasoning models or offer a reasoning mode.

For those who are interested in how reasoning models work, including inference scaling, reinforcement learning, and distillation, I wrote my recent Build Reasoning Model (From Scratch), which implements these from the ground up in PyTorch, which, in my experience, is still the best way to understand and learn how things really work.

Now, in the last few months, I also sat down and recorded accompanying lectures I wanted to share with you. So, below, you will find eight reasoning model-from-scratch coding videos (11 hours of content).

We start with a pre-trained LLM and implement methods to evaluate and improve it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up course last year, this is essentially the sequel (but you can also watch it without the prior course as a prerequisite).

I’m currently taking a short break and will be back soon with more exciting LLM research roundups next month. In the meantime, I hope you enjoy coding along!

As a small side note, and since I often get asked how reasoning models relate to today’s AI agents, here’s a short section on this before getting started with the videos.

A reasoning model is essentially an upgrade from a conventional LLM, that is, adding more post-training via reinforcement learning with verifiable rewards (and optionally inference scaling techniques) to make it more capable of solving more challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to other problem domains as well.

A reasoning model then functions as the engine inside an agent harness, where the agent harness is a software or tooling framework around the LLM/reasoning model to tackle even more challenging problems (bigger coding tasks, travel booking, and all other sorts of personal assistant tasks). The task of the harness is sophisticated context management and giving the LLM access to tools.

Figure 1: An engine analogy for conventional LLMs, reasoning models, and agent harnesses. Figure from my Components of a Coding Agent article.

In this analogy depicted in the figure above, the model is the engine, and the agent harness adds the machinery that lets us use it for various things (as an upgrade from a simple chatbot interface). An agent can use either a conventional LLM or a reasoning model, but reasoning models are the more typical choice as they are more capable. (As mentioned above, most current-gen LLMs are reasoning models with variable reasoning effort settings.)

So, the bottom line is that reasoning models, as the “engine,” remain an important part of understanding modern LLM agents.

The resources accompanying these topics shown above are:

• The Build a Large Language Model (From Scratch) book and videos

• The Build A Reasoning Model (From Scratch) book (and videos below)

• The Components of a Coding Agent article on Substack

The videos below focus on developing the reasoning model that can power such a system.

The reasoning-from-scratch videos focus on a small pre-trained Qwen3 model, where we implement the reasoning methods ourselves in Python and PyTorch. Starting with a pre-trained model lets us focus on the interesting part: evaluation, inference-time scaling, reinforcement learning, and distillation.

Regarding the prerequisites, I recommend being comfortable with Python and PyTorch. I designed the reasoning materials as standalone content, so the previous Build A Large Language Model (From Scratch) content is recommended but not strictly required.

Compute-resource-wise, you can work through the early text-generation and evaluation examples on a CPU. For the later experiments, especially training, a GPU makes the work much more practical. The first video covers the setup and some more recommendations; the repository also includes hardware and environment recommendations.

Figure 2: An overview of the topics being covered.

These videos accompany my book, Build a Reasoning Model (From Scratch). If you like to read through the explanation in more detail and at your own pace, work through exercises, and so on, I recommend the book as a companion here.

For the most in-depth learning experience, my suggestion is to read a chapter, work through its code alongside the corresponding video, and then try the exercises.

The book is available from Manning and Amazon , and the print edition is in color!

Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) print book.

Thank you to everyone who has picked up a copy or recommended it to a friend. Your support helps me keep creating these materials.

As a last, quick side note before getting to the main videos below: For a shorter, more conceptual overview, I also recommend my AI Reasoning Models course on LinkedIn Learning (it should be free for 24h when accessing it via the link there, which shouldn’t then require a LinkedIn Premium subscription).

The course is about 90 minutes and explains how reasoning models work, covering the main ideas behind inference-time scaling, reinforcement learning, and distillation. It’s a good optional preview if you’d like the main ideas without working through the full implementations below.

To get to the main part of this article, here are the eight main lecture videos.

The first video starts with an explanation of the relationship between conventional LLMs, reasoning models, and agents. Next, the main topic of the video is setting up the coding environment. This includes installing the Python and PyTorch dependencies (with uv), among others. I also discuss hardware choices and common setup issues.

Topics covered

• The relationship between conventional LLMs, reasoning models, and agent harnesses

• On the benefits of learning through code implementations

• The scope of this video series

• Installing Python dependencies

• Choosing hardware and checking CUDA or Apple Silicon support

• Using a remote GPU with VS Code and troubleshooting the environment

Watch video 1 · Setup instructions

Before adding reasoning methods in the upcoming videos, we need a working base model. Instead of training it from scratch, since that would be a whole other book (see Build A Large Language Model (From Scratch)), we start with a pre-trained model.

Here, we load the pre-trained weights of a Qwen3 0.6B model. Then, we implement the text-generation process via a streaming text generator and add KV caching and model compilation to make text generation more efficient.

These implementations give us the baseline and base components we will evaluate and improve in the following videos.

Topics covered

• Encoding text as tokens and decoding it back into text

• Downloading Qwen3 weights and loading the model on the selected compute device

• Going over tensor shapes, logits, and next-token predictions

• Implementing greedy decoding and streaming text generation

• Handling end-of-sequence tokens and measuring generation speed

• Adding a KV cache to reuse attention keys and values

• Combining KV caching with torch.compile and comparing hardware performance

Watch video 2 · Chapter 2 code

In this lecture, we build a math-answer verifier step by step, from answer extraction to normalization and checking mathematical equivalence. We then use it to evaluate the model on MATH-500.

We use this verifier to evaluate the base model’s performance on the MATH-500 evaluation dataset.

However, the verifier also has a second job later in the series: providing the reward signal for reinforcement learning with verifiable rewards (RLVR).

Topics covered

• Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)

• Reusing answer verification as a reward signal for RLVR (more on that in chapter 6)

• The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence

• Building an answer grader and an evaluation loop for MATH-500

• Comparing base and reasoning models on MATH-500

Watch video 3 · Chapter 3 code

Now we explore ways to improve answers while keeping the model weights fixed. But before implementing specific inference-scaling techniques, we first implement methods to generate diverse answers from the same prompt (this is standard practice in proprietary LLM chatbots, such as ChatGPT).

Concretely, we implement temperature scaling, multinomial sampling, and top-p filtering to generate diverse responses. Then we combine multiple responses through self-consistency, using majority voting over their final answers.

We also examine the tradeoff between answer accuracy and the extra computation needed to generate more candidates.

Topics covered

• Comparing how additional compute benefits model performance during training and during inference

• A gentle intro to inference-scaling through chain-of-thought prompts

• Implementing temperature scaling and multinomial sampling to generate diverse answers

• Adding top-p filtering to control the quality of the diverse answers

• Combining candidate answers through self-consistency, one of the most popular inference scaling techniques

• Evaluating MATH-500 accuracy against the cost of additional inference compute

Watch video 4 · Chapter 4 code

Building on the previous video, we implement scoring functions to compare multiple answers generated from the same prompt. We can also use these scoring methods to break ties in self-consistency (covered in the previous lecture).

The scoring methods covered here include computing token and sequence log probabilities (this will also be relevant in the later reinforcement learning chapters). We then implement a self-refinement loop in which the model critiques an answer and attempts a revision.

The evaluation lets us check how useful those additional steps are for our model.

Topics covered

• Using scoring functions for comparing multiple answers and breaking ties

• Implementing a simple heuristic scorer

• Computing token probabilities and sequence likelihoods (also relevant for the following lectures)

• Using log probabilities for improved numerical stability

• Building a self-refinement loop for generating critiques and revised answers

• Evaluating self-refinement on MATH-500

Watch video 5 · Chapter 5 code

In this lecture, we move on to implementing a training technique called Reinforcement Learning with Verifiable Rewards (RLVR). We first implement a simplified version of Group Relative Policy Optimization (GRPO), generate groups of responses, score them with our verifier, and use their relative rewards for the model weight updates during training.

Topics covered

• Generating reasoning traces and understanding the relationship between answer lengths and problem-solving abilities

• Discussion and comparison of two important LLM post-training methods: Reinforcement Learning with Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR)

• Turning verifier scores into rewards and relative advantages

• Computing sequence log probabilities and a simplified GRPO loss

• Assembling the training loop with logging and checkpoint saves

• Evaluating checkpoints and checking training stability and memory needs

Watch video 6 · Chapter 6 code

Now that we implemented the training loop in the previous lecture, we can revisit it and analyze the training runs in more detail.

Concretely, this video adds more training metrics for inspection and analysis, including entropy and advantage statistics, clipped policy ratios, KL loss term values, and formatted rewards.

We then review the training logs and evaluation results to understand what these additions do and whether additions such as clipped policy ratios and KL loss terms are helpful.

Topics covered

• Running longer training experiments and interpreting their logs

• Tracking advantages and entropy to understand training behavior

• Implementing clipping in the policy loss and analyzing its effects

• Adding a KL loss term and discussing reward hacking

• Understanding limitations of the simplified KL formulation

• Implementing format rewards and conditioning them on answer correctness

• Comparing checkpoints on MATH-500 and exploring further GRPO research

Watch video 7 · Chapter 7 code

Finally, we explore another way to train a small reasoning model, that is, learning from answers generated by a stronger (and bigger) teacher model. We discuss different forms of distillation, prepare reasoning data, and implement the supervised training loop.

We finish this lecture and course by evaluating the distilled model on MATH-500 and comparing the results. This rounds out the series with an alternative to developing reasoning capabilities through reinforcement learning.

Topics covered

• Comparing the different combination and approaches to distillation, including soft and hard distillation, and on-policy and off-policy distillation

• Creating and collecting teacher-generated answers for the MATH dataset (using DeepSeek R1)

• Preparing the distillation dataset, including tokenization and training/validation/test splits

• Computing answer-token log probabilities and understanding the cross-entropy loss

• Implementing supervised training and tracking training and validation losses

• Evaluating checkpoints on MATH-500 and comparing distillation results

Watch video 8 · Chapter 8 code

As you watch these lectures, I encourage you to pause occasionally and try a small experiment of your own if something interesting comes to mind.

For example, feel free to change the sampling settings, inspect a few failed answers, or try variations of different ideas similar to the suggestions in the exercises.

And if you would like the full written walkthrough, figures, and exercises alongside the videos, you can find Build a Reasoning Model (From Scratch) at Manning and Amazon.

Thanks for reading and supporting Ahead of AI. I am taking a short break to regenerate, and I look forward to being back with the next round of LLM research articles next month.

In the meantime, happy viewing and tinkering!