Show HN：DeepSeek-V4 潜在推理——将“思考”移入潜在空间

DeepSeek-V4-Flash-0731-Latent-Reasoning。一个在潜在空间中进行思考的自包含模型，采用 NVFP4 量化，并提供可用于生产服务运行时的 vLLM 形式。

https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

发布于 blog.n.ichol.ai

---

快速概览

该模型的目标是在开源模型中采用潜在思考（latent thinking）。其理念是压缩思考 token，并将其保留在潜在形式中，而不是通过输出生成出来。

骨干模型被量化至 NVFP4。

模型：nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

---

HuggingFace 仓库包含：

• DeepSeek-V4-Flash-0731 骨干模型。在 TP=2 时，每张 GPU 约需 79 GiB 权重（总计 158–164 GiB）。

• 保留自原始模型的 DSpark 草稿块（3 层）。

• 训练完成的潜在推理头。包含 3570 万参数，从单个 latent_reasoning_head.safetensors 文件（约 152 MB）加载。

初步评测

BBH（BIG-Bench Hard），cot_zeroshot**，27 个子任务：聚合得分 0.94 ± 0.008。**使用 lm-evaluation-harness 0.4.12，通过 OpenAI 兼容端点测量。DeepSeek latent 在分别获得 0.94 与 0.88 精确匹配结果后，对结果进行了重新评估。已启用思考。每个子任务 50 个样本，共计 1350 个样本。

子任务	得分	子任务	得分
tracking_shuffled_objects_three_objects	1.00	date_understanding	0.92
tracking_shuffled_objects_five_objects	1.00	sports_understanding	0.88
tracking_shuffled_objects_seven_objects	1.00	logical_deduction_five_objects	0.88
penguins_in_a_table	1.00	web_of_lies	0.86
formal_fallacies	1.00	snarks	0.84
boolean_expressions	1.00	ruin_names	0.84
word_sorting	0.98	movie_recommendation	0.84
temporal_sequences	0.98	salient_translation_error_detection	0.76
object_counting	0.98	geometric_shapes	0.74
navigate	0.98	causal_judgement	0.66
logical_deduction_three_objects	0.98	disambiguation_qa	0.58
reasoning_about_colored_objects	0.96	dyck_languages	0.26
hyperbaton	0.96
multistep_arithmetic_two	0.94
logical_deduction_seven_objects	0.94

它在推理意味着多步骤状态追踪的任务上最强。tracking_shuffled_objects、boolean_expressions、formal_fallacies、penguins_in_a_table 均达到 1.00。它在机械化、语法密集型任务上的表现最弱。dyck_languages（括号匹配）得分仅为 0.26，是明显的离群值。这是真实弱点，并非测量伪差。

1. Lm eval 使用的是 flexible-extract，而非 strict-match**。**BBH 的 strict-match 会用正则表达式匹配字面短语 The answer is X。该模型不会生成这一短语。其接近于零的 strict-match 得分是答案格式造成的伪差，并非推理失败。

2. **在每个子任务均为 50 个样本的情况下，单项数值约有 ±0.05–0.07 的波动。**聚合得分 0.880 才是可靠数字。

---

架构

                    第 35 层隐藏状态（4096 维）
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> 推理结束
        +---------------------------------------------+
                             | mu（1024 维潜在向量）
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  写回残差流
              DeepSeek-V4-Flash-0731 骨干模型（冻结，NVFP4）

配置	值
hidden_size	4096
latent_dim	1024
mlp_dim	2048
source_layer / target_layer	35 / 42
activation	SiLU
learned stop head	是
head + decoder params	35.7M（float32）
backbone layers	43

该推理头采用变分压缩。它预测 [mu, log_sigma]，并对 log_sigma 进行截断。解码器会将潜在表示重新分配回 4096 维流中。整个组件是一个通过键前缀区分的单一扁平张量字典。几何结构存储在检查点自身的元数据中，因此服务运行时需要零配置；它会直接从文件中读取推理头的形状。target_proj 是一个冻结投影，在训练期间用于定义回归目标；为完整性而保留，但在推理时不会使用。

---

VLLM 支持（某种程度上）

仓库	内容
nickmitchko/ds4-reasoning-addon	潜在推理插件：闭环驱动器、服务脚本、固定版本依赖项与 GPU 容量指南。请从这里开始。
nickmitchko/vllm-ds4-sm120	该插件依赖的 DS4 vLLM 分支（ds4-sm120-preview-dev 分支）。必需。上游 vLLM 无法服务该模型。

单独安装插件不会产生任何效果。插件会完成注册，但在设置 VLLM_DS4_REASONING_CKPT 前始终处于休眠状态。这是一项有意为之的安全设计；可以放心让它保持已安装状态。受支持的入口是一个单独脚本，它会为你设置所有经过实测验证的默认值：

# 1. 引擎（完整构建需要一些时间）
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. 插件及其固定版本依赖项
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. 服务：无需参数，推理头会从模型仓库中解析
release/serve_ds4_reasoning.sh

服务脚本默认使用随附的模型推理头（latent_reasoning_head.safetensors，约 152 MiB，首次运行后会被缓存），因此推荐配置已自动应用。如果你训练了自己的推理头，只需将 HEAD_BUNDLE 指向它即可。

为什么需要 fork？

标准 vLLM 无法服务该模型。

DeepSeek-V4 通过一个以 input_ids 为键的哈希值来路由 MoE 专家。当你提供嵌入时，vLLM 原生的 prompt_embeds 路径会将 input_ids 置空，这会让引擎在启动时因 DeepSeek V4 hash MoE routing requires input_ids 崩溃。因此，该插件无法使用标准嵌入路径。

替代方案是：在目标位置上覆写 embed_tokens 的输出，将其替换为解码后的潜在表示；与此同时，token id 仍会正常流动，从而使哈希 MoE 路由继续工作。潜在步骤携带一个保留的 pad token id，仅用于_记账_。一个潜在步骤会占用真实的 KV 位置，因此它必须令 num_tokens 与 num_computed_tokens 同步递增。该 id 永远不会传给客户端，其嵌入也不会被读取（该行被标记为 is_token_ids=False，所以注入的潜在表示可以保留）。由于这发生在 execute_model 接缝处，它仍可保持在 cudagraph 快速路径上运行。不需要 enforce_eager，并且可跨并发请求批处理。

在潜在阶段会抑制推测式草稿生成（VLLM_DS4_SUPPRESS_LATENT_DRAFTS，默认开启）。请保持开启。rider 会在最后一个查询行注入一个嵌入，因此草稿槽位会夺走它，导致真实位置获得 pad token 的嵌入，而非潜在表示。

---

运行时配置

服务器全局环境变量（默认值均已实测验证）

变量	默认值	作用
MAX_LATENT	256	潜在步骤的安全上限（限制 stop head 误触发的影响）。与推理头训练时的 K=256 一致。
MIN_LATENT	4	潜在步骤下限，确保回答绝不会不经思考。
USE_STOP	1	使用学习得到的停止机制；0 则采用固定 N 上限。
STOP_THRESHOLD	0.5	stop head 阈值。0.5 是正确值。不要通过调低它来“修复”预热问题。
MIN_OUTPUT_TOKENS	4096	答案 token 预算下限。
RIDER	1	0 时服务裸骨干模型（可作为潜在推理的 A/B 基线）。
DEBUG	0	1 时打印每个请求的潜在统计信息。
MAX_MODEL_LEN	262144	上下文窗口。
MAX_NUM_SEQS	2	并发序列数。与上下文长度存在权衡。
GPU_UTIL	0.95	可用范围很窄。0.97 会 OOM，更低则可能无法通过 KV 检查。
TP	2	张量并行大小。

与此前相同，另增加一项新配置。这些请求头优先于环境变量默认值，并且同时适用于 /v1/chat/completions 与 /v1/messages（对 Claude Code 等设置请求头但不设置请求体字段的客户端很有用）。

请求头	效果
x-ds4-thinking	无需请求体字段即可启用思考。
x-ds4-max-latent	每个请求的潜在步骤上限。
x-ds4-min-latent	每个请求的潜在步骤下限。
x-ds4-use-stop	切换学习得到的停止机制。
x-ds4-stop-threshold	每个请求的停止阈值。
x-ds4-min-output-tokens	每个请求的答案预算下限。

限制潜在思考

MAX_LATENT 是安全上限，而不是推理深度调节旋钮。通常由学习得到的停止机制结束这一阶段。该上限仅用于限制误触发。设置位置很重要：远远超过推理头训练时见过的 K=256，会逐渐进入无意义输出，而非更深入地思考。bundle 中的 compression_factor 在服务时同样不起作用。rider 中没有任何部分读取它。它描述的是推理头_学到了什么_，而非循环会强制执行的预算。

MIN_OUTPUT_TOKENS **的存在是因为潜在阶段与答案共享同一份预算。**每个潜在步骤都会消耗一个保留的记账 token。客户端发送较小的 max_tokens 时，可能将全部预算耗在思考上，随后收到一个空答案，且 stop_reason=length。该下限只会_提高_客户端的 max_tokens。一旦答案完成，学习得到的停止机制仍会提前结束生成，因此宽松的下限不会强制模型变得冗长。

---

常见客户端配置

任何与服务器通信的客户端都有两个不那么显而易见的要求。这两点都是经过实践才得出的经验。

**1. 必须要求模型进行思考。**推理阶段由 chat_template_kwargs={"thinking": true}（或请求头 x-ds4-thinking: 1）控制。没有它，输出会变成垃圾。这并非夸张。

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "Write a Python LRU cache."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)

**2. 为答案预留足够 token 空间。**推理与答案共享同一预算。紧张的 max_tokens 可能完全被潜在阶段消耗。不确定时，请使用 MIN_OUTPUT_TOKENS 下限（默认值 4096），而不是为每个请求手动调参。

---

已知限制

启动后的第一个或前两个请求可能出现退化重复。即使 stop head 正常触发，也可能出现很低的草稿接受率、be be be 式输出。它会自行稳定，之后便能保持正确。启动后先发送一个无关紧要的预热请求，并将单个早期错误答案视为_尚未预热_，而非模型损坏。也不要因此调低 STOP_THRESHOLD 或强制设定 SPEC_TOKENS=1。两者都不是原因。

**已启用的 rider 与静默休眠的 rider 会产生完全相同的服务器日志。**如果没有设置 VLLM_DS4_REASONING_CKPT，或者请求没有要求思考，插件会悄然不执行任何操作，而你无从得知。VLLM_DS4_REASONING_DEBUG=1 是确认注入是否实际发生的唯一方式。它会为每次完成打印 steps=N stop_step=... max_p=... end=stop|cap，并在没有注入任何内容时明确说明。

---

运行该模型所需条件

骨干模型采用 NVFP4，因此需要Blackwell 级（sm120）或更高规格硬件。没有替代方案。sm120 稀疏 MLA 内核路径是此 fork 特有的，Hopper 及更早硬件尚未经测试。容量规划完全取决于能否容纳约 158–164 GiB 权重与 fp8 KV 缓存。

显存（总计）	结论
≥ 192 GiB（2× 96，或 4× 48 以上）	**推荐。**可支持长上下文（256k），并为 KV 缓存留出空间。
160–192 GiB	可用。权重可以放下；将 MAX_MODEL_LEN 降至 32k–64k，以确保 KV 缓存可以放下。
< 160 GiB	对于任何上下文长度而言，都没有足够空间容纳 NVFP4 权重。

已验证配置为：2× RTX PRO 6000 Blackwell Max-Q（每张 96 GiB）、TP=2、上下文长度 262,144（443,012 个 fp8 KV token）、gpu_memory_utilization** 为 0.95，搭配推理头与 DSpark 推测解码的吞吐量为 11.1 ms/token（1.40×），草稿接受率为 89–91%。**

• max_num_seqs **会与上下文长度相互权衡。**在 256k 下，fp8 KV 缓存仅能容纳少数几个完整长度的序列。提高并发会挤占 KV 缓存，导致启动时 OOM，或静默缩短可用上下文。只有在降低 MAX_MODEL_LEN 的同时才应提高该值。

• 如果引擎在启动时未通过 KV-cache 检查，请先降低 MAX_MODEL_LEN，再降低 MAX_NUM_SEQS。gpu_memory_utilization 在两个方向上都有很窄的可用区间。

当一个请求到达服务器时，实际会发生什么

DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.

https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

Published on blog.n.ichol.ai

---

Quick overview

The goal of this model is to employ latent thinking in an open source model. The idea is to compress thinking tokens and keep them within latent forms not in the output emission.

The backbone is quantized down to NVFP4.

Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

---

HuggingFace repo:

• The DeepSeek-V4-Flash-0731 backbone. Roughly 79 GiB of weights per GPU at TP=2 (158–164 GiB total).

• The DSpark draft block (3 layers), preserved from the source.

• The trained latent reasoning head. 35.7M params, loaded from a single latent_reasoning_head.safetensors (~152 MB).

first evals

BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks: aggregate 0.94 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Deepseek latent reevaluated the results after giving separate 0.94 vs 0.88 exact match. Thinking enabled. 50 items per subtask, 1350 items total.

Subtask	Score	Subtask	Score
tracking_shuffled_objects_three_objects	1.00	date_understanding	0.92
tracking_shuffled_objects_five_objects	1.00	sports_understanding	0.88
tracking_shuffled_objects_seven_objects	1.00	logical_deduction_five_objects	0.88
penguins_in_a_table	1.00	web_of_lies	0.86
formal_fallacies	1.00	snarks	0.84
boolean_expressions	1.00	ruin_names	0.84
word_sorting	0.98	movie_recommendation	0.84
temporal_sequences	0.98	salient_translation_error_detection	0.76
object_counting	0.98	geometric_shapes	0.74
navigate	0.98	causal_judgement	0.66
logical_deduction_three_objects	0.98	disambiguation_qa	0.58
reasoning_about_colored_objects	0.96	dyck_languages	0.26
hyperbaton	0.96
multistep_arithmetic_two	0.94
logical_deduction_seven_objects	0.94

It is strongest where reasoning means multi-step state tracking. tracking_shuffled_objects, boolean_expressions, formal_fallacies, penguins_in_a_table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck_languages (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact.

1. Lm eval uses flexible-extract, not strict-match. BBH's strict-match regexes for the literal phrase The answer is X. This model does not emit that phrase. Its near-zero strict-match score is an answer-formatting artifact, not a reasoning failure.

2. Per-subtask values carry about ±0.05–0.07 at 50 items each. The aggregate of 0.880 is the reliable number.

---

Architecture

                    layer 35 hidden (4096-d)
                             |
                             v  LayerNorm
        +--------- ReasoningCompressionHead ----------+
        |  Linear 4096 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048  ->  [mu, log_sigma]   |
        |                                             |
        |  stop_head:                                 |
        |     Linear 4096 -> 1024 . SiLU              |
        |     Linear 1024 -> 1                        |  -> end-of-reasoning
        +---------------------------------------------+
                             | mu (1024-d latent)
                             v  LayerNorm
        +-------------- LatentDecoder ----------------+
        |  Linear 1024 -> 2048 . SiLU                 |
        |  Linear 2048 -> 2048 . SiLU                 |
        |  Linear 2048 -> 4096                        |
        +---------------------------------------------+
                             |
                             v  written back into the residual stream
              DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)

Config	Value
hidden_size	4096
latent_dim	1024
mlp_dim	2048
source_layer / target_layer	35 / 42
activation	SiLU
learned stop head	yes
head + decoder params	35.7M (float32)
backbone layers	43

The head is a variational compression. It predicts [mu, log_sigma] and clamps log_sigma. The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs zero configuration ; it reads the head's shape straight from the file. target_proj, a frozen projection that defined the regression target during training, is included for completeness but is not used at inference.

---

VLLM support (sort of)

Repository	What it is
nickmitchko/ds4-reasoning-addon	The latent-reasoning addon: the closed-loop driver, serve script, pinned requirements and GPU-sizing guide. Start here.
nickmitchko/vllm-ds4-sm120	The DS4 vLLM fork it runs on (branch ds4-sm120-preview-dev). Required. Upstream vLLM cannot serve this model.

Installing the addon does nothing by itself. The plugin registers but stays dormant until VLLM_DS4_REASONING_CKPT is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you:

# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation

# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
    --extra-index-url https://flashinfer.ai/whl/cu130/torch2.11

# 3. serve: no arguments needed, the head is resolved from the model repo
release/serve_ds4_reasoning.sh

The serve script defaults to the bundled model head (latent_reasoning_head.safetensors, ~152 MiB, cached after the first run), so the recommended configuration is already applied. If you trained your own head, point HEAD_BUNDLE at it and you are done.

Why a fork at all?

The stock vLLM can't serve this model.

DeepSeek-V4 routes MoE experts by a hash keyed on input_ids. vLLM's native prompt_embeds path nulls input_ids when you supply embeddings, which would crash the engine at startup with DeepSeek V4 hash MoE routing requires input_ids. So the addon cannot use the standard embeddings path.

Instead, injection overwrites the embed_tokens output at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for accounting. A latent step occupies a real KV position, so it must advance num_tokens in lockstep with num_computed_tokens. That id never reaches the client, and its embedding is never read (the row is marked is_token_ids=False, so the injected latent survives). Because this runs at the execute_model seam, it stays on the cudagraph fast path. No enforce_eager, and it batches across concurrent requests.

Speculative drafting is suppressed during the latent phase (VLLM_DS4_SUPPRESS_LATENT_DRAFTS, on by default). Leave it on. The rider injects one embedding at the last query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent.

---

Runtime Configs

Server-wide environment variables (defaults = measured-good)

Variable	Default	What it does
MAX_LATENT	256	Safety cap on latent steps (bounds a stop-head misfire). Matches the head's K=256 training.
MIN_LATENT	4	Floor on latent steps, so answers never no-think.
USE_STOP	1	Learned stop; 0 uses a fixed-N cap.
STOP_THRESHOLD	0.5	Stop-head threshold. 0.5 is correct. Don't "fix" warmup by lowering it.
MIN_OUTPUT_TOKENS	4096	Floor on the answer's token budget.
RIDER	1	0 serves the bare backbone (A/B baseline against latent reasoning).
DEBUG	0	1 prints per-request latent stats.
MAX_MODEL_LEN	262144	Context window.
MAX_NUM_SEQS	2	Concurrent sequences. Trades against context.
GPU_UTIL	0.95	Narrow viable band. 0.97 OOMs, and lower can fail the KV check.
TP	2	Tensor-parallel size.

Same as before, plus one new one. These win over env defaults and work on both /v1/chat/completions and /v1/messages (useful for clients like Claude Code that set headers but not body fields).

Header	Effect
x-ds4-thinking	Enable thinking without a body field.
x-ds4-max-latent	Per-request latent cap.
x-ds4-min-latent	Per-request latent floor.
x-ds4-use-stop	Toggle the learned stop.
x-ds4-stop-threshold	Per-request stop threshold.
x-ds4-min-output-tokens	Per-request answer-budget floor.

Bounding latent thinking

MAX_LATENT is a safety cap, not a reasoning-depth dial. The learned stop normally ends the phase. The cap only bounds a misfire. Where you set it matters: pushing far past the K=256 the head saw in training drifts into garbage rather than thinking harder. The bundle's compression_factor is also inert at serve time. Nothing in the rider reads it. It describes what the head learned , not a budget the loop enforces.

MIN_OUTPUT_TOKENS exists because the latent phase and the answer share one budget. Each latent step bills one reserved accounting token. A client sending a modest max_tokens can spend the whole budget thinking, then get an empty answer back with stop_reason=length. The floor only ever raises a client's max_tokens. The learned stop still ends generation early once the answer is done, so a generous floor does not force verbosity.

---

Common client configs

Two non-obvious requirements for any client that talks to the server. Both were learned the hard way.

1. You must ask for thinking. The reasoning phase is gated on chat_template_kwargs={"thinking": true} (or the header x-ds4-thinking: 1). Without it the output is garbage. That is not hyperbole.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")

resp = client.chat.completions.create(
    model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
    messages=[{"role": "user", "content": "Write a Python LRU cache."}],
    extra_body={"chat_template_kwargs": {"thinking": True}},   # REQUIRED
    max_tokens=4096,
    temperature=0.6,
)
print(resp.choices[0].message.content)

2. Give the answer token headroom. Reasoning and answer share one budget. A tight max_tokens can be consumed entirely by the latent phase. When in doubt, use the MIN_OUTPUT_TOKENS floor (default 4096) rather than hand-tuning per request.

---

Known limitations

The first request or two after startup can be degenerate repetition. Low draft acceptance, "be be be" output, even with the stop head firing normally. It settles by itself and then stays correct. Send one throwaway request after startup and treat a single bad early answer as unwarmed , not broken. And do not lower STOP_THRESHOLD or force SPEC_TOKENS=1 in response. Neither is the cause.

An armed rider and a silently-dormant one produce identical server logs. If VLLM_DS4_REASONING_CKPT is not set, or the request does not ask for thinking, the plugin quietly does nothing and you have no idea. VLLM_DS4_REASONING_DEBUG=1 is the only way to confirm injection is actually happening. It prints steps=N stop_step=... max_p=... end=stop|cap per completion, and says so explicitly when nothing was injected.

---

What it takes to run it

The backbone is NVFP4, so this is Blackwell-class hardware (sm120) or better. There is no way around it. The sm120 sparse-MLA kernel path is specific to this fork, and Hopper and earlier are untested. Sizing is entirely about fitting the ~158–164 GiB of weights plus the fp8 KV cache.

VRAM (total)	Verdict
≥ 192 GiB (2× 96, or 4× 48+)	Recommended. Long context (256k) with room for the KV cache.
160–192 GiB	Workable. Weights fit; drop MAX_MODEL_LEN to 32k–64k so KV fits.
< 160 GiB	Not enough for NVFP4 weights at any context.

The verified configuration: 2× RTX PRO 6000 Blackwell Max-Q (96 GiB each), TP=2, context 262,144 (443,012 fp8 KV tokens), gpu_memory_utilization 0.95, throughput 11.1 ms/token with the head plus DSpark spec decode (1.40×), and 89–91% draft acceptance.

• max_num_seqs trades against context. At 256k the fp8 KV cache only holds a couple of full-length sequences. Raising concurrency starves KV and either OOMs at startup or silently truncates usable context. Raise it only alongside a lower MAX_MODEL_LEN.

• If the engine fails its KV-cache check at startup , lower MAX_MODEL_LEN first, then MAX_NUM_SEQS. gpu_memory_utilization has a narrow viable band in both directions.

what actually happens when a request hits the server: The backbone prefills your prompt once , then takes autoregressive latent decode steps. Each step's input embedding is decoder(head(previous layer-35 hidden)), fed back from the model's own hidden state via a per-request anchor store. This continues until the learned stop head crosses its threshold, emits response, and the answer decodes as ordinary tokens.

h_src (layer 35)  --layer_norm-->  head     -> mu   (latent, 1024-d)
mu                --layer_norm-->  decoder  -> hidden vector (4096-d)

This runs batched across concurrent requests and on the cudagraph fast path (no enforce_eager, no --max-num-seqs 1), and it streams. One latent step stands in for roughly six reasoning tokens. The model thinks compressed, then speaks.

---

Limitations and Future Work

• Blackwell. NVFP4 needs sm120 native kernels, and the sparse-MLA path is fork-specific.m.

• The surfaced trace is not the computation. Reasoning happens in latent space, so the text you see is not a faithful token-level record of the thinking that produced the answer. The latent space remains opaque.

• Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context suite reported. The aggregate is solid; the coverage is still narrow.

• dyck_languages at 0.26 is a real weak spot , Syntax-heavy, mechanical tasks are precisely what compressed-latent reasoning struggles with.

Future work includes:

• A broader benchmark suite: multi-task and long-context, so the BBH 0.880 is not one number carrying the whole story.

• Making the latent reasoning interpretable, so we can see what the model thinks, not just that it thinks.

• Better handling of the mechanical-syntax tasks (dyck_languages) that are the current weak point.

• Pushing the DSpark interplay further. The closed latent loop and speculative decoding together already give 1.40× at 89–91% draft acceptance. There is more to squeeze there.

---

---

This blog was authored with the help of nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning

Implementation: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning · ds4-reasoning-addon · vllm-ds4-sm120 Blog: blog.n.ichol.ai