DeepInfer 是为 DeepSeek 大规模生产级服务提供支持的高性能推理框架。本次开源版本提供了离线推理能力，并包含规模化部署所需的核心特性。

支持的模型：

支持的特性：

• 优化的算子（Kernels）、模型与采样器运行时
• 并行策略：Attention DP / 均衡 CP，MoE EP + EPLB
• DSpark 与基于置信度调度的验证机制（Confidence-scheduled verification）
• SWA 有界重放（SWA bounded replay）
• 异步调度（零开销批处理调度器）
• 批次无关推理（Batch-invariant inference）
• 确定性采样（Deterministic sampling）
• 结构化输出（Structured outputs）

特性详情与配置

• 优化的算子（Optimized kernels）： FlashMLA、DeepGEMM、DeepEP、TileKernels、DeepSelect，以及自定义 TileLang 算子。
• DSpark 与基于置信度调度的验证： 利用草稿标记（draft tokens）加速解码，并利用置信度估计来调度验证过程。设置 DEEPINFER_ENABLE_MTP_VARLEN=1；通过 DEEPINFER_USE_NEXTN_DECODE 控制最大草稿长度（默认值：5）。
• Decoder SWA 有界重放： 通过仅重放有界后缀来减少 Decoder prefill 阶段的工作量。默认的 V4.1 配置可设置 DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact 以及 DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140；保留的后缀必须覆盖 SWA 窗口以及最大回溯长度。
• 异步调度： 实现 CPU 批处理调度与设备端执行的重叠掩盖。默认启用。
• 批次无关推理： 确保在不同的批次组合下，模型输出结果始终保持一致。通过 DEEPINFER_BATCH_INVARIANT=1 启用（仅限 NVIDIA）。
• 确定性采样： 在相同的输入 tokens 和采样参数下生成完全一致的输出。设置 DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0，并在 SamplingParams 中传入固定的 seq_seed（仅限 NVIDIA）。
• 结构化输出： 将生成结果约束在正则表达式、JSON schema 或 EBNF 语法范围内。向 SamplingParams 传递 so_args={"schema_type": ..., "schema": ...}；详见 示例。

支持的硬件：

• NVIDIA SM100 GPU
• 华为昇腾 950 NPU

构建完整推理服务所需组件（本开源版本暂未包含）：

• KV 缓存系统（KV cache system）
• VE / Prefill / Decode 解耦架构（大部分所需的引擎底层功能已实现）
• 面向 prefill 和 decode 阶段的数据并行副本负载均衡系统

环境依赖：

• Linux
• GCC >= 13.0
• Python 3.12
• CUDA 13（仅限 NVIDIA）
• CANN 9.2（仅限昇腾）

在构建 DeepInfer 之前，请创建并激活一个干净的 Python 3.12 环境。

bash

昇腾平台请替换为 ascend

请在运行 DeepInfer 的环境中持续 export 此环境变量

export DEEPINFER_ACCELERATOR=nvidia ./build.sh

默认模型权重 deepseek-ai/DeepSeek-V4.1-Flash-1017 会在首次运行时自动下载到 Hugging Face 缓存中；如需使用本地权重副本，请设置 DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017。

bash python examples/offline_inference.py

详见 examples。

Prefill	Decode

上述基准测试在单个 scale-up 域内测得：Prefill 阶段使用 4 块 SM100 GPU，Decode 阶段使用 64 块 SM100 GPU。如需复现基准测试，请参考 run_sweep.sh。

详见 eval/README.md。

DeepInfer 基于 Apache License 2.0 协议开源。详情请见 LICENSE。

DeepInfer 最初主要基于 vLLM 构建，并吸纳了来自 SGLang 与 TensorRT-LLM 的灵感。在此向这些项目的开发者们致谢。

bibtex @misc{deepinfer2026, title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models}, author = {DeepSeek-AI}, year = {2026}, publisher = {GitHub}, howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}}, }

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

Supported features:

• Optimized kernels, model and sampler runtime
• Parallelism: Attention DP / balanced CP, MoE EP + EPLB
• DSpark and confidence-scheduled verification
• SWA bounded replay
• Asynchronous scheduling (zero-overhead batch scheduler)
• Batch-invariant inference
• Deterministic sampling
• Structured outputs

Feature details and configuration

• Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
• DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
• Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
• Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
• Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
• Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
• Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

• NVIDIA SM100 GPUs
• Huawei Ascend 950 NPUs

Required for a complete inference service (not included in this release):

• A KV cache system

• VE/Prefill/Decode disaggregation (most required engine functionality is already implemented)

• Load balancing across data-parallel replicas for both prefill and decode

• Linux

• GCC >= 13.0

• Python 3.12

• CUDA 13 (NVIDIA only)

• CANN 9.2 (Ascend only)

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Prefill	Decode

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

See eval/README.md.

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}