DeepSeek-V4.1-Flash-1017 混合精度 GGUF 权重发布：GSQ 与 RCO 算法加持，12GB 显存即可运行

更新（2026 年 10 月 27 日）： 模型文件现已采用 llama.cpp 原生的 DeepSeek-V4.1 布局，支持在 llama.cpp b11870 或更新版本中加载（包含图像推理）。

• 已下载过早期文件？ 您无需重新下载整个模型。只需在模型目录下运行 convert_ds41_upstream.py：它会在本地将文件精准重写为当前最新格式，仅需额外拉取 80 KB 数据。详见“更新已有下载文件”。

• 全新下载？ 共享的 Engram 文件是每个构建版本的第 6 个分片（Shard 6），因此为您使用的每个版本创建一次软链接即可（<ID> 可为 IQ1_M、IQ2_XXS 或 IQ2_XS；详见 llama.cpp 说明）：

  ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-00006-of-00006.gguf
  

---

概述

本仓库提供了 DeepSeek-V4.1-Flash-1017 的三种规格 GGUF 量化版本：IQ1_M（平均每个权重 1.75 比特）、IQ2_XXS（2.00 比特）和 IQ2_XS（2.25 比特）。与对所有权重张量套用同一种量化类型的均匀量化不同，此处的每个模型都为各个张量单独分配量化类型。这一分配方案通过基于梯度的搜索得出，在满足总体大小预算的前提下，根据各张量的敏感度动态分配精度。最终生成的文件均为标准 GGUF 格式。

DeepSeek-V4.1-Flash-1017 是一个拥有 7632 亿参数（763.2B） 的稀疏混合专家（MoE）模型：每层包含 384 个路由专家，每个 token 激活 6 个专家，支持 100 万（1M）token 上下文窗口。由于路由专家占据了绝大多数参数量，因此也占据了体积预算的大头；搜索算法将大部分优化自由度应用在专家部分，并按层分配量化类型（而非按单个专家分配，目前 GGUF 格式尚无法表达单专家粒度）。

配合 Strata，每个构建版本均可在单张配备 12 GB 显存的 GPU 上运行。详见“使用方法”。

方法摘要： GSQ 在给定目标量化类型下提供高精度的低比特标量量化；RCO 则在总体体积预算约束下为各个张量分配量化类型。两者结合即可输出满足指定体积要求的非均匀精度 GGUF 模型。

方法	描述
GSQ（Gumbel-Softmax Quantization，论文，代码）	训练后标量量化方法，通过 Gumbel-Softmax 松弛联合学习按坐标的网格分配与按组缩放因子。GSQ 在 2 到 3 比特下填补了标量量化与矢量量化之间的大部分差距，同时保持可直接部署在 GGUF 等标准标量格式中。
RCO（Riemannian Constrained Optimization，论文，代码）	在总体体积预算约束下，为 N 个张量中的每一个分配 K 种量化类型之一。预算约束被重构为 Logit 空间中的光滑黎曼流形，从而可以直接基于任务损失进行梯度优化，在严格满足预算的同时无需针对特定约束调整超参数。

两项技术均由奥地利科学技术研究所 深度算法与系统实验室（DASLab） 开发。

---

可用文件

每个构建版本均由 5 个分片的 Backbone GGUF 组成，以其量化标识符命名：DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-0000i-of-00006.gguf（i = 1 到 5）。分片 1 仅包含元数据，分片 2 至 5 包含 Transformer 权重，共 5519 亿参数（551.9B）。bpw（bits per weight）即这些权重的平均比特宽度，也是搜索算法控制的核心指标。

Engram 文件 DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf 包含模型采用 IQ4_NL 格式（4.50 bpw）的两个 Engram 查找表（1966 亿参数，196.6B）。该文件对所有构建版本通用，因此仅存储一份，在尝试多个版本时也只需下载一次。llama.cpp 将其作为对应版本的第 6 个分片读取，因此每个版本需要为其建立名为 ...-<ID>-00006-of-00006.gguf 的链接（一条命令即可完成，详见 llama.cpp 说明）。由于这些表属于查表操作而非矩阵乘法权重，因此在所有构建版本中保持固定类型，不参与精度搜索。在基准评测模型中它们同样被量化，并非仅在发布版本中做处理。

构建版本	bpw	Backbone 大小	Engram 大小	总计大小	备注
IQ1_M	1.75	120.72 GB	110.60 GB	231.32 GB	最小体积
IQ2_XXS	2.00	138.24 GB	110.60 GB	248.84 GB	推荐版本；最接近基模型
IQ2_XS	2.25	155.23 GB	110.60 GB	265.83 GB

借助 Strata，以上三个版本均可在单张 12 GB 显存的显卡上运行。

每行中的 Engram 列均为相同的共享文件。视觉投影器文件 mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf（0.97 GB）同样适用于所有版本。此处 GB 按 10^9 字节计算。

IQ2_XXS 是更稳健的折中点：它在所有三个基准测试中均与基模型表现持平，达到基模型任务平均得分（93.54）的 100.0%。IQ1_M 版本相比之下体积缩减了 17.5 GB，任务平均得分仅损失 1.2 分。

显存/内存要求

仅 Backbone 部分包含 Transformer 权重。Engram 表采用稀疏读取机制（每个 token 仅读取一行），因此可常驻磁盘并通过内存映射（memory-mapped）访问，无需占用大量显存或系统内存。配合 Strata，Backbone 也无需完全放入显存：每个版本均可在 12 GB 显存 GPU 上运行，模型的其余部分存放在系统内存中。建议将 Engram 文件放置在 SSD 上，以保持低开销的分页调度。

---

评测结果

所有模型均与官方发布的基模型 deepseek-ai/DeepSeek-V4.1-Flash-1017（密集权重 FP8，路由专家 FP4）进行对比测试。我们汇报了三个推理与生成基准：AIME25、GPQA-Diamond 以及 LiveCodeBench v6。任务平均分（Task avg）为三项指标的均值；恢复率（Recovery）为相对基模型的任务平均分比例。

本卡片中的所有数据均在 思考深度（reasoning effort）为 50 的设定下测得。

变体	bpw	AIME25↑	GPQA-D↑	LCB v6↑	任务平均得分↑	恢复率
基模型 (FP8 / FP4)	n/a	100.00	90.91	89.71	93.54	100.0%
GSQ-RCO IQ1_M	1.75	100.00	88.38	88.57	92.32	98.7%
GSQ-RCO IQ2_XXS	2.00	100.00	90.91	89.71	93.54	100.0%
GSQ-RCO IQ2_XS	2.25	100.00	90.40	89.71	93.37	99.8%

IQ2_XXS（2.00 bpw）保留了基模型 100.0% 的任务平均能力：在 AIME25（100.00）、GPQA-Diamond（90.91）和 LiveCodeBench v6（89.71）上完全追平基模型。IQ2_XS（2.25 bpw）在 AIME25 和 LiveCodeBench v6 上与基模型一致，在 GPQA-Diamond 上取得 90.40 分，整体恢复率为 99.8%。体积最小的 IQ1_M（1.75 bpw）在 AIME25 上同样拿下满分 100.00，整体保持了 98.7% 的水平。

---

使用方法

运行环境支持： llama.cpp 自 b11870（2026 年 10 月 25 日）版本起支持 DeepSeek-V4.1（包含图像功能），本仓库文件自 2026 年 10 月 27 日起采用该布局。在此日期之前下载的文件可在本地进行格式转换，详见“更新已有下载文件”。这些 GGUF 文件同样受 Strata 原生支持。

Strata 引擎

Strata 允许在单张配备 12 GB 显存的 NVIDIA GPU 上运行这些 GGUF 模型，提供一键式安装程序以及本地运行的 OpenAI 兼容 API 端点。这些模型构建版本与 Strata 协同优化，专门针对本地推理设计。

llama.cpp

要求 llama.cpp 版本为 b11870 或更新版本。

bash hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
--include "IQ2_XXS" "Engram" --local-dir .

ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf

llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf
-p "Explain mixed-precision quantization." -ngl 99

-m 参数指定首个分片；llama.cpp 会自动打开同目录下的分片 2 至 6。若要尝试其他版本，只需下载其专属分片（例如 --include "*IQ1_M*"）并创建对应软链接即可。在 Windows 系统上，可使用 mklink /H <link> DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf 创建硬链接；在目标文件夹中运行 convert_ds41_upstream.py 也会自动补齐缺失的链接。

Engram 表常驻磁盘内存映射，每个 token 仅按需读取一行（llama.cpp 默认的 --lazy-mode auto 会对超过 4 GiB 的张量自动执行此操作），因此既不占用显存也不占用内存。详见“显存/内存要求”。

图像多模态推理

下载视觉投影器并通过 --mmproj 传入：

bash hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
--include "mmproj-*" --local-dir .

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf
--mmproj mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf -ngl 99

使用 llama-mtmd-cli 时，还需传入 --jinja 参数以使用 GGUF 中内嵌的对话模板。

对话模板与思考深度（Reasoning Effort）

基础仓库未附带 Jinja 对话模板；其 Prompt 格式由 DeepSeek 的编码器 encoding/encoding.py 定义。本 GGUF 模型嵌入了改编自 Unsloth DeepSeek-V4.1 发布版本 的模板，并对照官方编码器校验了文本对话、工具调用（tool use）以及图像格式化行为。思考深度支持 1 到 100 的整数（也支持数字字符串），或 low、high、max（分别对应 50、75、100）。默认思考深度为 DeepSeek 的 high（75），且该配置仅在思考模式（thinking mode）下生效。结构化图像块（包括工具调用返回的图片）会保留图像占位符。图像推理需要配合视觉投影器，详见“图像多模态推理”。

bash llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf -ngl 99
--reasoning on --reasoning-effort 50

50 是本文档所有结果测定时所用的设置。在单次请求中，可在 Chat Completion 请求体中传入 "reasoning_effort": "50"（使用字符串形式）。--reasoning off 可关闭思考模式。

更新已有下载文件

在 2026 年 10 月 27 日之前下载的文件（每个版本 4 个分片以及单独的 DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf）是针对早期 llama.cpp 分支构建的，无法直接在新版 llama.cpp 中加载。convert_ds41_upstream.py 可以在无需重新下载模型的前提下，逐字节将其转换为本仓库当前的最新格式。脚本需要 Python 3.8+ 且无需额外依赖库，只会从本仓库抓取两个微小文件（chat_template.jinja 与 exp_probs_b_vl.f32，共计 80 KB）。

bash cd /path/to/downloaded/model-folder hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF convert_ds41_upstream.py --local-dir . python3 convert_ds41_upstream.py .

它会转换文件夹内的所有版本以及 mmproj，并在旧分片同级写入新的 Backbone 分片；每个写入的张量都会被重新读出并与源数据对比校验。Engram 文件为原地转换：仅修改其文件头，其 110 GB 的数据表保持不变，脚本还会自动创建分片 6 的软链接。所需空闲磁盘空间即为新分片的大小（每个版本约 121 至 155 GB）。使用 --delete-old 参数后，旧分片在新分片验证完毕后会立即被删除，因此只需备足单个构建版本的剩余空间即可。转换后的新分片 1 会直接集成 chat_template.jinja 中的模板，因此不再需要运行 patch_chat_template.py。

---

量化流程

1. 构建张量候选库（Per-tensor database）： 使用 GSQ 将每个权重张量在候选的各种 GGUF 量化类型下分别进行量化，生成可供搜索的张量变体数据库。
2. RCO 优化搜索（RCO search）： 在预算约束下通过黎曼流形搜索为每个张量分配一种量化类型，确保 Backbone 的平均比特宽度符合目标设定。
3. 组装拼接（Assembly）： 将选定的各张量变体拼接打包成标准的 GGUF 文件。

密集张量（注意力机制、共享专家、Embedding 词嵌入层、输出头）与路由专家矩阵采用两套独立的量化阶梯，因为在任何给定预算下，路由专家的比特数都显著低于密集通道：密集张量在 Q5_K、Q6_K 和 Q8_0 之间选择，而路由专家则在 IQ1_S 至 IQ3_S 之间选取。搜索过程中的教师模型是官方发布的检查点本身（路由专家保留原生的 FP4），校准数据由模型自身生成。

参考实现：GSQ 代码位于 IST-DASLab/GSQ，RCO 代码位于 IST-DASLab/RCO。

可复现性产物

每个发布的版本均附带审计构建过程所需的文件：

文件	内容说明
tensor-allocation/DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>.rco-allocation.txt	该版本中分配给每个张量的量化类型（覆盖全部 6 个分片，共 1043 个张量），包含量化类型分布直方图、目标比特数以及实测比特数。这是 RCO 搜索的直接产物，因此无需加载模型文件即可审阅精度分配情况。
convert_ds41_upstream.py、chat_template.jinja、exp_probs_b_vl.f32	用于转换 2026 年 10 月 27 日之前下载文件的转换脚本（详见“更新已有下载文件”）、GGUF 内嵌的对话模板，以及转换脚本添加的视觉路由偏置参数（40 x 384 F32，对应原始检查点中的 ffn.gate.bias_vl）。

---

引用

如果您使用了这些模型或算法，请引用以下两篇论文：

bibtex @article{gsq2026, title = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling}, author = {Dadgarnia, Alireza an

Update, October 27, 2026. The files now use llama.cpp's own DeepSeek-V4.1 layout and load in llama.cpp b11870 or newer, including images.

• Downloaded the earlier files? You do not need to download the model again. Run convert_ds41_upstream.py on your model folder: it rewrites the files locally into exactly the current ones and fetches only 80 KB. See Updating an existing download.

• New download? The shared Engram file is shard 6 of every build, so link it once per build you use (<ID> is IQ1_M, IQ2_XXS or IQ2_XS; see llama.cpp):

  ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-00006-of-00006.gguf
  

---

Overview

This repository provides GGUF quantizations of DeepSeek-V4.1-Flash-1017 at three sizes: IQ1_M (1.75 bits per weight), IQ2_XXS (2.00) and IQ2_XS (2.25). In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with 763.2B parameters : 384 routed experts per layer , of which 6 are active per token, and a 1M-token context window. The routed experts dominate the parameter count, so they also dominate the size budget, and the search spends most of its freedom there, assigning a quantization type per layer (not per expert, which GGUF cannot express).

With Strata, every build runs on a single GPU with 12 GB of VRAM. See Usage.

Method summary. GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.

Method	Description
GSQ (Gumbel-Softmax Quantization, paper, code)	Post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation. GSQ closes most of the gap between scalar and vector quantization at 2 to 3 bits while remaining deployable in standard scalar formats such as GGUF.
RCO (Riemannian Constrained Optimization, paper, code)	Assigns one of K quantization types to each of N tensors under a total size budget. The budget constraint is reformulated as a smooth Riemannian manifold in logit space, which permits gradient-based optimization directly on the task loss while enforcing the budget exactly, without constraint-specific hyperparameter tuning.

Both methods were developed at the Deep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

---

Available files

Each build is a backbone GGUF in five shards, named after its quantization identifier: DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-0000i-of-00006.gguf, i = 1 to 5. Shard 1 holds the metadata only, shards 2 to 5 the transformer weights, 551.9B parameters. The bpw is the average bit-width over these, which is what the search controls.

The Engram file , DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf, holds the model's two Engram lookup tables (196.6B parameters) at IQ4_NL, 4.50 bpw. It is identical for all builds, so it is stored once and needs to be downloaded only once, also when trying several builds. llama.cpp loads it as shard 6 of a build, so each build needs a link to it named ...-<ID>-00006-of-00006.gguf (one command, see llama.cpp). The tables are a lookup, not a matmul weight, so they are held at a fixed type in every build and excluded from the search. They are quantized in the evaluated models, not just the shipped ones.

Build	bpw	Backbone	Engram	Total	Notes
IQ1_M	1.75	120.72 GB	110.60 GB	231.32 GB	Smallest
IQ2_XXS	2.00	138.24 GB	110.60 GB	248.84 GB	Recommended; closest to the base model
IQ2_XS	2.25	155.23 GB	110.60 GB	265.83 GB

All three builds run on a single 12 GB GPU with Strata.

The Engram column is the same shared file in every row. The vision projector, mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf (0.97 GB), also serves all builds. GB is 109 bytes.

The IQ2_XXS build is the stronger operating point: it matches the base model on all three benchmarks and so reaches 100.0% of its task average (93.54). The IQ1_M build trades 1.2 points of task average for 17.5 GB less.

Memory requirements

Only the backbone holds transformer weights. The Engram tables are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM. With Strata, the backbone does not need to fit in VRAM either: every build runs on a 12 GB GPU , with the rest of the model held in system memory. Keep the Engram file on an SSD so the paging stays cheap.

---

Results

All models are evaluated against the base model deepseek-ai/DeepSeek-V4.1-Flash-1017 as released (FP8 dense weights, FP4 routed experts). We report three reasoning and generation benchmarks: AIME25 , GPQA-Diamond , and LiveCodeBench v6. Task average is the mean of the three; recovery is the task average relative to the base model.

All numbers in this card were measured at reasoning effort 50.

Variant	bpw	AIME25↑	GPQA-D↑	LCB v6↑	Task avg↑	Recovery
Base (FP8 / FP4)	n/a	100.00	90.91	89.71	93.54	100.0%
GSQ-RCO IQ1_M	1.75	100.00	88.38	88.57	92.32	98.7%
GSQ-RCO IQ2_XXS	2.00	100.00	90.91	89.71	93.54	100.0%
GSQ-RCO IQ2_XS	2.25	100.00	90.40	89.71	93.37	99.8%

IQ2_XXS (2.00 bpw) keeps 100.0% of the base model's task average: it matches the base model on AIME25 (100.00), GPQA-Diamond (90.91) and LiveCodeBench v6 (89.71). IQ2_XS (2.25 bpw) matches it on AIME25 and LiveCodeBench v6 and scores 90.40 on GPQA-Diamond, 99.8% overall. The smallest build, IQ1_M (1.75 bpw), also scores 100.00 on AIME25 and retains 98.7% overall.

---

Usage

Runtime support. llama.cpp supports DeepSeek-V4.1, including images, since release b11870 (October 25, 2026), and the files here use its layout since October 27, 2026. Downloads from before that date can be converted locally, see Updating an existing download. The GGUFs are also supported in Strata.

Strata Engine

Strata runs these GGUFs on a single NVIDIA GPU with 12 GB of VRAM , with a one-click installer and an OpenAI-compatible endpoint on localhost. The builds were optimized together with Strata for local inference.

llama.cpp

Requires llama.cpp b11870 or newer.

hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "*IQ2_XXS*" "*Engram*" --local-dir .


ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf

llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  -p "Explain mixed-precision quantization." -ngl 99

-m takes the first shard; llama.cpp opens shards 2 to 6 from the same directory. To try another build, download only its shards (--include "*IQ1_M*") and add its link. On Windows, mklink /H <link> DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf makes the link; convert_ds41_upstream.py, run on the folder, also creates missing links.

The Engram table stays memory-mapped on disk and is read one row per token (llama.cpp's default --lazy-mode auto does this for tensors over 4 GiB), so it needs neither VRAM nor RAM. See Memory requirements.

Images

Download the vision projector and pass it with --mmproj:

hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "mmproj-*" --local-dir .

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  --mmproj mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf -ngl 99

With llama-mtmd-cli, also pass --jinja so that the chat template embedded in the GGUF is used.

Chat template and reasoning effort

The base repository ships no Jinja chat template; its prompt format is defined by DeepSeek's encoder, encoding/encoding.py. The GGUFs embed a template adapted from Unsloth's DeepSeek-V4.1 release, checked against that encoder for text chat, tool use, and image formatting. Reasoning effort accepts an integer from 1 to 100 (also a numeric string), or low, high, max (50, 75, 100). The default effort is DeepSeek's high (75), and effort applies only in thinking mode. Structured image blocks, including images in tool results, retain their image placeholders. Image inference needs the vision projector, see Images.

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf -ngl 99 \
  --reasoning on --reasoning-effort 50

Effort 50 is the setting all results in this card were measured with. Per request, set "reasoning_effort": "50" in the chat completion body (a string, also for numbers). --reasoning off disables thinking.

Updating an existing download

Files downloaded before October 27, 2026 (four shards per build and a separate DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf) were written for an earlier llama.cpp branch and do not load in llama.cpp. convert_ds41_upstream.py turns them into the current files of this repository, byte for byte, without downloading the model again. It needs Python 3.8+ and no extra packages, and fetches only two small files of this repository (chat_template.jinja and exp_probs_b_vl.f32, 80 KB).

cd /path/to/downloaded/model-folder
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF convert_ds41_upstream.py --local-dir .
python3 convert_ds41_upstream.py .

It converts every build in the folder and the mmproj, and writes the new backbone shards next to the old ones; every written tensor is read back and compared with its source. The Engram file is converted in place: only its header changes, its 110 GB of tables stay as they are, and the script adds the shard-6 links. The free space needed is the size of the new shards (121 to 155 GB per build). With --delete-old, the old shards of a build are deleted as soon as its new ones are verified, so space for one build suffices. The new first shards take the chat template from chat_template.jinja, so patch_chat_template.py is not needed.

---

Quantization procedure

1. Per-tensor database. Each weight tensor is quantized at every candidate GGUF quantization type with GSQ, yielding a searchable database of quantized tensor variants.
2. RCO search. The budget-constrained Riemannian search assigns one quantization type per tensor such that the backbone's average bit-width meets the target.
3. Assembly. The selected per-tensor variants are stitched into a standard GGUF file.

The dense tensors (attention, shared experts, embeddings, output head) and the routed-expert matrices draw on separate quantization ladders, because the routed experts sit at a much lower bit-width than the dense path at any given budget: the dense tensors choose among Q5_K, Q6_K and Q8_0, the routed experts among types from IQ1_S to IQ3_S. The search's teacher is the released checkpoint itself, with the routed experts in their native FP4, and its calibration data was generated by the model.

Reference implementations: GSQ at IST-DASLab/GSQ and RCO at IST-DASLab/RCO.

Reproducibility artifacts

Each released build ships the files needed to audit how it was built:

File	Contents
tensor-allocation/DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>.rco-allocation.txt	The quantization type assigned to every tensor of that build (all six shards, 1043 tensors), with a quant-type histogram and the target and measured bit-width. This is the RCO search result, so the allocation can be inspected without opening the model.
convert_ds41_upstream.py, chat_template.jinja, exp_probs_b_vl.f32	The converter for downloads from before October 27, 2026 (see Updating an existing download), the chat template the GGUFs embed, and the vision router bias the converter adds (40 x 384 F32, ffn.gate.bias_vl of the original checkpoint).

---

Citation

If you use these models or methods, please cite both papers:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

---

Acknowledgements

We thank Verda and Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.

---

License

These quantized weights inherit the license of the base model (DeepSeek-V4.1-Flash-1017 , MIT). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.