DeepSeek-V4.1-Flash-1017-abliterated GGUF

🛡️ Orca 新品 — OrcaCyber Zero 1.5 针对网络安全进行后训练：支持漏洞分析、漏洞复现、红队测试与攻击性安全研究。CyberGym Level 1 pass@1 达 98.01% · 1M 上下文 · 函数调用（function calling）· OpenAI 兼容 API。

🛡️ Orca 新品 — OrcaCyber Flash 1.0 没有 236 GB 显存/内存配置的机器？可在 OrcaRouter 上将 DeepSeek V4.1 Flash Abliterated 作为托管的 OpenAI 兼容 API 运行：无需下载、无需 GPU、只需一个 API 密钥。
🔌 Orca 新品 — Orca Connect 想在 llama.cpp 中运行此 GGUF？Orca Connect 可将你桌下的硬件设备挂接到常用的 OrcaRouter API 密钥下，与 200+ 托管模型并列，随时可通过笔记本电脑、手机或云端 Agent 访问。仅需出站连接：无需端口转发、无需打洞、无需配置防火墙规则。使用自有硬件处理的请求费用为 $0，且通过 orca-connect claude 命令可单次会话将 Claude Code 直连至你的本地机器。

763B 参数无审查版 DeepSeek-V4.1-Flash-1017，单台机器即可运行。

内存占用仅 236 GB · 197B 参数 Engram 直接从 SSD 流式读取 · 基于 llama.cpp

OrcaRouter AI Gateway · X · Discord · GitHub · 所有模型

---

完整版 DeepSeek-V4.1-Flash-1017，本地运行。

这是去拒绝（abliterated / 移除拒绝响应机制）的 DeepSeek-V4.1-Flash-1017 的 GGUF 构建版本。该模型为 763B 参数的 MoE 架构，拥有 384 个路由专家（top-6）、197B 参数的 Engram n-gram 记忆表、压缩稀疏注意力机制以及 1M token 上下文。路由专家是唯一耗费位宽预算的部分：每一层的专家精度均由实测的 KL 散度决定，而非人为设定规则。

Engram 表是一个每 token 读取 48 行的查找表。llama.cpp 通过内存映射（mmap）直接从磁盘按需读取，因此它驻留在你的 SSD 上，而非 RAM 中。只有其余部分需要装入内存/显存。

---

⚠️ 免责声明 — 使用前必读

该模型基于去拒绝（abliterated）底模构建，其安全对齐程度已被大幅削弱：模型已通过正交化从所有写入残差流的矩阵中消除了拒绝方向。它将顺从执行许多原版 DeepSeek-V4.1-Flash-1017 会拒绝的有害、不道德、冒犯性或非法请求。

• 本模型仅限用于合法研究：包括可解释性分析、AI 安全与拒绝机制研究、红队测试（red-teaming）及鲁棒性评估。
• 使用者对如何使用该模型及其生成的所有内容承担全部责任。在进行任何实际部署前，必须添加您自己的安全审核、内容审查与防滥用层。
• 使用必须遵守继承自基础模型的 MIT License 以及所有适用法律。
• 作者对任何滥用行为不承担任何责任。其输出不代表上传者或 DeepSeek 的观点。

---

选择构建版本

构建版本	内存占用	下载大小	专家位宽	KLD ↓	Top-1 ↑	相比原版 PPL 变化
Mix-256G	235.8 GB	346.5 GB	3.36 bpw	0.051	93.7%	+2.4%
Mix-128G	114.0 GB	225.0 GB	1.59 bpw	0.520	78.5%	+54.7%

推荐使用 Mix-256G 版本。 它需要一台具备 256 GB RAM、统一内存或 VRAM 的设备。在每个专家权重 3.36 位的精度下，模型在留出测试集上的下一词选择与未量化模型的一致率达到 93.7%。

Mix-128G 虽适配 128 GB 内存，但无法替代主力版本。 仅将所有专家量化为 llama.cpp 的最小格式（IQ1_S）就已占据 111 GB，因此 128 GB 设备已达极限：困惑度（PPL）上升 55%，且每五个 token 中就有一个与原版不同。适合用来尝鲜体验模型，不建议依赖其输出。

内存占用 指除 Engram 表外的全部参数。下载大小 包含了 110.6 GB 保留在磁盘上的 Engram 表。

---

评测数据

两个构建版本均在留出文本上与同一 Checkpoint 的无损 GGUF（MXFP4 专家、BF16 Dense、MXFP8 Engram，比特级精准转换）进行了对比测试：测试集为 wikitext-2 测试集，字符范围 [0, 300000)，共 32 个 2048 token 窗口，全词表 KLD 计算。精度搜索阶段从未接触过此文本。各构建版本还与相同权重、相同重要性矩阵量化出的类似体积 llama.cpp 自带混合方案进行了对比：

| 内存占用 | KLD ↓ | KLD p99 | Top-1 ↑ | PPL（原版 2.259） ---|---|---|---|---|--- Mix-256G（本仓库） | 235.8 GB | 0.0507 | 0.64 | 93.74% | 2.313 (+2.4%) llama.cpp IQ3_XXS | 212.2 GB | 0.2704 | 3.16 | 85.12% | 2.841 (+25.8%) Mix-128G（本仓库） | 114.0 GB | 0.5202 | 4.95 | 78.50% | 3.494 (+54.7%) llama.cpp IQ1_S | 111.1 GB | 1.7307 | 9.02 | 54.80% | 11.858 (×5.2)

在 Engram 表为 MXFP8 精度下测得。 发布的实际文件采用 IQ4_NL 格式（体积为 110.6 GB 而非 202.8 GB）。该格式替换未重新跑分。根据 unsloth 在该模型上的实测数据，Engram 的精度对 Top-1 准确率几乎无影响。

方法说明。 上述指标为 2k 上下文下留出文本的全词表 KLD 和下一词一致率，而非下游基准测试结果。

去拒绝（Abliterated）模型的拒绝率与能力评测

在未量化的去拒绝 checkpoint 上测得，未在这些 GGUF 构建版本上重复测量。 上文的 KLD 反应了各量化版本偏离该 checkpoint 的程度。

测试设置：

• 硬件与运行时：8×H200，在张量并行（Tensor Parallelism）下运行该 checkpoint 自带的参考实现代码。
• 解码参数：贪婪解码（greedy），thinking_mode="chat"，生成 64 个新 token。
• 评分标准：拒绝行为由基于规则的开头短语分类器判定。caveat 表示模型给出了回答，但在回答中附带了免责声明。请将这些数字视为参考指标，而非正式论文级数据。

安全表现：有害提示词拒绝率（数值越低 = 越无审查 / 越顺从）。 全部 500 个提示词的加权均值：0.074，而基础模型 baseline 为 0.874。

基准测试	n	基础版	Abliterated 版	Δ	Caveat（免责比例）
JBB-harmful	100	0.950	0.030	−97%	0.360
MaliciousInstruct	100	0.930	0.020	−98%	0.610
ForbiddenQuestions	150	0.667	0.073	−89%	0.447
SimpleSafetyTests	50	1.000	0.120	−88%	0.560
AdvBench	100	0.990	0.150	−85%	0.550

过度拒绝表现：良性提示词被误拒绝率（数值越低越好）。 全部 350 个提示词的加权均值：0.006，而基础模型 baseline 为 0.069。

基准测试	n	基础版	Abliterated 版	Δ	Caveat（免责比例）
XSTest-safe	250	0.052	0.008	−85%	0.028
JBB-benign	100	0.110	0.000	−100%	0.140

模型核心能力保留情况：

基准测试	n	基础版	Abliterated 版
MMLU（零样本，从首个生成 token 读取答案选项字母）	300	0.910	0.920

MMLU 分数保持不变；1 个百分点的差异完全在 300 道题目的采样误差范围内。这是唯一进行测量的能力指标。

---

精度分配策略与搜索过程

每个档位均为独立搜索，显存/内存预算严格限制在必须常驻内存的字节大小：

1. 测量。 针对每组张量，搜索算法都会实测将其上调一级精度或下调一级精度带来的真实 KLD 变化。一个“组”对应某一层的专家（gate+up 与 down 分开处理），或某一类 Dense 矩阵。每次测量均调用 llama.cpp 自身的 kernel 运行。
2. 选择。 利用背包算法挑选出能够精确匹配内存预算的调整组合。
3. 接受。 仅保留候选 KLD 实测更低的优化步。搜索过程循环迭代直至收敛。

经搜索优化后的 256G 版本将位宽用在了最关键的部位：

层数	专家配置
24–30 及 32 层	完全保持 DeepSeek 原生 MXFP4
15, 20, 23, 31, 33, 37, 39 层	ffn_down 采用 MXFP4
0–3, 6, 7 层	gate/up 采用 IQ2_S
其余所有部分	IQ3_XXS / IQ3_S

在相同预算下，均匀分配策略得到的 KLD 为 0.146，而搜索优化版（样本内）达到 0.045。在均匀分配的基础上，仅将任意单个中间层升级为 MXFP4，就能消除约 0.04 的 KLD。中间层的量化误差在后续网络中呈现级联放大趋势，而非简单累加。

在 128G 版本中，几乎已经没有可自由调度的余量。除少数后层采用 IQ1_M 或 IQ2_XXS 外，各层专家基本均为 IQ1_S。

其它结构分配如下：

• 注意力层与共享专家在 256G 版中为 Q5_K–Q8_0，在 128G 版中为 Q4_K–Q6_K。
• 路由网络（Router）、Norms 以及超连接混合器（hyper-connection mixers）保持原厂精度不变。
• 重要性矩阵（imatrix）基于 61.4 万 token 的中英文对话和维基百科语料计算得出。

---

仓库文件清单

路径	文件大小	内容说明
Mix-256G/…-00001-of-00007.gguf … 00007	346.5 GB	推荐构建版本；00002 为 Engram 表（110.6 GB）
Mix-128G/…-00001-of-00005.gguf … 00005	225.0 GB	128 GB 构建版本；00002 为 Engram 表（110.6 GB）
mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf	0.97 GB	图像输入视觉投影器（vision projector），无损 BF16
mmproj-DeepSeek-V4.1-Flash-1017-abliterated-F16.gguf	0.97 GB	同上，F16 格式，适用于不支持 BF16 的后端
imatrix.gguf	1.5 GB	两个构建版本量化时使用的重要性矩阵

视觉投影器可与任一构建版本配合使用。它是从**去拒绝（abliterated）**的 checkpoint 转换而来，其图像入口点（aligner.w2）及图像标记均包含了去拒绝修改，因此切勿替换为原版模型的 mmproj。DeepSeek-V4.1 对话模板已内置于各构建版本中。本仓库未包含 DSpark 投机推测解码器。

---

部署与推理（llama.cpp）

DeepSeek-V4.1 尚未合并入 llama.cpp master 分支。需编译添加该支持的 PR：ggml-org/llama.cpp#30817。本模型文件在 commit b72050d03 下构建并测试：

bash git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/30817/head:pr-30817 && git checkout b72050d03 cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli

hf download orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF
--include "Mix-256G/" "mmproj--BF16.gguf" --local-dir ./dsv41-gguf

llama-server -m dsv41-gguf/Mix-256G/DeepSeek-V4.1-Flash-1017-abliterated-Mix-256G-00001-of-00007.gguf
--mmproj dsv41-gguf/mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf
-ngl 99 -c 32768 -np 1 --no-reasoning-preserve

llama-server -m dsv41-gguf/Mix-256G/DeepSeek-V4.1-Flash-1017-abliterated-Mix-256G-00001-of-00007.gguf
--mmproj dsv41-gguf/mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf
-ngl 0 --no-repack --fit off -c 8192 -np 1 --no-reasoning-preserve

• 请将文件存放于 NVMe SSD，并保持 mmap 开启（默认行为）。 Engram 表始终不常驻 GPU，而是按需惰性读取。若开启 --mlock 或 --no-mmap，会导致全部 110 GB 数据被强行载入 RAM。
• 纯 CPU 运行说明：必须传入 --no-repack。 在 b72050d03 版本的 PR 中，CPU 后端的权重重新打包会导致生成某个没有后端支持的操作，启动时将抛出 GGML_ASSERT(*cur_backend_id != -1) 错误中断。加上 --no-repack --fit off 参数可正常运行。
• 多模态/图像输入： 传入 --mmproj 参数，并发送 OpenAI 风格的 image_url 内容对象（支持 base64 data URL）。
• 配备 256 GB 系统 RAM 与单张 GPU： 可添加 --cpu-moe 参数，将路由专家保留在内存中，注意力机制在 GPU 上计算。此配置未在此处实测速度。
• --no-reasoning-preserve： 该参数会从当前 Prompt 中剥除之前轮次的思考过程（thinking），与 DeepSeek 官方参考编码器的行为一致。开启后，在 DeepSeek 测试用例上 llama.cpp 渲染的 Prompt 与 encoding.py 逐 token 完全一致。
• 思考模式默认开启。 传入 "chat_template_kwargs": {"thinking_mode": "chat"} 可直接输出回答；或传入 "reasoning_effort": "low" | "medium" | "high" | "max"（或 1–100）调节思考深度。思考链将通过 message.reasoning_content 返回。
• 务必设置 System Prompt。 若未提供系统提示词，模型在思考模式下偶尔会出现自称混淆。

在短提示词、单并发请求下的实测速度：

构建版本	硬件配置	Prompt 处理	生成速度
Mix-256G	2 x H200 (CUDA)	~110 tok/s	~70 tok/s
Mix-128G	1 x H200 (CUDA)	~110 tok/s	~84 tok/s
Mix-256G + mmproj	纯 CPU，192 核服务器分配 64 线程	~52 tok/s	~10 tok/s

CPU 测试时机器正在并发运行不相关的 GPU 训练任务，因此该数据代表性能下限而非峰值。未测量 Apple Metal 及部分卸载（--cpu-moe）的速度。

图像输入功能在 CPU 环境下通过 DeepSeek 官方示例图片进行了校验。当被询问各为何种食物及食用部位时，模型均正确作答：

• 胡萝卜（carrots）：食用主根（the taproot）
• 玉米（corn）：食用玉米棒上的玉米粒（the kernels on the cob）

---

模型详情

属性	描述
基础模型	deepseek-ai/DeepSeek-V4.1-Flash-1017 的去拒绝（abliterated）版本
模型架构	llama.cpp 中的 deepseek41。40 层，Hidden 维度 5120，压缩稀疏注意力，LoRA 分解输出投影（8 组），384 个路由专家（top-6） + 1 个共享专家，Engram 记忆查找表

DeepSeek-V4.1-Flash-1017-abliterated GGUF

🛡️ New from Orca —OrcaCyber Zero 1.5 Post-trained for cybersecurity: vulnerability analysis, vulnerability reproduction, red teaming & offensive-security research. 98.01% pass@1 on CyberGym Level 1 · 1M context · function calling · OpenAI-compatible API.

🛡️ New from Orca —OrcaCyber Flash 1.0 No 236 GB box? Run DeepSeek V4.1 Flash Abliterated as a hosted, OpenAI-compatible API on OrcaRouter: no download, no GPUs, one API key.
🔌 New from Orca —Orca Connect Running this GGUF in llama.cpp? Orca Connect puts the rig under your desk behind your usual OrcaRouter API key, next to 200+ hosted models, reachable from a laptop, a phone or a cloud agent. Outbound only: no port forwarding, no tunnel, no firewall rule. Requests your own hardware serves cost $0 , and orca-connect claude points Claude Code at your rig for one session.

The 763B uncensored DeepSeek-V4.1-Flash-1017, on one box.

236 GB in memory · 197B-parameter Engram streamed from your SSD · llama.cpp

OrcaRouter AI Gateway · X · Discord · GitHub · All Models

---

The full DeepSeek-V4.1-Flash-1017, run locally.

These are GGUF builds of the abliterated (refusal-removed) DeepSeek-V4.1-Flash-1017. The model is a 763B-parameter MoE with 384 routed experts (top-6), a 197B-parameter Engram n-gram memory, compressed sparse attention and a 1M-token context. The routed experts are the only part the bits were spent on: every layer's experts got their own precision from measured KL divergence, not a hand-set rule.

The Engram table is a lookup that reads 48 rows per token. llama.cpp maps it from disk and reads it on demand, so it lives on your SSD, not in RAM. What has to fit in memory is everything else.

---

⚠️ Disclaimer — read before use

This model is built on an abliterated base whose safety alignment has been substantially reduced : the refusal direction is orthogonalized out of every matrix that writes the residual stream. It will comply with many harmful, unethical, offensive, or illegal requests that the original DeepSeek-V4.1-Flash-1017 would refuse.

• It is released strictly for legitimate research : interpretability, AI-safety and refusal-mechanism study, red-teaming, and robustness evaluation.
• You assume full responsibility for how you use it and for everything it generates. Add your own safety, moderation, and abuse-prevention layers before any deployment.
• Use must comply with the MIT License inherited from the base model and with all applicable law.
• The authors accept no liability for misuse. Its outputs do not reflect the views of the uploaders or of DeepSeek.

---

Pick a build

Build	In memory	Download	Experts	KLD ↓	Top-1 ↑	PPL vs. original
Mix-256G	235.8 GB	346.5 GB	3.36 bpw	0.051	93.7%	+2.4%
Mix-128G	114.0 GB	225.0 GB	1.59 bpw	0.520	78.5%	+54.7%

Mix-256G is the build to use. It needs a box with 256 GB of RAM, unified memory, or VRAM. At 3.36 bits per expert weight, the model's next-token choice matches the unquantized model on 93.7% of held-out tokens.

Mix-128G fits 128 GB, but it is not a substitute. All experts at llama.cpp's smallest type (IQ1_S) already take 111 GB, so a 128 GB box is at the floor: perplexity is 55% higher and one token in five differs from the original. Use it to try the model, not to rely on it.

In memory is everything except the Engram table. Download adds the 110.6 GB Engram table, which stays on disk.

---

Evaluation

Both builds were scored against the lossless GGUF of the same checkpoint (MXFP4 experts, BF16 dense, MXFP8 Engram, converted bit-exact) on held-out text : wikitext-2 test, characters [0, 300000), 32 windows of 2048 tokens, full-vocabulary KLD. The precision search never saw this text. Each build is compared with llama.cpp's own mixture at a similar size, quantized from the same weights with the same importance matrix:

| In memory | KLD ↓ | KLD p99 | Top-1 ↑ | PPL (original 2.259)
---|---|---|---|---|---
Mix-256G (this repo) | 235.8 GB | 0.0507 | 0.64 | 93.74% | 2.313 (+2.4%)
llama.cpp IQ3_XXS | 212.2 GB | 0.2704 | 3.16 | 85.12% | 2.841 (+25.8%)
Mix-128G (this repo) | 114.0 GB | 0.5202 | 4.95 | 78.50% | 3.494 (+54.7%)
llama.cpp IQ1_S | 111.1 GB | 1.7307 | 9.02 | 54.80% | 11.858 (×5.2)

Measured with the Engram table at MXFP8. The released files carry it at IQ4_NL (110.6 GB instead of 202.8 GB). The swap was not rescored. Per unsloth's measurements on this model, the Engram precision barely moves top-1.

Method note. These are full-vocabulary KLD and next-token agreement on held-out text at 2k context, not a downstream benchmark.

Refusal and capability of the abliterated model

Measured on the unquantized abliterated checkpoint, not re-measured on these GGUF builds. The KLD above is how far each build sits from that checkpoint.

Setup.

• Hardware and runtime: 8×H200, the checkpoint's own reference implementation under tensor parallelism.
• Decoding: greedy, thinking_mode="chat", 64 new tokens.
• Scoring: refusal is judged by a rule-based opening-phrase classifier. caveat means the model answered but wrapped the answer in a disclaimer. Treat the numbers as indicative, not publication-grade.

Safety: harmful-prompt refusal (lower = more uncensored). Weighted mean over all 500 prompts: 0.074 , against 0.874 for the base checkpoint.

Benchmark	n	Base	Abliterated	Δ	Caveat
JBB-harmful	100	0.950	0.030	−97%	0.360
MaliciousInstruct	100	0.930	0.020	−98%	0.610
ForbiddenQuestions	150	0.667	0.073	−89%	0.447
SimpleSafetyTests	50	1.000	0.120	−88%	0.560
AdvBench	100	0.990	0.150	−85%	0.550

Over-refusal: benign prompts wrongly refused (lower = better). Weighted mean over all 350 prompts: 0.006 , against 0.069 for the base checkpoint.

Benchmark	n	Base	Abliterated	Δ	Caveat
XSTest-safe	250	0.052	0.008	−85%	0.028
JBB-benign	100	0.110	0.000	−100%	0.140

Capability retention.

Benchmark	n	Base	Abliterated
MMLU (zero-shot, answer letter read from the next token)	300	0.910	0.920

MMLU is unchanged; the one-point difference is within the sampling error of 300 questions. It is the only capability measured.

---

How the precision was chosen

Each tier was searched separately, with the budget set on the bytes that have to be in memory:

1. Measure. For every group of tensors, the search measures the real KLD of moving it one precision level up and one level down. A group is one layer's experts, with gate+up and down kept apart, or one kind of dense matrix. Each measurement runs llama.cpp's own kernels.
2. Choose. A knapsack picks the combination of moves that fits the budget exactly.
3. Accept. Only steps whose candidate KLD was actually measured lower are kept. The search repeats until it converges.

The searched 256G build puts its bits where they count:

Layers	Experts
24–30 and 32	entirely at DeepSeek's own MXFP4
15, 20, 23, 31, 33, 37, 39	ffn_down at MXFP4
0–3, 6, 7	gate/up at IQ2_S
everything else	IQ3_XXS / IQ3_S

In the same budget, a uniform allocation scores KLD 0.146 and the searched one 0.045 (in-sample). Starting from the uniform allocation, upgrading any single one of the middle layers to MXFP4 removes ~0.04 KLD. Their errors compound downstream rather than add up.

In the 128G build there is almost nothing left to place. Experts are IQ1_S everywhere except a few later layers at IQ1_M or IQ2_XXS.

The other parts are set as follows:

• Attention and shared experts are Q5_K–Q8_0 in 256G and Q4_K–Q6_K in 128G.
• The router, norms and hyper-connection mixers are left as shipped.
• The importance matrix comes from 614k tokens of English/Chinese chat and Wikipedia.

---

What's in this repo

Path	Size	Contents
Mix-256G/…-00001-of-00007.gguf … 00007	346.5 GB	recommended build; 00002 is the Engram table (110.6 GB)
Mix-128G/…-00001-of-00005.gguf … 00005	225.0 GB	128 GB build; 00002 is the Engram table (110.6 GB)
mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf	0.97 GB	vision projector for image input, lossless BF16
mmproj-DeepSeek-V4.1-Flash-1017-abliterated-F16.gguf	0.97 GB	the same in F16, for backends without BF16
imatrix.gguf	1.5 GB	the importance matrix both builds were quantized with

The vision projector works with either build. It was converted from the abliterated checkpoint, whose image entry point (aligner.w2) and image markers carry the refusal edit, so do not swap in the mmproj of the original model. The DeepSeek-V4.1 chat template is embedded in each build. The DSpark speculative drafter is not included.

---

Serving (llama.cpp)

DeepSeek-V4.1 is not in llama.cpp master yet. Build the PR that adds it, ggml-org/llama.cpp#30817. These files were made and tested at commit b72050d03:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git fetch origin pull/30817/head:pr-30817 && git checkout b72050d03
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli

hf download orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF \
  --include "Mix-256G/*" "mmproj-*-BF16.gguf" --local-dir ./dsv41-gguf


llama-server -m dsv41-gguf/Mix-256G/DeepSeek-V4.1-Flash-1017-abliterated-Mix-256G-00001-of-00007.gguf \
  --mmproj dsv41-gguf/mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf \
  -ngl 99 -c 32768 -np 1 --no-reasoning-preserve


llama-server -m dsv41-gguf/Mix-256G/DeepSeek-V4.1-Flash-1017-abliterated-Mix-256G-00001-of-00007.gguf \
  --mmproj dsv41-gguf/mmproj-DeepSeek-V4.1-Flash-1017-abliterated-BF16.gguf \
  -ngl 0 --no-repack --fit off -c 8192 -np 1 --no-reasoning-preserve

• Keep the files on an NVMe SSD and leave mmap on (the default). The Engram table is always kept off the GPU and read lazily. --mlock or --no-mmap would pull all 110 GB into RAM.
• CPU only: pass--no-repack. With the PR at b72050d03, the CPU backend's weight repacking leaves an operation no backend will take. Startup then aborts with GGML_ASSERT(*cur_backend_id != -1). --no-repack --fit off runs.
• Images: pass --mmproj and send OpenAI-style image_url content parts (base64 data URLs work).
• 256 GB of system RAM and one GPU: add --cpu-moe to keep the routed experts in RAM and run attention on the GPU. This setup was not measured here.
• --no-reasoning-preserve drops earlier turns' thinking from the prompt, which is what DeepSeek's reference encoder does. With it, llama.cpp's rendered prompts match encoding.py token for token on DeepSeek's test cases.
• Thinking is on by default. Pass "chat_template_kwargs": {"thinking_mode": "chat"} for a direct answer, or "reasoning_effort": "low" | "medium" | "high" | "max" (or 1–100) to set its depth. The reasoning comes back in message.reasoning_content.
• Set a system prompt. Without one, the model sometimes misnames itself in thinking mode.

Measured with short prompts, one request at a time:

Build	Hardware	Prompt	Generation
Mix-256G	2 x H200 (CUDA)	~110 tok/s	~70 tok/s
Mix-128G	1 x H200 (CUDA)	~110 tok/s	~84 tok/s
Mix-256G + mmproj	CPU only, 64 threads of a 192-core server	~52 tok/s	~10 tok/s

The CPU run shared the machine with an unrelated GPU training job, so it is a floor rather than a peak. Apple Metal and partial-offload (--cpu-moe) speeds were not measured.

Image input was checked on CPU with DeepSeek's own example images. Asked what each food is and which part is eaten, the model answered correctly:

• carrots: the taproot
• corn: the kernels on the cob

---

Model details

|
---|---
Base model | The abliterated build of deepseek-ai/DeepSeek-V4.1-Flash-1017
Architecture | deepseek41 in llama.cpp. 40 layers, hidden 5120, compressed sparse attention, LoRA-factorised output projection (8 groups), 384 routed experts top-6 + 1 shared expert, Engram n-gram lookup at layers 1 and 14, 4-wide Hyper-Connections
Weights in these files | 748.5B: 543.6B routed experts, 196.6B Engram, 8.2B attention / shared experts / embeddings / head. ~6 of 384 experts active per token
Quantization | Per-layer mixed precision found by measured-KLD search. Experts are IQ2_S–MXFP4 (256G) or IQ1_S–IQ2_XXS (128G). Dense parts are Q5_K–Q8_0 (256G) or Q4_K–Q6_K (128G). Engram is IQ4_NL. Uses an importance matrix.
Context | 1,048,576 tokens (inherited; evaluated at 2k) · Vocabulary 129,280
Runtime | llama.cpp with PR #30817; text and images in (with --mmproj), text out; tool calls in DeepSeek's DSML format

---

Bias, risks, and limitations

• Safety guardrails are substantially reduced (inherited from the abliterated base); see the disclaimer. The refusal and MMLU numbers above come from the unquantized checkpoint. Mix-128G in particular sits far enough from it (KLD 0.52) that its behaviour may differ.
• Mix-128G is heavily degraded. This is the limit of 1.6-bit post-training quantization on this model, not a tuning issue.
• The IQ4_NL Engram in the released files was not rescored. The table above was measured with MXFP8.
• Vision is only lightly checked (two example images, on CPU). The refusal edit also touches the image entry point, and no image-conditioned refusal or quality evaluation has been run. There is no DSpark drafter.
• Inherits the biases and limitations of DeepSeek-V4.1-Flash-1017. Long-context behaviour was not tested: all measurements are at 2k context or on short chats.
• Requires a llama.cpp build with PR #30817. Stock llama.cpp, Ollama and LM Studio will not load these files until it is merged.

License

MIT , inherited from the base model deepseek-ai/DeepSeek-V4.1-Flash-1017. The embedded chat template is unsloth's port of DeepSeek's encoder (unsloth/DeepSeek-V4.1-Flash-1017-GGUF, MIT).

---

One Gateway. Every Model.

Route Smarter · Ship Safer · Spend Less

Website · Models · GitHub · Discord · X