为软件工程团队自托管 DeepSeek V4
英文标题：🔶 Self-hosting DeepSeek V4 for a software engineering org
Parity 工程团队以 20% 工时启动了开放权重模型与自管理推理试验，并通过 LiteLLM、vLLM 和 RunPod GPU 为开发者工具提供统一模型接口。原计划仅持续半天的 DeepSeek V4 Flash 试用最终运行近一个月，25 名工程师处理了超过 14.4 万次请求、约 129 亿个 tokens，GPU 计算成本约为 1200 美元。调查中，8 名受访者里有 7 人认为该模型可以作为主力模型，但多数人认为其表现只是勉强够用。Parity 的结论并非要替代托管 AI 服务，而是建设混合式 AI 基础设施，以获得对成本、可用性、模型选择、数据和未来能力的更多控制，并针对高可靠性软件开发建立自己的评测与验证体系。
5.4 KB
501 B
203.7 KB
552.3 KB
574.1 KB
497.4 KB
5.4 KB
430 B
574 B
976 B
中文
过去几个月里，Parity 的工程师一直在测试开放权重 AI 模型和自管理推理，以了解我们应当在多大程度上自主掌控 AI 工程技术栈。以下是第一个月的收获。
简而言之，我们做了什么
- 一项由 Parity 工程团队以 20% 工时推进的计划，探索开放权重模型和自托管推理。
- 对 DeepSeek V4 Flash 的一次试用原计划只运行一个下午，后来持续了三天，随后仍在继续：25 名工程师在一个月内使用了约 129 亿个 tokens，GPU 计算成本约为 1200 美元。
- 在回应调查的 8 名工程师中，有 7 人表示该模型可以作为他们的主力模型，尽管多数人认为它只是勉强够用。
- 目标不是替代托管服务商，而是确保我们拥有选择权。
AI 编程助手融入 Parity 日常工程工作的速度，快于我们的预期。Codex、Cursor、Claude、OpenCode、Pi 等工具如今已在工程组织的大部分团队中使用。随着采用率提高，我们开始提出一个更根本的基础设施问题：如果这些系统正成为我们构建软件的方式之一，那么其中多少能力应完全依赖外部服务商？
过去几个月中，少数 Parity 工程师一直在探索另一种方案。Parity AI 是一项以 20% 工时开展的工程计划，研究开放权重模型、自托管推理，以及通过工程师既有工具提供不同模型所需的基础设施。
目标并不是取代每一项托管 AI 服务，而是了解在哪些场景下，掌控更多技术栈能让我们获得对成本、可用性、模型选择、数据以及未来能力的有效控制。
我们的第一次实验给出了一些有价值的答案。我们提供了一款低成本开放权重模型；即使原定试用期结束后，工程师仍持续使用它。一场原本只计划持续半天的测试，最终变成了一个月的真实使用。
让 AI 模型可替换，而无需改变开发者工具
随着生成式 AI 模型持续快速迭代，每次我们想尝试不同模型时，工程师都不应被迫改变工作方式。因此，我们当前的架构有意保持简单。LiteLLM 在开发者工具与后端模型之间提供统一接口。vLLM 负责模型服务。目前的 GPU 容量则从 RunPod 租用。
从技术上说，这是运行在第三方 GPU 上的自管理推理，而非使用内部硬件。工程师的代码和提示词仍会离开我们的基础设施，但数据保留和日志记录行为现在由我们控制，而不是由供应商控制。
当前的 Parity AI 架构通过统一接口路由请求，使不同模型和推理服务商能够位于同一套开发者工具之后。
工程师可以继续使用熟悉的客户端和编码环境，而处理请求的模型则可在统一端点背后切换。我们不再将开发者工作流绑定于单一 AI 服务商，而是能够测试开放权重模型和托管模型，依据自身需求比较它们，并逐步在外部基础设施和内部可控基础设施之间迁移工作负载。
模型成为工程技术栈中又一个可评估、可替换的组成部分。这正是此次实验的初衷。
从半天试用到近 130 亿个 tokens
8 月中旬，我们使用开放权重模型 DeepSeek V4 Flash 进行了短期试用。它原计划只运行一个下午。随着使用量持续增长，我们让它运行了三天。之后工程师仍在继续使用它。8 月 16 日至 9 月 12 日期间，系统处理了来自 25 名工程师的超过 14.4 万次请求，以及近 129 亿个 tokens。
前三天约占其中 32 亿个 tokens，GPU 计算成本约为 300 美元，即每百万 tokens 约 0.10 美元。这个数字仅涵盖 GPU 计算，不包括工程时间——而后者显然是更高的成本：这是一个由少数工程师投入数周、按 20% 工时推进的项目。
我们的判断是，基础设施、基准测试套件和运维知识的投入，会在接下来的 N 个工作负载中得到回报；而如果我们从不建设这些能力，就永远无法验证这一点。
AI 推理数据揭示了什么
我们还了解了工程师实际产生的工作负载。上下文长度的分布是这里值得关注的形态：
- 99% 的请求所需上下文少于 50 万 tokens
- 55% 的时间处于空闲状态，26% 的时间用于服务 1 个请求，9% 的时间用于服务 2 个并行请求，4% 的时间用于服务 3 个并行请求
- 1 个并行请求的平均速度为 345 tokens/秒
- 2 个并行请求的平均速度为 251 tokens/秒
- 3 个并行请求的平均速度为 209 tokens/秒
- 4 个并行请求的平均速度为 152 tokens/秒
- 在峰值时，prefill 速度最高达到 16.8 万 tokens/秒
- 平均首 token 时间：3.34 秒
- 平均排队时间：2.25 秒
- 单请求平均解码速率：182 tokens/秒
- 提示词 tokens：7.70 亿个 tokens
- 补全 tokens：555 万个 tokens
- GPU KV cache 峰值使用率：9.4%
- 生成 tokens 中位数（即每个请求生成的 token 数量）：349 个 tokens
- p99 首 token 时间：70 秒（也就是一些人报告过的那些卡顿）
上下文长度本身并不是基础设施约束：KV cache 内存、并发量和吞吐量更为重要。不过，它仍是一个有用的规划信号。实际情况是，我们的工程师并未触及当前模型宣传的上下文窗口上限。
仅凭使用量无法判断模型是否足够胜任日常工作，因此我们还询问了使用者。共有 25 人使用该系统，其中 8 人回应了调查。8 人中有 7 人认为该模型能够作为他们的主力模型，但多数人认为它只是勉强够用。
无论调查结果如何，这些数据本身都表明，该模型已经足够有用，能让工程师无需推动便持续使用。
Parity 为何采用混合式 AI 基础设施路线
托管 AI 和自托管 AI 并不是互相对立的选择。一些模型适合通过外部 API 使用，另一些则更适合直接运行。正确方案取决于具体工作负载，因此我们正在构建混合式架构。
统一的推理层使我们能够决定每项工作负载应在哪里运行，而无需围绕它重建工具。随着时间推移，我们希望自主作出更多这类决策。
我们正在探索专用推理硬件，使模型能够直接运行在 Parity 控制的基础设施上。我们也在基于自身工程需求开发基准测试，而不是完全依赖通用模型排行榜。工程组织的需求与通用 AI 基准测试截然不同。代码库、编程语言、长上下文、Agent 工作流以及内部工程规范，都会影响一个模型是否真正有用。
闲置 GPU 容量或许比成本更重要
成本是这次实验的一个结果，但容量可能更为重要。
试用期间，GPU 约有 55% 的时间处于空闲状态。根据你如何利用这些富余资源，这既可以成为反对专用硬件的理由，也可以成为支持它的理由。更好的请求打包和持续批处理，可以将每个 token 的实际成本降至 0.10 美元这一表面数字以下。但未使用的容量也暗示了另一种可能性：与其将推理视为一种一次只能消费一个提示词的稀缺资源，不如利用空闲算力运行持续性或机会性工作负载。
这可以包括更深入了解我们代码库和工程规范的助手、组织专属检索，以及在代码变更合入时自动进行分析。这些工作负载在每个 token 都作为独立外部 API 调用计费时很难证明合理性，而在硬件已经持续运行时则更容易成立。
AI 基础设施正在成为一项工程决策
随着编程助手成为日常工程基础设施的一部分，其架构至关重要。我们使用哪些模型、它们在哪里运行、如何轻松替换它们、它们可以访问哪些信息，以及组织自行运行推理后能够实现什么，都是工程决策。
我们还没有最终答案。但在经历一个月和 130 亿个 tokens 后，我们已经知道了一个更好的研究方式：构建、测量，并具体说明哪些有效、哪些无效。
Parity 接下来测试什么
我们正在测试更多模型，围绕自身工作负载构建基准测试，扩大内部访问范围，并研究专用硬件。随着这些基准测试逐步成熟，我们计划公开方法论和结果，而不是将其留在内部。
更根本的问题是：当生成代码几乎免费时，一个工程组织会发生什么？我们认为，诚实的答案是，目前还没有人知道。但对于 Parity 这样的公司，答案不能只是“更多代码”。我们编写的是共识、密码学和区块链基础设施，其中一个 bug 不是一张工单，而是一次事故。在这种环境下，产出不是交付物，正确性才是。
无限的生成能力不会改变我们构建什么，但会改变压力落在哪里：落在评审、测试、CI，以及我们能否有把握地说明某项变更为何安全的能力上。甚至可以说，验证问题扩张得比生成问题更快，因为生成的每一行代码都需要有人为其辩护。对 Parity 而言，挑战在于快速但安全地端到端闭合这一循环。这要求我们掌控足够多的技术栈，以便根据自身对正确性的定义测试模型，同时也要知道何时根本不该使用它们。
English / 原文
Over the past few months, Parity engineers have been testing open-weight AI models and self-managed inference to understand how much of our AI engineering stack we should control ourselves. Here's what we learned from the first month.
What we did, in short
- A 20%-time Parity engineering initiative exploring open-weight models and self-hosted inference.
- A trial on DeepSeek V4 Flash meant to run for one afternoon. It ran for three days, then kept going: 25 engineers, ~12.9 billion tokens, ~$1200 in GPU compute over a month.
- Seven of the eight engineers who responded to our survey said the model could serve as their primary model, although most felt it was only just good enough.Â
- The aim is not to replace hosted providers but to make sure we can choose.
AI coding assistants became part of everyday engineering at Parity faster than any of us expected. Codex, Cursor, Claude, OpenCode, Pi, and others are now used across much of the engineering organization. As adoption increased, we started asking a more fundamental infrastructure question: if these systems are becoming part of how we build software, how much of that capability should depend entirely on external providers?
Over the past few months, a small group of Parity engineers has been exploring an alternative. Parity AI is a 20%-time engineering initiative exploring open-weight models, self-hosted inference, and the infrastructure needed to make different models available through the tools engineers already use.
The aim isn't to replace every hosted AI service, but to understand where owning more of the stack gives us useful control over cost, availability, model choice, data, and future capabilities.
Our first experiment gave us some useful answers. We made a cheap open-weight model available; engineers kept using it after the original trial was supposed to end. What began as an afternoon test became a month of real-world use.Â
Making AI models replaceable without changing developer tools
As generative AI models continue to change quickly, engineers shouldn't have to change how they work every time we want to try a different model. Our current architecture is therefore deliberately simple. LiteLLM provides a common interface between developer tools and the models behind them. vLLM serves the models. GPU capacity is currently rented from RunPod.
Technically, this is self-managed inference on third-party GPUs rather than in-house hardware. Engineers' code and prompts still leave our infrastructure, but we now control retention and logging behavior rather than a vendor.
The current Parity AI architecture routes requests through a common interface, allowing different models and inference providers to sit behind the same developer tools.
Engineers can continue using familiar clients and coding environments while the model serving the request changes behind a common endpoint. Instead of coupling a developer workflow to one AI provider, we can test both open-weight and hosted models, compare them against our own requirements, and progressively move workloads between external and internally controlled infrastructure.
The model becomes another part of the engineering stack we can evaluate and replace. This was the original purpose of the exercise.
From an afternoon trial to nearly 13 billion tokens
In mid-August, we ran a short trial using DeepSeek V4 Flash, an open-weight model. It was supposed to run for an afternoon. Usage kept growing, so we left it running for three days. Then engineers kept using it. Between 16 August and 12 September, the system handled more than 144,000 requests and almost 12.9 billion tokens from 25 engineers.
The first three days account for ~3.2 billion of those tokens and roughly $300 in GPU compute, about $0.10 per million tokens. That figure covers GPU compute only. It excludes engineering time, which is easily the higher cost: this was a 20%-time project running for a couple of weeks across a handful of engineers.
The bet is that the infrastructure, benchmark suite, and operational knowledge pay back over the next N workloads, and that if we never build them, we never find out.
What the AI inference data showed
We also learned something about the workloads our engineers actually generate. The distribution of context lengths is the useful shape here:
- 99% of requests needed less than 500k of context
- 55% of total time was spent idle, 26% of time was spent serving 1 request, 9% of time was spent serving 2 parallel requests, 4% of time was spent serving 3 parallel requests
- 1 parallel request gets 345 tokens/sec on average
- 2 parallel requests get 251 tokens/sec on average
- 3 parallel requests get 209 tokens/sec on average
- 4 parallel requests get 152 tokens/sec on average
- at its peak, prefill went as fast as 168k tokens/sec
- mean time to first token: 3.34s
- mean queue time: 2.25s
- mean per-request decode rate: 182 tokens/sec
- prompt tokens: 770M tokens
- completion tokens: 5.55M tokens
- peak GPU KV cache usage: 9.4%
- median generation tokens (how many tokens were generated for a request): 349 tokens
- p99 time to first token: 70 seconds (those hiccups some people were reporting)
Context length is not, on its own, an infrastructure constraint: KV-cache memory, concurrency, and throughput matter more. But it is a useful planning signal. In practice, our engineers are not pushing against the advertised context window of current models.
Usage alone doesn't tell you whether a model is good enough for everyday work, so we also asked the engineers who used it. Twenty-five people used the system and eight responded to our survey. Seven of those eight considered the model capable of serving as their primary model, though most felt it was only just good enough.
What the numbers do show, independent of the survey, is that the model was useful enough for engineers to keep using without prompting.
Why Parity is taking a hybrid approach to AI infrastructure
Hosted and self-hosted AI are not opposing choices. Some models make sense to consume through external APIs, and others are better run directly. The right approach depends on the workload, so we're building a hybrid architecture.
A common inference layer lets us decide where each workload should run without rebuilding the tools around it. Over time, we want to make more of those decisions ourselves.
We're exploring dedicated inference hardware that would allow models to run directly on infrastructure controlled by Parity. We're also developing benchmarks based on our own engineering requirements rather than relying entirely on generic model leaderboards. An engineering organization has very different requirements from a general AI benchmark. Codebases, languages, long contexts, agentic workflows, and internal engineering conventions influence whether a model is really useful.
Why spare GPU capacity may matter more than cost
Cost was one result of the experiment, but capacity may matter more.
During the trial, the GPUs were idle roughly 55% of the time. That is one argument against dedicated hardware and one argument for it, depending on what you do with the slack. Better request packing and continuous batching would push effective cost per token lower than the $0.10 headline. But unused capacity also suggests another possibility: instead of treating inference as a scarce resource consumed one prompt at a time, spare compute can run continuous or opportunistic workloads.
That could include assistants with deeper knowledge of our codebases and conventions, organization-specific retrieval, and automated analysis of changes as they land. Workloads that are hard to justify when every token is billed as a separate external API call, and easy to justify when the hardware is already running.
AI infrastructure is becoming an engineering decision
As coding assistants become part of everyday engineering infrastructure, their architecture matters. Which models we use, where they run, how easily we can replace them, what information they can access, and what becomes possible when an organization runs its own inference are engineering decisions.
We don't have final answers. After a month and thirteen billion tokens, however, we know a better way to investigate them: build, measure, and be specific about what worked and what didn't.
What Parity is testing next
We're testing more models, building benchmarks around our own workloads, expanding access internally, and investigating dedicated hardware. As those benchmarks mature, we intend to publish the methodology and results rather than keep them in-house.
The more fundamental question is what happens to an engineering organization when generating code is nearly free. We think the honest answer is that nobody knows yet. But for companies like Parity, the answer cannot simply be "more code." We write consensus, cryptographic, and blockchain infrastructure where a bug is not a ticket but an incident. In this environment, output is not the deliverable. Correctness is.
Unlimited generation does not change what we build, but it changes where the pressure lands: on review, on testing, on CI, on our ability to say with confidence why a change is safe. If anything, the verification problem scales faster than the generation problem because every line produced is a line someone has to defend. For Parity, the challenge is closing that loop end to end, quickly but safely. That requires owning enough of the stack to test models against our own definition of correctness, while also knowing when not to use them at all.