使用 NVIDIA Magpie TTS 构建低延迟多语言语音 Agent：开放权重与完整部署控制

每一次语音交互都有延迟预算。

当用户听到应用响应时，宝贵的毫秒数早已消耗在采集音频、转录语音、运行 LLM、检索上下文和生成回复上。文本转语音（TTS）是最后一步，也是用户感知最明显的一步。如果语音生成缓慢，整个体验都会显得迟滞。

你能够自行运行和调优的流水线环节越多，就越能收回更多延迟预算。

语音 AI 正在快速发展。集成式语音模型提供了简单性——一次 API 调用，音频输入、音频输出——但代价是难以针对你的领域微调每个组件、在更优模型发布时进行替换、落实数据驻留要求，以及准确了解延迟究竟来自何处。若要获得更强控制力，采用级联式架构——由专用 ASR、TTS 和 LLM 组件协同运行——可以使每一层都能独立调优，并部署在你拥有的基础设施上。

NVIDIA Magpie Multilingual TTS 正是为此而构建。借助开放权重、面向生产环境的 NVIDIA NIM以及对 12 种语言的支持，你可以在自有基础设施内部部署多语言语音能力，针对自身工作负载优化延迟，并为特定领域定制模型——从端到端全程都在自己的环境中完成。

最新版本新增现代标准阿拉伯语、韩语和巴西葡萄牙语，扩展了多语言覆盖范围；同时还通过更新训练数据和改进模型，提升了多种既有语言的质量。

无论你是在构建客户支持 Agent、医疗助手、企业 Copilot、翻译系统，还是对话式 AI 应用，Magpie 都为生产级语音 AI 提供了一个开放基础。

语音 AI 正在默认走向多语言

如今的语音应用并不只服务于单一语言。

全球客户支持、企业助手、医疗文档、零售自动化和翻译工作流，正日益需要跨多种语言进行自然对话——同时还必须保持低延迟。

支持更多语言只是挑战的一部分。开发者还需要能够：

• 在数据所在位置部署
• 满足企业隐私要求
• 自定义发音和音色
• 预测生产工作负载下的延迟
• 在自有基础设施上扩展

开放模型改变了上述每一项能力的可能性。

一个开放模型，十二种语言

Magpie TTS Multilingual 是一个拥有 3.64 亿参数的开放权重模型，支持：

英语 · 西班牙语 · 法语 · 德语 · 意大利语 · 越南语 · 普通话 · 印地语 · 日语 · 现代标准阿拉伯语（新增）· 韩语（新增）· 巴西葡萄牙语（新增）

每种语言均通过共享的多语言说话人表示，提供男性和女性说话人音色。

本次发布还扩展了对印地语和日语代码切换的支持，增强了多语言灵活性。该能力由 IPA 字素到音素处理以及自定义发音词典实现，可更轻松地准确读出姓名、技术术语和混合语言内容。

开发者无需为不同地区维护彼此独立的 TTS 模型，而是可以基于一个开放基础构建多语言应用。

用户真正能感知到的延迟

在对话式 AI 中，文本转语音是用户听到回复前的最后一个阶段。这使得首音频时间（Time to First Audio，TTFA）——即语音生成开始到首段音频抵达用户之间的延迟——成为语音流水线中最重要的延迟指标之一。

由于 Magpie TTS 可以部署在你自己的环境中，因此你测得的延迟就是你实际能够控制的服务器端延迟，数值中不包含托管服务往返通信的时间。

GPU	单流 TTFA	单流 RTFX	64 流 TTFA	64 流 RTFX
B200	32 ms	12.1×	239 ms	319.81×
H100	47 ms	14.7×	275 ms	290.79×
DGX Spark	53 ms	9.8×	962 ms	75.88×
A100	79 ms	12.2×	395 ms	197×

来源：NVIDIA TTS NIM Performance documentation（v26.07），本地部署环境下进行三次测试所得平均值。 TTFA = 首段音频延迟；RTFX = 以实时速度倍数表示的吞吐量。

在 B200 上，Magpie 的 TTFA 为 32 毫秒，为 ASR 和 LLM 处理保留了剩余的延迟预算——使端到端总延迟维持在自然对话所需的 200 毫秒以内。跨 NVIDIA GPU 平台，Magpie 在单流情况下可在 32–79 毫秒内生成首段音频。在 64 路并发流条件下，B200 的 TTFA 达到 239 毫秒，同时吞吐量达到实时速度的 320 倍——即使在并发负载下，生成音频的速度也超过播放速度的 300 倍。

上表展示的是通过 NVIDIA NIM 提供服务的 Magpie，并在本地部署环境中测得——即运行在你自有 GPU 上、经过优化的容器。开放的 Hugging Face checkpoint 是同一模型，可用于研究和微调；NIM 则是经过调优的推理服务栈，能够实现这些生产级延迟。二者均运行在你所控制的硬件上。

由于模型运行在自有基础设施中，你可以直接进行性能基准测试、针对自身部署进行调优，并根据工作负载扩展。对于实时语音 Agent 而言，这正是对话体验显得灵敏，还是显得迟缓的关键差别。

针对实时语音生成进行了优化

低延迟并非偶然。Magpie 引入了两项互补的架构改进，在保持语音质量的同时缩短推理时间。

帧堆叠。 解码器在每一个解码步骤中预测两个音频帧，而非一个。这会将解码器迭代次数减半，从而缩短生成时间并提升吞吐量。

局部 Transformer。 仅使用帧堆叠会因同时生成的码本 token 之间产生依赖关系而降低音频质量。局部 Transformer 对这些依赖关系进行建模并优化生成的音频，恢复了帧堆叠原本可能牺牲的质量。

这两项技术结合后，既能实现更快的生成速度，也能提供自然的语音合成效果。相关架构详见 Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation（ICASSP 2026）。

如果听起来不自然，速度再快也没有意义

此次发布不仅新增了语言，还改善了许多既有语言的合成质量。与上一版本相比，Magpie 在多种语言上表现出更低的字符错误率（CER）和更高的说话人相似度（SSIM），其中法语和西班牙语的提升最为明显：

语言	CER（上一版本）	CER（本次发布）	SSIM（上一版本）	SSIM（本次发布）
法语	2.70%	1.54%	0.703	0.747
西班牙语	1.14%	0.60%	0.715	0.793
德语	0.66%	0.80%	0.626	0.742

来源：Magpie TTS Multilingual model card。CER 越低越好；SSIM 越高越好。

新加入的阿拉伯语（CER 为 1.62%）、韩语（2.69%）和巴西葡萄牙语（2.91%）模型，为未来质量改进建立了基准。

尽管客观指标有助于衡量进展，语音质量最终仍取决于感知体验。你可以在 NVIDIA Build 或 Hugging Face 演示中亲自聆听差异。

为什么开放权重如此重要

可测量的延迟很有用。可控制的延迟则更好。

开放权重赋予开发者源自部署所有权的能力。使用 Magpie，你可以：

• 部署在你所控制的基础设施上 —— 可完全在自有基础设施内运行，包括私有或物理隔离环境。
• 掌控自己的延迟预算 —— 无需承担托管服务往返通信时间，并可直接针对自身硬件和工作负载进行优化。
• 自定义发音和音色 —— 使用 NeMo 为自己的品牌、领域词汇或说话人数据进行微调。
• 按自己的方式扩展 —— 针对你的基础设施和工作负载优化推理服务栈。
• 保持企业级控制权 —— 将敏感对话和客户数据保留在自己的环境中。

对于构建生产级语音 AI 的企业而言，对部署、性能和定制化的这种控制力往往最为重要。

构建完整语音 Agent，而不仅是更好的语音

生产环境中的语音 AI 是由一组模型构成的系统，而不是单一模型。Magpie TTS 是 NVIDIA Nemotron Voice Agent Developer Example 的一部分。该参考实现展示了专用语音、语言和推理模型如何作为协调系统协同工作，使你能够构建始终在线的语音 Agent，而不只是让语音听起来更自然。

开发者可以组合：

• 用于流式语音识别的 Nemotron Speech
• 用于自然多语言语音合成的 Magpie TTS
• 用于推理、工具调用和多模态理解的 Nemotron 语言与多模态模型
• 用于 GPU 优化、生产就绪推理微服务的 NVIDIA NIM
• 用于定制和微调的 NeMo

Nemotron Voice Agent 开发者示例提供了一个端到端参考实现，开发者可在数小时内克隆、定制并部署。它包含以下生产模式：

• 实时可中断（插话，barge-in）对话
• 具备视觉理解能力的多模态语音 Agent
• 多 Agent 编排和工具调用
• 多语言语音交互
• 使用 NVIDIA NIM 实现亚秒级端到端延迟

开发者无需从零开始组装各个组件，而是可以从完整的参考架构起步，并将其适配至自己的应用。

开始使用

试用模型

部署到生产环境

针对你的领域进行定制

构建完整语音 Agent

开放权重与许可证

推荐推理配置：

cfg_scale = 2.5
temperature = 0.6 top_k = 80 apply_attention_prior = True prior_epsilon = 0.1

Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS

Every voice interaction has a latency budget.

By the time a user hears your application respond, you've already spent precious milliseconds capturing audio, transcribing speech, running an LLM, retrieving context, and generating a response. Text-to-speech (TTS) is the final step — and the one users notice most. If speech generation is slow, the whole experience feels slow.

The more of that pipeline you can run and tune yourself, the more of the latency budget you get back.

Voice AI is moving fast. Integrated speech models offer simplicity — one API call, audio in, audio out — but they trade the ability to fine-tune each component for your domain, swap in better models as they ship, enforce data residency, and understand exactly where latency is coming from. For more control, a cascaded architecture — purpose-built ASR, TTS, and LLM components running together — keeps each layer independently tunable and deployable on infrastructure you own.

NVIDIA Magpie Multilingual TTS is built for that. With open weights, production-ready NVIDIA NIM, and support for 12 languages, you can deploy multilingual speech inside your own infrastructure, optimize latency for your workload, and customize the model for your domain — end to end, in your own environment.

The latest release expands multilingual coverage with Modern Standard Arabic, Korean, and Brazilian Portuguese, while improving quality across many existing languages through updated training data and model improvements.

Whether you're building customer support agents, healthcare assistants, enterprise copilots, translation systems, or conversational AI applications, Magpie provides an open foundation for production voice AI.

Voice AI Is Becoming Multilingual by Default

Today's voice applications don't serve a single language.

Global customer support, enterprise assistants, healthcare documentation, retail automation, and translation workflows increasingly require natural conversations across multiple languages — all while maintaining low latency.

Supporting more languages is only part of the challenge. Developers also need the ability to:

• Deploy where their data lives
• Meet enterprise privacy requirements
• Customize pronunciation and voices
• Predict latency under production workloads
• Scale on their own infrastructure

Open models change what's possible on every one of these.

One Open Model, Twelve Languages

Magpie TTS Multilingual is a 364M-parameter open-weights model supporting:

English · Spanish · French · German · Italian · Vietnamese · Mandarin · Hindi · Japanese · Modern Standard Arabic (new) · Korean (new) · Brazilian Portuguese (new)

Each language includes male and female speaker voices through a shared multilingual speaker representation.

This release also improves multilingual flexibility with expanded code-switching support for Hindi and Japanese, enabled through IPA grapheme-to-phoneme processing and custom pronunciation dictionaries — making it easier to accurately pronounce names, technical terminology, and mixed-language content.

Instead of maintaining separate TTS models for different regions, developers can build multilingual applications on a single open foundation.

The Latency Your Users Actually Notice

In conversational AI, text-to-speech is the final stage before users hear a response. That makes Time to First Audio (TTFA) — the delay between speech generation beginning and the first audio reaching the user — one of the most important latency metrics in a voice pipeline.

Because Magpie TTS can be deployed inside your own environment, the latency you measure is the server-side latency you actually control, with no managed-service round-trip in the number.

GPU	1-stream TTFA	1-stream RTFX	64-stream TTFA	64-stream RTFX
B200	32 ms	12.1×	239 ms	319.81×
H100	47ms	14.7×	275 ms	290.79×
DGX Spark	53 ms	9.8×	962 ms	75.88×
A100	79 ms	12.2×	395 ms	197×

Source:NVIDIA TTS NIM Performance documentation (v26.07), average of three trials, on-prem.
TTFA = latency to first audio; RTFX = throughput as a multiple of real time.

At 32ms on B200, Magpie's TTFA leaves the rest of the latency budget for ASR and LLM processing — keeping total end-to-end latency within the sub-200ms window natural conversation requires. Across NVIDIA GPUs, Magpie delivers first audio in 32–79ms on a single stream. At 64 concurrent streams, B200 reaches 239ms TTFA while delivering throughput at 320× real time — generating audio more than 300 times faster than it plays back, even under concurrent load.

The table above shows Magpie served as the NVIDIA NIM, measured on-prem — the optimized container running on your own GPU. The open Hugging Face checkpoint is the same model and your path for research and fine-tuning; the NIM is the tuned serving stack that produces these production latencies. Both run on hardware you control.

Because the model runs on your own infrastructure, you can benchmark performance directly, tune it for your deployment, and scale according to your workload. For real-time voice agents, that's the difference between conversations that feel responsive and conversations that feel delayed.

Optimized for Real-Time Speech Generation

Low latency isn't accidental. Magpie introduces two complementary architectural improvements that reduce inference time while maintaining speech quality.

Frame stacking. The decoder predicts two audio frames during each decoding step rather than one. This cuts the number of decoder iterations in half, shortening generation time and improving throughput.

Local transformer. Frame stacking alone would reduce audio quality by introducing dependencies between simultaneously generated codebook tokens. The local transformer models those dependencies and refines the generated audio, recovering the quality that frame stacking would otherwise sacrifice.

Together, these techniques deliver both faster generation and natural speech synthesis. The architecture is described in Frame-Stacked Local Transformers for Efficient Multi-Codebook Speech Generation (ICASSP 2026).

Faster Doesn't Matter If It Doesn't Sound Natural

This release doesn't only add languages — it also improves synthesis quality across many existing ones. Compared to the previous release, Magpie shows reduced character error rates (CER) and higher speaker similarity (SSIM) on several languages, with the clearest gains on French and Spanish:

Language	CER (prev)	CER (this release)	SSIM (prev)	SSIM (this release)
French	2.70%	1.54%	0.703	0.747
Spanish	1.14%	0.60%	0.715	0.793
German	0.66%	0.80%	0.626	0.742

Source:Magpie TTS Multilingual model card. CER lower is better; SSIM higher is better.

The newly added Arabic (1.62% CER), Korean (2.69%), and Brazilian Portuguese (2.91%) models establish baseline quality for future improvements.

While objective metrics help measure progress, speech quality is ultimately perceptual. You can hear the difference yourself on NVIDIA Build or the Hugging Face demo.

Why Open Weights Matter

Latency you can measure is useful. Latency you can control is even better.

Open weights give developers capabilities that come from owning the deployment. With Magpie you can:

• Deploy on infrastructure you control — run entirely within your own infrastructure, including private or air-gapped environments.
• Own your latency budget — no managed-service round-trip, and you optimize directly for your hardware and workload.
• Customize pronunciation and voices — fine-tune with NeMo for your own brand, domain vocabulary, or speaker data.
• Scale on your own terms — optimize the serving stack for your infrastructure and workload.
• Maintain enterprise control — keep sensitive conversations and customer data inside your environment.

For enterprises building production voice AI, this control over deployment, performance, and customization is often what matters most.

Build Complete Voice Agents — Not Just Better Speech

Voice AI in production is a system of models, not a single one. Magpie TTS is part of the NVIDIA Nemotron Voice Agent Developer Example, a reference implementation showing how purpose-built speech, language, and reasoning models work together as a coordinated system — so you can build always-on voice agents, not just better-sounding speech.

Developers can combine:

• Nemotron Speech for streaming speech recognition
• Magpie TTS for natural multilingual speech synthesis
• Nemotron language and multimodal models for reasoning, tool calling, and multimodal understanding
• NVIDIA NIM for GPU-optimized, production-ready inference microservices
• NeMo for customization and fine-tuning

The Nemotron Voice Agent developer example provides an end-to-end reference implementation that developers can clone, customize, and deploy in hours. It includes production patterns for:

• Real-time interruptible (barge-in) conversations
• Multimodal voice agents with vision understanding
• Multi-agent orchestration and tool calling
• Multilingual voice interactions
• Sub-second end-to-end latency using NVIDIA NIM

Rather than assembling individual components from scratch, developers can start from a complete reference architecture and adapt it to their own applications.

Get Started

Try the model

Deploy to production

Customize for your domain

Build complete voice agents

Open weights and license

Recommended inference configuration:

cfg_scale = 2.5          
temperature = 0.6
top_k = 80
apply_attention_prior = True
prior_epsilon = 0.1