Voxlocal：用 Rust 编写的极简语音 Agent

“ man muss immer umkehren” —— Carl Gustav Jacob Jacobi

“如有疑问，就做减法”

我已经与语音 Agent 打交道有一段时间了；它们是一项迷人的技术。与语音 Agent 对话并完成工作，感觉如同魔法一般。你是否想过语音 Agent 在幕后是如何运作的？这篇博客文章是我试图理解其内部机制的一次尝试。

语音 Agent 究竟是什么？

AI Agent 是一种自主软件系统，它利用 AI 模型进行推理、规划步骤并执行任务，以达成特定目标。语音 Agent 则是其一种形态——能够以自然、类人的口语对话来完成任务的系统。

简单来说，语音 Agent 会倾听、思考、执行任务并说话。

鸟瞰图

这张图的信息量可真不少。因为我很好奇这一切如何运作，所以我用 Rust 创建了一个基础项目，以理解幕后发生的细节。

在深入语音 Agent 部分之前，先把传统组件讲清楚。

电信服务商

电信网络是一个复杂的话题，远超本文范围。在这里，电信服务商这一环节负责连接传统公共电话网络（PSTN/SIP）与基于云的 AI 应用。它承担了呼叫生命周期管理、双向流式传输、转码以及保持低延迟等复杂任务。

这一环节的一项重要工作，是为丢失或延迟的数据包进行抖动缓冲（jitter buffering），以及进行音频上采样以保持 STT 准确率。该环节会努力将网络延迟控制在阈值以下。

现代电信是工程学的奇迹，也与现代互联网紧密相关。

说完这些，下面来看看我们要做的东西。

介绍 voxlocal：我们的前卫语音 Agent

Voxlocal 是一款面向 macOS 的完全本地、低延迟语音 Agent。它是一个用 Rust 编写的 CLI 工具，用户可以与语音 Agent 对话并获得回复。出于实验精神，我使用了最基础的组件和微型模型，尽可能接近这种魔法的本质。请不要将它用于生产环境！

它的关注范围非常狭窄：一个使用英语的汽车服务场景。我还使用了 Whisper 和 Piper 等现成组件，而不是重复造轮子。为了让整个管线可检查且透明，voxlocal 有意避免使用复杂的异步流式处理或 VAD（语音活动检测）。我可能会在后续版本中进一步研究这些内容。

探索源代码：

开源项目 · Rust · macOS samkhawase / voxlocal 一个极简、完全本地的语音 Agent，旨在让对话中的每一步都易于检查。查看仓库 ↗

让我们逐步探索它。

1. 当你对语音 Agent 说话时，会发生什么？

当你与语音 Agent 对话，或向它提供音频样本时，它会将任务拆分为多个更小的工作单元。它先从音频中识别单词，并通过删除“Um”“ah”等词清理文本。接着，它利用这些词查找相关信息，并据此选择行动。最后，它再将回复转换回语音。

在 voxlocal 中，首先会捕获麦克风音频，并将其传给语音识别模块：

let capture = record_until(RECORD_SECS)?;
let transcript = stt.transcribe(&capture.samples)?;

随后，转录文本进入共享的轮次处理管线。在该管线中，Agent 会规范化文本、检索匹配的上下文、将请求路由至相关工具，最后准备回复：

run_turn(
    &norm, &rag, &embedder, &mut llm, tts.as_mut(),
    transcript, &mut budget,
)?;

各个阶段使用不同技术来完成操作。

• 语音识别从音频中确定单词。
• 检索使用数学方法1（余弦相似度）比较数值化后的文本块。
• LLM 预测应采取的动作。
• 传统 Rust 代码负责调用工具并执行动作。
• 语音合成根据最终回复生成音频。

不过，这样说太过枯燥和学术化了。让我们通过一个实际案例拆开各层：用户向 Agent 询问：“换一次机油要多少钱？”

2. 从声波到文字

让我感到惊讶的是，麦克风通过持续测量空气压力来捕获声音，并将这些读数转变为音频样本。voxlocal 随后将捕获的音频转换为单声道2、每秒 16,000 个采样点，再将其交给 Whisper 进一步处理。

Whisper 是我们训练好的神经网络，它被设计用于寻找音频中类似语音音素的模式，然后估计这些声音是否构成一个词序列。Whisper 的创建者已针对这项任务完成训练和微调；我们的程序只需借助它并调用模型3。

let mut params =
    whisper_rs::FullParams::new(
        whisper_rs::SamplingStrategy::Greedy { best_of: 1 }
    );
params.set_language(Some("en"));
params.set_temperature(0.0);
state.full(params, audio)
    .map_err(|e| anyhow!("whisper decode failed: {e}"))?;

在本例中，我们使用贪婪解码而非采样，以选取最可能的下一个 token。温度设为 0，意味着它只会返回概率最高的第一个匹配项，例如 oil 或 change。之所以能够这样做，是因为我们的上下文非常有限，而且我们知道对话内容。

从 Whisper 获得分段结果之后，voxlocal 的任务便是将它们拼接成一份完整转录文本。

let mut out = String::new();
for i in 0..state.full_n_segments() {
    if let Some(segment) = state.get_segment(i) {
        if let Ok(text) = segment.to_str_lossy() {
            out.push_str(&text);
        }
    }
}
let transcript = out.trim().to_string();

在我们的案例中，输出为：

“What does an oil change cost?”

在这一阶段，输出是根据输入信号作出的估计，可能与实际说出的词略有差异。受噪音、口音、语速节奏、音色等多种因素影响，模拟声音到数字文本的转录始终很棘手。因此，我们需要额外阶段来清理文本，使其在后续处理中更一致。

3. 清理识别出的语音

人类说话并不总是一致的，常常会加入“Um”等填充词、尴尬的停顿，或对数字进行特殊发音（例如将 “Nine Thirty” 念成 “noyynTherTee”）。在将文本送入后续阶段前，voxlocal 会应用简单且可预测的规则对文本进行清理。

例如，voxlocal 中的规范化器会移除常见填充短语、修剪空白字符并合并分隔符：

s = self.thousands.replace_all(&s, "$1$2").into_owned();
s = self.fillers.replace_all(&s, " ").into_owned();
s = self.space.replace_all(&s, " ").into_owned();

当我们说“Um yeah, what does an oil change cost?”时，它可能会变成“What does an oil change cost”。程序只是使用既有、已知的转换规则，并让后续阶段解释背后的含义。它本质上只是一些会在每轮对话中复用的正则表达式，这也符合我们采用基础实现的思路。

4. 将文字转换为向量

现在 voxlocal 已经获得清理后的文本，接下来需要从中识别用户意图。用户可能会说“What does an oil change cost”，也可能说“How much for an oil service”，但两者具有相同意图。那么，我们如何识别这种意图？

答案是使用嵌入模型，它会把文本转换为一串数字（如果想听起来更酷，你可以称它为 vector，即向量）。借助这一方法，语义相关的文本往往会得到指向相近方向的向量。

我们在 voxlocal 中使用 MiniLM，为规范化后的查询生成一个向量。

pub fn embed_one(&self, text: &str) -> Result<Vec<f32>> {
    Ok(self.embed_batch(&[text])?.remove(0))
}

在函数 embed_batch 内部，MiniLM 会为每个 token 生成一个表示。随后，代码通过掩码均值池化（masked mean pooling）将这些 token 表示合并为一个句子向量。简单来说：它对真实 token 取平均值，并忽略为了让批次长度一致而添加的填充内容。

let mask = attention.to_dtype(DType::F32)?.unsqueeze(2)?;
let summed = hidden.broadcast_mul(&mask)?.sum(1)?;
let counts = mask.sum(1)?;
let pooled = summed.broadcast_div(&counts)?.contiguous()?;

模型如何判断哪些模式对语义和意图重要？答案在于模型学习到的权重；这些权重经过预训练，可以直接供 voxlocal 使用。voxlocal 的池化代码则将 token 级输出归约为完整句子的单一向量。这完全是模型输出的表示；Rust 代码并未定义每个坐标的具体含义。

最后，向量会进行 L2 归一化。

let norm = row.iter().map(|v| v * v).sum::<f32>().sqrt();
let unit_vector = row.iter().map(|v| v / norm).collect::<Vec<_>>();

简单来说，归一化确保每个非零向量的长度都是 1。这是下一阶段所必需的，因为下一阶段会用简单的点积比较方向。根据数学原理，即便措辞不同，含义相近的文本也会指向相似方向。如果你好奇，是的，它确实会从字面和几何意义上计算角度的余弦值。我仿佛能看见我的高中数学老师 Bhalérao 先生，正带着得意的微笑看着我。

5. 查找相关知识

现在，查询已经成为单位长度向量，voxlocal 可以将它与现有服务文档进行比较。voxlocal 将查询向量与每个文档向量进行对比，并找出有用的上下文。

系统会计算并比较向量的点积，也就是余弦相似度。匹配度越高，意味着向量指向的方向越相似。搜索会比较各项分数，并丢弃低于参考阈值的结果；剩余的匹配结果会被排序，并返回请求数量。

let mut scored: Vec<(&Doc, f32)> = self.docs
    .iter()
    .map(|doc| {
        let sim: f32 = doc.embedding.iter()
            .zip(query.iter())
            .map(|(a, b)| a * b)
            .sum();
        (doc, sim)
    })
    .filter(|(_, sim)| *sim >= RAG_MIN_SIM)
    .collect();
scored.sort_by(|a, b| {
    b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)
});
scored.truncate(top_k);

我们需要设置阈值，是因为希望丢弃那些虽然排名较高、但并不相关的文档。

如果这些数学内容开始让你感到吃力，让我换一种简单说法：假设查询的向量是 [1.0, 0.0]，而一个方向相似的文档向量是 [0.8, 0.6]。两者的点积为 1.0×0.8 + 0.0×0.6 = 0.8。一个主要指向其他方向、代表无关术语的向量 [0.1, 0.995]，得分约为 0.1。哪个点积更接近 1.0，哪个就是赢家。该查询匹配结果为 $\text{Similarity} = (1.0 \times 0.8) + (0.0 \times 0.6) = 0.8$。

文档还必须至少获得 0.35 分才会被纳入结果。示例中的换机油查询得分为 0.816，而诸如“hi”之类的无关问候语在该语料库上的得分约为 0.10–0.15。这有助于我们剔除无关术语。

这是一个极简 RAG 示例，搜索最多只返回 2 份文档。管线会为语言模型准备这些文档，并将上下文限制在 320 个字符以内。语言模型将基于这些证据决定应回复什么，以及应执行什么动作。

6. 语言模型如何选择动作

现在，我们已经拥有用户的问题以及检索得到的上下文，可以进入下一阶段。在 voxlocal 中，我们会提示语言模型（SmolLM2）作出决策，并返回结构化工具调用，而不是自由形式的回答。

例如，对于“What does an oil change cost?”，预期输出是工具名称及其参数：

{
  "tool": "check_price",
  "args": {
    "service": "oil change"
  }
}

模型会一次生成一个 token；一个 token 可能是完整单词、单词的一部分、标点符号，或 JSON 片段。在每一步中，我们的模型都会基于前面的提示词和 token，预测哪个 token 最有可能成为下一个 token。Rust 代码通过 Candle 运行这一生成过程，并使用包含已学习信息的模型文件。

路由器会限制可生成的 token 数量，然后将结果解析为 ToolCall：

let (raw, truncated) = self.generate(&prompt, max_new)?;
let call = parse_tool_call(&raw)
    .or_else(|| truncated.then(|| repair_truncated(&raw)).flatten())
    .ok_or_else(|| anyhow!("model did not emit valid tool JSON: {raw:?}"))?;

上述代码片段展示了生成与解释之间的边界。模型返回的是文本值，而程序会尝试将该文本解释为规范化动作。voxlocal 不会自动将模型输出当作可执行命令。

小型语言模型可能会遇到问题，例如在 JSON 中途停止，或生成格式错误的输出。当发现生成结果被截断时，voxlocal 会尝试修复它。如果解析仍然失败，voxlocal 的路由组件会针对该轮对话返回错误。下一节将说明应用程序代码如何检查并执行有效工具调用。

7. 从模型输出到动作

正如我们所见，在这个阶段，模型生成的工具调用仍然只是文本。为了执行指定动作，voxlocal 必须解析它，并决定该工具被允许执行什么操作。

voxlocal 会解析模型生成的 JSON，然后将其提出的工具名称映射到受支持的工具集合。它还会从检索结果中获取服务项，以防

“ man muss immer umkehren” – Carl Gustav Jacob Jacobi

“When in doubt, subtract”

I’ve been working with voice agents for a while now, and they’re a fascinating piece of technology. It is magical to talk to a voice agent and get the work done. Ever wondered how voice agents work behind the scenes? This blog post is my attempt to understand their inner workings.

What is even a Voice agent?

An AI agent is an autonomous software system that uses AI models to reason, plan steps, and execute tasks to reach a specific goal. Voice agent is a version of it — A system that can engage in natural human-like spoken conversations to complete tasks.

In simple words, voice agents listen, think, do tasks, and speak.

Bird’s eye view

That’s a mouthful of a diagram. Since I was curious how all of this works, I created a bare-bones project in Rust to understand the finer details of what’s happening behind the scenes.

Let us get the legacy components out of our way before we dive into the voice agents part.

Telephony Provider

Telephony networking is a complex topic and way beyond the scope of this article. In our case, the Telephony Provider segment bridges legacy public telephone networks (PSTN/SIP) with cloud-based AI applications. It does complex tasks like handling the call lifecycle, bi-directional streaming, transcoding and keeping the latency low.

One major task this segment does is jitter buffering for dropped or delayed packets, and audio upsampling to preserve STT accuracy. This segment tries to keep the network latency under the threshold.

Modern telephony is an engineering marvel and closely related to the modern internet.

With that out of the way, let’s take a look at what we are cooking.

Introducing voxlocal, our avant-garde voice agent

Voxlocal is a fully local, low-latency voice agent for macOS. It’s a CLI tool written in Rust that lets the user talk to the voice agent and get a response back. In the spirit of experimentation, I have used bare bones components and micro models to get as close to the magic as possible. Don’t use it in production!

Its focus is very narrow: An automotive-service use case, through English language. I’m also using a few off-the-shelf components like Whisper and Piper instead of reinventing the wheel. To keep the pipeline inspectable and transparent,voxlocal deliberately avoids complex async streaming or VAD (Voice Activity Detection). I might dive deeper into them in the upcoming revisions.

Explore the source code:

Open-source project · Rust · macOS samkhawase / voxlocal A minimal, fully local voice agent built to make each step of the conversation easy to inspect. View repository ↗

Let us explore it step by step

1. What happens when you speak to a voice agent?

When you talk to a voice agent, or give it an audio sample, it breaks it down into multiple smaller tasks to do the work. It recognizes the words from the audio and cleans up the text by removing ‘Um’s, ‘ah’s. It uses the words to find relevant information and chooses an action based on them. Finally it turns the response back into speech.

In voxlocal, microphone audio is captured and passed to speech recognition first:

let capture = record_until(RECORD_SECS)?;
let transcript = stt.transcribe(&capture.samples)?;

The transcript then enters the shared turn pipeline. In the pipeline the agent normalizes the text, searches for matching context, routes the request to the relevant tool, and finally prepares a reply:

run_turn(
    &norm, &rag, &embedder, &mut llm, tts.as_mut(),
    transcript, &mut budget,
)?;

Each of the stages use a different technique to carry out their operations.

• Speech recognition determines the words from the audio
• Retrieval compares the text blocks in its numerical form using Maths1 (cosine similarity)
• An LLM predicts an action
• Plain old Rust code does the tool calls and executes the action.
• Speech synthesis generates audio from the final reply

However this is too dry and academic. Let’s unwrap the layers through a working example where the user asks the agent: “How much does an oil change cost?”

2. From sound wave to words

I was surprised to learn that the microphone captures sound by measuring air pressure continuously and turns the readings into audio samples. voxlocal then converts the captured audio to mono2 at 16,000 samples per second and then passes it on to Whisper for further processing.

Whisper is our trained neural network and it is designed to look for patterns in the audio that resemble speech sounds. It then estimates if the sound makes a sequence of words. The Whisper creators have trained it and fine tuned it for this task, our program simply piggybacks on it and calls the model3.

let mut params =
    whisper_rs::FullParams::new(
        whisper_rs::SamplingStrategy::Greedy { best_of: 1 }
    );
params.set_language(Some("en"));
params.set_temperature(0.0);
state.full(params, audio)
    .map_err(|e| anyhow!("whisper decode failed: {e}"))?;

For this example we use greedy decoding instead of sampling to pick the most likely next token. Temperature 0 means it will just return the first match of highest likelihood like oil or change. We can do this because our context is very limited and we know our dialogues.

Once we get the response from Whisper in segments, it is the job of voxlocal to join them into one transcript.

let mut out = String::new();
for i in 0..state.full_n_segments() {
    if let Some(segment) = state.get_segment(i) {
        if let Ok(text) = segment.to_str_lossy() {
            out.push_str(&text);
        }
    }
}
let transcript = out.trim().to_string();

In our case the output is:

“What does an oil change cost?”

At this stage the output is an estimate made from the output signal. It could vary a bit from the actual spoken words. Analog to digital transcribing is always tricky due to various factors like noise, accents, cadence, timbre etc. Hence we need an extra stage to clean it up and make it more consistent for further processing.

3. Cleaning up recognized speech

Humans are not always consistent when speaking and often add filler words like “Um”, awkward pauses or peculiar pronunciation of numbers (“noyynTherTee” instead of “Nine Thirty”). voxlocal applies simple and predictable rules to clean up the text before feeding it to the next stages.

For example, the normalizer in voxlocal removes common filler phrases, trims whitespace, and joins the separators:

s = self.thousands.replace_all(&s, "$1$2").into_owned();
s = self.fillers.replace_all(&s, " ").into_owned();
s = self.space.replace_all(&s, " ").into_owned();

When we say “Um yeah, what does an oil change cost?”, it might become “What does an oil change cost”. The program simply uses existing known transformations and lets the next stages interpret the meaning behind it. It’s simply regular expressions which are reused on each turn, keeping in line with our bare bones approach.

4. Turning words into vectors

Now that voxlocal has got the words sanitized, it need to find users intent from it. User can say “What does an oil change cost” and “How much for an oil service” both have the same intent. So how do we figure out the intent?

The answer is to use an embedding model, which converts the text into a list of numbers (you can say vector to sound cool). With this trick text with related meanings tend to have vectors pointing in the same direction.

We use MiniLM in voxlocal to create one vector for the normalized query.

pub fn embed_one(&self, text: &str) -> Result<Vec<f32>> {
    Ok(self.embed_batch(&[text])?.remove(0))
}

Inside the function embed_batch, MiniLM produces a representation for each token. The code then combines these token representations into one sentence vector using masked mean pooling. In simple words: it averages the real tokens and ignores padding added to make a batch the same length.

let mask = attention.to_dtype(DType::F32)?.unsqueeze(2)?;
let summed = hidden.broadcast_mul(&mask)?.sum(1)?;
let counts = mask.sum(1)?;
let pooled = summed.broadcast_div(&counts)?.contiguous()?;

How does the model determine which patterns matter for meaning and intent? The answer is in the model’s learned weights, which are pretrained and ready to use for voxlocal. The pooling code of voxlocal then reduces the token-level outputs to a single vector for the complete sentence. This is purely the model’s output representation; the Rust code does not define the meaning of each coordinate.

Finally, the vector is L2-normalized.

let norm = row.iter().map(|v| v * v).sum::<f32>().sqrt();
let unit_vector = row.iter().map(|v| v / norm).collect::<Vec<_>>();

In simple words, normalization makes sure every nonzero vector has length one. This is needed for the next stage which compares direction using a simple dot product. According to Maths, similar meanings point in the similar direction even when the wordings differ. And yes, it literally calculates the cosine of the angle literally and geometrically, if you are wondering. I can see my high school maths teacher Mr. Bhalérao, smiling at me with the smug look.

5. Finding relevant knowledge

With the query which is a unit-length vector voxlocal is now ready to compare it with the existing service documents. voxlocal compares the query vector with each document vector and finds useful context.

The dot products of the vectors, which is cosine similarity, are calculated and compared for similarity. A better match means the vectors point in a more similar direction. Our search compared the scores and drops the ones that are below our reference threshold. The remaining matches are sorted and the request number is returned.

let mut scored: Vec<(&Doc, f32)> = self.docs
    .iter()
    .map(|doc| {
        let sim: f32 = doc.embedding.iter()
            .zip(query.iter())
            .map(|(a, b)| a * b)
            .sum();
        (doc, sim)
    })
    .filter(|(_, sim)| *sim >= RAG_MIN_SIM)
    .collect();
scored.sort_by(|a, b| {
    b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)
});
scored.truncate(top_k);

We need the threshold because we want to discard documents that rank higher but are irrelevant.

If it’s getting too mathematical, let me explain in simple terms: Imagine the vector of our query is [1.0, 0.0], and a document vector pointing in a similar direction has a vector [0.8, 0.6]. The dot product of them is 1.0×0.8 + 0.0×0.6 = 0.8. A vector (unrelated term) pointing mostly another way and having a vector [0.1, 0.995], scores about 0.1. Whichever dot product is closer to 1.0 is the winner. The query match yields $\text{Similarity} = (1.0 \times 0.8) + (0.0 \times 0.6) = 0.8$.

A document must also score at least 0.35 to be included. The sample oil-change query scored 0.816 while an unrelated greeting such as “hi” scores around 0.10–0.15 against this corpus. This helps us weed out unrelated terms.

This is a minimal RAG example and the search returns only 2 documents at max. The pipeline prepares them for the language model and caps the context at 320 chars. The language model decides what response and action to perform based on this evidence.

6. How a language model chooses an action

We now have the user’s question and the retrieved context available for the next stage. In voxlocal, our language model (SmolLM2) is prompted to decide and return a structured tool call instead of a free-form answer.

E.g. For “What does an oil change cost?”, the intended output is a tool name plus its arguments:

{
  "tool": "check_price",
  "args": {
    "service": "oil change"
  }
}

The model produces this output one token at a time and a token can be a whole word, part of a word, punctuation, or a JSON fragment. At each step, our model predicts which token is the most likely the next one based on the preceding prompt and tokens. The Rust code runs that generation through Candle and it uses the model files which contain the learned information.

The router limits how many tokens can be generated, and then parses the result as a ToolCall:

let (raw, truncated) = self.generate(&prompt, max_new)?;
let call = parse_tool_call(&raw)
    .or_else(|| truncated.then(|| repair_truncated(&raw)).flatten())
    .ok_or_else(|| anyhow!("model did not emit valid tool JSON: {raw:?}"))?;

This snippet above shows the boundary between generation and interpretation. The model returns a text value, and the program attempts to interpret that text as a canonical action. The model’s output is not automatically treated as an executable command by voxlocal.

Small language models can run into trouble and stop mid-JSON or produce malformed output. When it notices a generation was truncated, voxlocal attempts a repair to overcome this. If the parsing still fails, the router component of voxlocal returns an error for that turn. The next section explains how a valid tool call is checked and executed by the application code.

7. From model output to action

As we’ve seen, the model-generated tool call is still text at this stage. In order to perform the specified action, voxlocal has to parse it and decide what the tool is allowed to do.

voxlocal parses the model’s JSON and then maps its proposed tool name onto the supported tool set. It also takes the service from the retrieved result in case one is available:

let call = parse_tool_call(&raw)
    .or_else(|| truncated.then(|| repair_truncated(&raw)).flatten())
    .ok_or_else(|| anyhow!("model did not emit valid tool JSON: {raw:?}"))?;
let mut call = call;
call.tool = snap_tool(&call.tool, query);
if let Some(hit) = top_hit {
    call.args.insert(
        "service".to_string(),
        serde_json::Value::String(hit.title.clone()),
    );
}

What we see here is an interesting boundary between a statistical prediction and programmatic behavior. The rust program parses the suggestions from the model and applies rules before execution.

For example, the router extracts a time from the current query instead of blindly trusting a time the model may have copied from an example.

The executor then handles the supported actions we have setup for this example:

match call.tool.as_str() {
    "book_appointment" => { /* prepare booking reply */ }
    "check_price" => { /* look up service price */ }
    other => format!("Unknown tool {other}."),
}

For this project nothing is actually executed, because this is a mock. In real life the app might call a calendar or an API.

8. Turning a reply back into sound

At this stage we are ready with an answer for the user and it’s time for the program to convert the text response back into speech. Piper converts words into an audio representation and then using its trained voice model generates the final audio. Like in the case of Whisper, we are piggybacking on the learned model weights that help us with timing, pitch, timbre etc.

voxlocal calls Piper through piper-rs and synthesize the voice:

pub fn speak(&mut self, text: &str) -> Result<(Vec<f32>, u32)> {
    self.piper
        .create(text, false, None, None, None, None)
        .map_err(|e| anyhow!("piper synth failed: {e}"))
}

voxlocal passes the resulting audio samples to rodio for playback, along with their sample rate.

let source = rodio::buffer::SamplesBuffer::new(
    1,
    sample_rate,
    samples.to_vec(),
);
player.append(source);
player.sleep_until_end();

We need to keep in mind that the neural synthesis happens inside Piper’s ONNX voice model4, not in this Rust wrapper. voxlocal gives Piper text and receives audio which is then played by rodio through the selected output device.

This final step completes the path from the caller’s speech to the agent’s spoken response. In a phone system, the final audio would be sent back through the call’s media stream instead of being played through local speakers.

Measuring the Pipeline

9. Where the time goes

The voice agent’s response time includes both computation and time spent recording or playing audio. voxlocal has a five-second recording window, but the model computation is not 5 seconds. We can see the details by running the program locally and here is one (abridged) request that shows the pipeline. The logs at different stages show the transformations and the latency budget shows how long the measured stages took.

[Stage 1: Voice capture]: capture complete: 80554 samples (5150 ms)
[Stage 2: Speech to text]: produced 37 characters: "What is the price for the oil change?"
[Stage 3: Text normalization]: output: "What is the price for the oil change?" (0.9 ms)
[Stage 4: Retrieval augmented generation]: selected document: oil change (0.816)
[Stage 4: Retrieval augmented generation]: using 86 context characters (12.2 ms)
[Stage 5: LLM tool routing]: generating up to 48 tokens with 86 context characters
  [llm]    {"tool": "check_price", "args": {"service": "brake pads"}}
  [tool]   check_price {"service": String("oil change")}
[Stage 5: LLM tool routing]: reply prepared: "oil change is $79." (68.6 ms)
[Stage 6: Text to speech]: generated 47616 samples at 22050 Hz (72.1 ms)
[Stage 7: Audio playback]: playback complete
latency budget:
  capture     5149.6 ms
  stt           55.1 ms
  normalize      0.9 ms
  rag           12.2 ms
  llm           68.6 ms
  tts           72.1 ms
  play        1135.3 ms
  TOTAL       6493.9 ms  <- total compute

Note: Interestingly, the model mistakenly reused brake pads from the appointment example in its prompt. The router corrected the service to oil change using the retrieved document before executing the price lookup.

10. One conversation, several kinds of computation

The diagram shows a single conversation moving through several representations:

This minimal voice agent helped me understand how each transformation makes the request flow through the pipeline. Whisper transcribes audio, MiniLM vectorizes the text, SmolLM2 predicts the tool call, Rust executes it, and Piper sends the reply back to the user. It’s fun to observe and debug this voice agent and it’s moving parts.

Final Thoughts

I have been struggling with the deluge of information from the past year, and it is hard to keep up. In such cases, whenever I’m drowning in information, I peel the layer and go back to the first principles to reorient myself. An afternoon of vibe engineering, and imagining voice agents from the first principles helped me understand how all these stages and models come together to provide a seamless experience.

Just like what Jacobi and Munger said.