在浏览器中用小型语言模型运行 RAG

几周前，MicroLLM lab 登上了 Hacker Newsletter。它展示了如何在用户自己的 GPU（或 CPU）上，通过浏览器运行参数量为 2500 万至 3.6 亿的小型语言模型。无需服务器、API 密钥或账户。这很有意思，也让我开始思考此前为博客添加语义搜索之后还能做什么：这个功能已经能为问题提供 RAG 中的“R”，而且完全在浏览器中运行，只需要一张 4 MB 的查找表。如果再让一个同样运行在浏览器中的小模型阅读这些段落并回答问题，会怎么样？

于是我做了这个功能：“向我的博客提问”。搜索会找到最相关的四个段落，浏览器中运行的小语言模型再用两句话概括它们。你输入的任何内容都不会离开你的设备。不过，“小模型负责概括内容”这样的说法很容易掩盖大量凭空编造的内容，因此我测量了它到底有多经常能忠实反映我写的东西。结果发现，小模型确实有一些局限。

视频演示：在 WebGPU 上使用“向我的博客提问”，已缓存的 Qwen3-0.6B 先作答，随后切换到 SmolLM2-360M 并下载该模型。

工作原理

这个页面采用检索增强生成（RAG），所有环节都在客户端运行：

1. 检索。 问题会经过与首页相同的静态嵌入搜索，但结果不会按文章汇总，而是保留得分最高的四个段落。
2. 门控。 如果最佳段落的得分低于阈值，页面根本不会调用模型，而是提示未能在博客中找到相关内容。其思路是：如果用户的问题没有对应数据，就不必费力生成答案。下文会解释为什么这种方法只能部分奏效。
3. 提示。 将编号后的段落与系统提示一起交给模型：最多用两句话回答，只能使用这些段落；如果段落不能回答问题，就明确说明。
4. 生成。 Web Worker 使用 transformers.js 在 WebGPU 上运行模型；如果浏览器不支持 WebGPU，则通过 WebAssembly 在 CPU 上运行，并将答案流式输出到页面。一旦生成两句完整的话，就立即停止。

页面会先以卡片形式展示段落，摘要则显示在段落上方。完全不加载模型也可以搜索，只是只能看到段落。加载模型需要用户明确点击，因为这意味着要从 Hugging Face 下载几百 MB 的数据（只需下载一次，浏览器会缓存）。页面还有一个清除缓存的按钮。

选择哪个模型？

我希望读者点击下载时不会觉得负担太重：体积应该是几百 MB，而不是几个 GB（万一有人在公交车上阅读呢）。具备可直接用于浏览器的 ONNX 构建版本的候选模型包括 SmolLM2（135M 和 360M 参数）、Gemma 3 270M、LFM2.5-350M、Qwen2.5-0.5B 和 Qwen3-0.6B。为了比较，我还加入了两个绝不会实际部署在页面上的参考模型：Gemma 4 E2B，Google 新款 Gemma 4 系列中最小的模型，下载体积为 3.1 GB；以及通过 API 调用的 Claude Haiku 5.5，也就是“直接付费让别人提供服务”的选项。

还没开始运行，就有一个意外发现：其中几个模型的“4 位”（q4）构建版本比结合 16 位浮点数的 4 位版本（q4f16）更大，因为 q4 导出版本以完整的 32 位精度保留了嵌入表。SmolLM2-360M 的 q4 版本为 390 MB，q4f16 版本则为 275 MB，输出完全相同。下文均使用 q4f16。

评测

我复用了语义搜索文章评测中的 30 个问题，把它们改写成真正的问句，例如“博客如何使用 Lunr 添加搜索功能？”，并额外加入了 15 个博客肯定没有回答的问题。其中少数完全不相关，例如“我怎么在家烤酸面包？”，但大多数是主题相近却没有答案的问题，旨在诱使模型预测出听起来合理、实际错误的词元，例如“我应该买什么 GPU 来本地运行 Llama 70B？”“BM25 排序相比 TF-IDF 有什么改进？”“如何微调 Kokoro，让它用我自己的声音说话？”博客谈到过这些话题或类似内容，但并没有回答这些问题。

每个模型都拿到相同的四个段落和相同的提示。Opus 5.5 根据这些段落将每个回答评为忠实（每项主张都能在段落中找到依据）、无依据（至少一项主张没有依据，即使它恰好是真的）或未作答。我在没有看到评判模型结论的情况下，亲自标注了其中 20 条；在回答是否忠实的问题上，我们有 16 条判断一致。问题集也是由 Claude 生成、再由我审核的。

忠实回答数与下载体积。 下图比较各模型在 30 个可回答问题中的忠实回答比例与下载体积，体积采用对数刻度。350 MB 以下的模型得分为 0 至 3；Qwen3-0.6B（579 MB）为 15；Gemma 4 E2B（3.1 GB）为 28；通过 API 调用的 Claude Haiku 5.5 为 27。原图可将鼠标悬停在数据点上查看详情。

模型	下载体积	30 个问题中的忠实回答数	15 个问题中的编造回答数
下载体积不超过 350 MB
SmolLM2-135M	120 MB	0	14
LFM2.5-350M	258 MB	2	14
SmolLM2-360M	275 MB	3	15
Gemma 3 270M	293 MB	3	13
下载体积更大
Qwen3-0.6B（默认）	579 MB	15	5
Gemma 4 E2B	3.1 GB	28	0
托管 API
Haiku 5.5	API	27	1

事实证明，现成的 350 MB 以下模型没有一个适合这个用途。它们不是稍微差一点，而是在 30 个问题中最多只有三次忠实作答，而且几乎无论你问什么，它们都会乐于给出答案。当被问到应该买哪款 GPU 来运行 Llama 70B，而提供给它们的四段内容只涉及嵌入和 MacBook 上的编码 Agent 时：

“……在本地运行 Llama 70B 的最佳选择可能是 NVIDIA GTX 960 或 GTX 970。……它们还提供高内存带宽，非常适合处理 Llama 70B 所需的 640 万份文档等大量数据。”（SmolLM2-360M）

“……NVIDIA A100 系列等高性能显卡（GPU）（例如 RTX 3090、RTX 3080、3070、3080、3070M、3080M）。”（Gemma 3 270M）

“……热门选项包括 RTX 3090、RTR 5900，或者 TensorRT 60+ 等更新一代产品，以提高计算效率。”（LFM2.5-350M）

顺便说一句，Gemma 4 E2B 正确判断：“所提供的段落不包含关于应该购买哪款 GPU 来本地运行 Llama 70B 的信息。”Google，做得好。

即使面对博客确实回答过的问题，小模型也很容易把内容搞乱。下面是 LFM2.5 对我那篇 Bloom 过滤器文章中最基本问题的回答，在我笔记本的 GPU 上，生成速度为每秒 73 个 token：

又快又流畅，而且是错的。Bloom 过滤器不是哈希表：它是一个位数组；只有当元素的每一个 k 个哈希函数都映射到已置位的位时，才会被判定为“可能在集合中”。回答的最后一部分实际上是正确的：如果某个位没有置位，就意味着该元素从未被加入，这也是 Bloom 过滤器不会出现假阴性的原因。但其余内容都是编造的。公平地说，搜索并没有把解释 Bloom 过滤器工作原理的那一节交给模型；它拿到的四个段落讲的是使用场景和交互式演示。摘要的质量不可能超过它所阅读的段落。看来我这个小搜索引擎的相关性评分还有改进空间。

Qwen3-0.6B 则比较有意思。它有 579 MB，相当大，超过了我设定的预算。它有一半的时间能忠实作答，基本上相当于抛硬币。不过，它是最小的一个表现达到“抛硬币水平”而不是“几乎肯定答错”的模型，因此“向我的博客提问”默认加载它。你也可以选择三个更小的模型，亲自看看未经微调的 300 MB 模型会怎样回答你的问题。

小模型会陷入循环

我第一次运行这些小模型时，它们的大多数回答与其说是错误，不如说是卡住了。问 SmolLM2-135M 如何为网站添加 OpenSearch 支持时，它回答：

“1. 自定义搜索：从你的 URL 地址栏搜索 2. 从你的 URL 地址栏搜索 3. 从你的 URL 地址栏搜索 4. 从你的 URL 地址栏搜索 [1] 自定义搜索：从你的 URL 地址栏搜索……”

Gemma 3 270M 回答 GPU 问题时则输出 I'5555555555555555555555...，直到耗尽 token。总是选择概率最高的下一个 token，又没有任何机制抑制重复，就会出现这种情况：小模型会陷入循环，无法脱身（我玩 MicroLLMLab 时也经常看到这种情况）。简单的解决办法是设置重复惩罚，本例中为 1.15，即略微降低模型已经见过的 token 再次出现的概率。这并不能修复忠实度或质量问题；模型只是改为生成更长、更多样的胡言乱语。上面的所有数据都使用了 1.15 的重复惩罚。

用相关性门槛拦截明显不相关的问题

相关性门槛看起来是解决编造答案问题的一种低成本办法：如果搜索找不到相关内容，就不要问模型。它对明显跑题的问题有效。例如，“我怎么烤酸面包？”与博客中最佳匹配段落的得分为 0.25，设置 0.45 的阈值就能拦截它。这是好事，因为 Gemma 3 给出的答案是一份面包食谱。

但那些主题相近的问题得分为 0.54 至 0.63，高于博客实际上回答过的几个问题，因为它们会匹配到相同主题的文章。要拦住这些问题，就必须同时拒绝 30 个真实可回答问题中的 13 个。

因此，门槛只能抓住明显跑题的问题，其余的还得靠模型自己处理。这才是模型大小在这里很重要的真正原因：Gemma 4 和 Haiku 会说“这些段落没有涵盖这个问题”，而小模型从来不会。

租 GPU 或付费使用 API 值得吗？

通常值得。按目前的价格来看（竞争真好），除非你有充分理由不这么做，例如合规、隐私或隔离网络部署等，否则自行运行几乎从来不划算。不过，我们不妨做个思想实验：3.1 GB 的 Gemma 4 E2B 在质量上与 Haiku 相当，那么，如果由我们运行模型供读者使用，而不是让他们下载，成本会是多少？适合它的最便宜云 GPU 是 Google Cloud 上的 NVIDIA L4，价格约为每小时 0.71 美元，持续运行一个月需要 516 美元。按照这个博客每月的读者数量，假设每个人都提一个问题，每个问题的成本仍会超过一美元。

另一方面，Haiku 回答全部 45 个评测问题只花了 0.006 美元。如果每月有 500 个问题——人总得有点梦想——成本约为 7 美分。换句话说，如果你有真正的应用场景，就从 Haiku 开始。

回答越短越安全

如果 Qwen3 两句话的回答只有一半正确，让它多说一点会有帮助吗？我用 Qwen3-0.6B 和 Gemma 4 E2B 做了一次快速实验，分别允许它们最多回答两句、四句或六句。

忠实回答数与允许的回答长度。 在 30 个可回答问题中，允许最多两句、四句或六句时，Gemma 4 E2B 的忠实回答数分别为 28、26、24；Qwen3-0.6B 分别为 15、9、5。对于博客未覆盖的 15 个问题，Gemma 4 E2B 在三种长度限制下的编造回答数均为 0；Qwen3-0.6B 分别为 5、7、6。

两个模型的回答空间越大，忠实度都越低，但较小的模型下降得尤其厉害：Qwen3 在 30 个问题中的忠实回答数从 15 降到 9，再降到 5。每多一句话，就多一次产生幻觉的机会。允许六句话时，它对 GPU 问题完全不再拒答，而是热心地概括每一个段落，最后得出结论：“答案取决于用户需要高性能 GPU 还是标准 GPU。”

Gemma 4 的表现也有所下降，依次为 28、26、24，但它几乎没有用满额外的篇幅：允许六句话时，平均只用了约 2.8 句，而且从未对博客没有涵盖的问题编造答案。顺便向 Gemma 4 致敬，这个模型系列相当出色。

Listen to this article instead

▶ 0:00 / 0:00 0.75×1×1.25×1.5×2×

Your browser does not support the audio element

A few weeks ago MicroLLM lab was featured in Hacker Newsletter. It shows off how you can run small language models (25 to 360 million parameters) in a browser on the users’ own GPU (or CPU). No server, no API key, no account. Pretty nifty, so it got me thinking about the steps after semantic search I added to this blog: that already provides the “R” in RAG for a question, and it does that entirely in the browser, from a 4 MB lookup table. What if a tiny model, also in the browser, read those passages and answered the question?

So I built that: “ask my blog”. Search finds the four most relevant passages, and a small language model running in your browser summarises them in two sentences. Nothing you type leaves your machine. And because “a small model summarizes things” is exactly the sort of sentence that hides a lot of made-up nonsense, I measured how often it’s actually faithful to what I wrote. Turns out small models have some limitations.

HD

Ask my blog on WebGPU: Qwen3-0.6B (already cached) answers, then switching to SmolLM2-360M downloads it.

How it works#

The page is retrieval-augmented generation (RAG) with every piece running client-side:

1. Retrieve. Your question goes through the same static-embedding search as the homepage, but instead of rolling results up to posts, it keeps the four best-scoring passages.
2. Gate. If the best passage scores below a threshold, the model isn’t called at all and the page says it couldn’t find this on the blog. The idea is to not bother if a user asks something there’s no data for. More on why that only half works below.
3. Prompt. The passages, numbered, plus a system prompt: answer in at most two sentences, use only these passages, say so if they don’t answer the question.
4. Generate. A Web Worker runs the model with transformers.js on WebGPU, or on the CPU via WebAssembly if your browser doesn’t have WebGPU, and streams the answer into the page. It stops as soon as it has two complete sentences.

The passages are shown first, as cards, and the summary will sit above them. You can search without loading a model at all; you just get the passages. Loading one is an explicit click, because it means downloading a few hundred megabytes from Hugging Face (once; your browser caches it). There’s also a button to clear the cache/

Which model?#

I wanted something that would be a reasonable click for a reader: a few hundred megabytes, not gigabytes (what if someone reads this on the bus). The candidates that have browser-ready (ONNX) builds were SmolLM2 (135M and 360M parameters), Gemma 3 270M, LFM2.5-350M, Qwen2.5-0.5B and Qwen3-0.6B. For comparison I added two references that would never ship on the page: Gemma 4 E2B, the smallest of Google’s new Gemma 4 models at a 3.1 GB download, and Claude Haiku 5.5 through the API, the “just pay someone” option.

One surprise before anything ran: for several of these models the “4-bit” (q4) build is bigger than the 4-bit-with-16-bit-floats (q4f16) one, because the q4 export keeps the embedding table in full 32-bit precision. SmolLM2-360M is 390 MB as q4 and 275 MB as q4f16, with identical output. Everything below uses q4f16.

The eval#

I reused the 30 questions from the eval I did for the semantic search post, rephrased as actual questions (“How did the blog use Lunr to add search?”), and added 15 questions this blog definitely doesn’t answer. A few of those are far off (“How do I bake sourdough bread at home?”), but most are near-misses designed to tempt a model into predicting plausible-sounding-but-wrong-tokens: “Which GPU should I buy to run Llama 70B locally?”, “How does BM25 ranking improve on TF-IDF?”, “How do I fine-tune Kokoro to speak in my own voice?” The blog talks about these, or similar things, but doesn’t have and answer for those.

Every model got the same four passages and the same prompt, and Opus 5.5 graded each answer against those passages as faithful (every claim is in the passages), unsupported (at least one claim isn’t, even if it happens to be true) or a non-answer. I labelled 20 of those grades myself without seeing the judge’s verdict; we agreed on 16 of 20 about whether an answer was faithful. Claude also generated the question set, which I reviewed.

Faithful answers vs download sizeShare of 30 answerable questions answered faithfully, per model, against download size on a log scale. Models up to 350 MB score 0 to 3 out of 30; Qwen3-0.6B (579 MB) 15; Gemma 4 E2B (3.1 GB) 28; Claude Haiku 5.5 via API 27.fits a 350 MB budget0%25%50%75%100%Faithful answers (of 30)100 MB300 MB1 GB3 GBAPIDownload size (log scale)SmolLM2-135M: 0/30 faithful; invented an answer for 14/15 questions the blog doesn't cover (120 MB download)SmolLM2-360M: 3/30 faithful; invented an answer for 15/15 questions the blog doesn't cover (275 MB download)Gemma 3 270M: 3/30 faithful; invented an answer for 13/15 questions the blog doesn't cover (293 MB download)LFM2.5-350M: 2/30 faithful; invented an answer for 14/15 questions the blog doesn't cover (258 MB download)Qwen3-0.6B: 15/30 faithful; invented an answer for 5/15 questions the blog doesn't cover (579 MB download)Gemma 4 E2B: 28/30 faithful; invented an answer for 0/15 questions the blog doesn't cover (3131 MB download)Claude Haiku 5.5: 27/30 faithful; invented an answer for 1/15 questions the blog doesn't coverHaiku 5.5 · 27/304 models up to 293 MB0 to 3 of 30 faithfulQwen3-0.6B · 15/30Gemma 4 E2B · 28/30Faithful answers out of 30 answerable questions, against download size. Hover a dot for details.

Model	Download	Faithful of 30	Invented of 15
Fits a 350 MB download
SmolLM2-135M	120 MB	0	14
LFM2.5-350M	258 MB	2	14
SmolLM2-360M	275 MB	3	15
Gemma 3 270M	293 MB	3	13
Larger downloads
Qwen3-0.6B default	579 MB	15	5
Gemma 4 E2B	3.1 GB	28	0
Hosted API
Haiku 5.5	API	27	1

Turns out nothing off the shelf under 350 MB is usable for this. They aren’t slightly worse; they’re faithful at most three times out of thirty , and they’ll happily answer almost any question you throw at them. Asked which GPU to buy for Llama 70B, with four passages about embeddings and a coding agent on a MacBook:

“…the best option to run Llama 70B locally is likely to be a NVIDIA GTX 960 or GTX 970. […] They also offer high memory bandwidth, making them ideal for handling large amounts of data such as the 6.4 million documents required by Llama 70B.” (SmolLM2-360M)

“…a high-performance graphics card (GPU) such as a NVIDIA A100 series (e.g., RTX 3090, RTX 3080, 3070, 3080, 3070M, 3080M)” (Gemma 3 270M)

“…popular options include RTX 3090, RTR 5900, or newer generations like TensorRT 60+ for improved compute efficiency.” (LFM2.5-350M)

Gemma 4 E2B, for the record, correctly decided that “The provided passages do not contain information about which GPU you should buy to run Llama 70B locally.” Good job Google.

Even on questions the blog does answer, the small models are prone to garble things. Here’s LFM2.5 on the most basic question about my Bloom filter post, at 73 tokens per second on my laptop’s GPU:

Fast, fluent, and wrong. A Bloom filter isn’t a hash table: it’s a bit array, and an element counts as “probably in the set” only if every one of its k hash functions lands on a bit that’s set. The last clause is actually correct (a missing bit means the element was never added, which is why Bloom filters can’t give false negatives), but the rest is made up. To be fair to the model, search didn’t hand it the section that explains how Bloom filters work; the four passages it got were about use cases and the interactive demo. A summary can only be as good as the passages it reads. Seems like there’s some work to do on the relevance scoring in my little search engine that could.

Qwen3-0.6B is the interesting one. At 579 MB it’s pretty thicc, and over the budget I’d set myself. It’s faithful half the time, which is basically a coin flip. However, it’s the smallest model that’s a coin flip rather than a pretty much guaranteed miss, so that’s what “ask my blog” loads by default. You can also choose three smaller models so you can see for yourself what a non-finetuned 300 MB model does with your question.

Small models loop#

The first time I ran the small models, most of their answers weren’t wrong so much as stuck. SmolLM2-135M, asked how to add OpenSearch support to a website:

“1. Custom Search: Search from your URL bar 2. Search from your URL bar 3. Search from your URL bar 4. Search from your URL bar [1] Custom Search: Search from your URL bar…”

Gemma 3 270M answered the GPU question with I'5555555555555555555555... until it ran out of tokens. That’s what happens when you always pick the single most likely next token and nothing discourages repeating it: tiny models fall into a groove and can’t get out (I saw it happen pretty often when I was playing around with MicroLLMLab too). The easy fix is a repetition penalty (1.15 in this case, which slightly discourages tokens the model has already seen). It doesn’t fix the faithfulness or quality; the models just write longer, more varied nonsense instead. All the numbers above are with the penalty set to 1.15.

A relevance gate to catch the obvious ones#

A relevance gate seemed like the cheap fix for made-up answers: if search can’t find anything relevant, don’t ask the model. It works for the far-off questions; “How do I bake sourdough bread?” scores 0.25 against the best passage on the blog, and a threshold of 0.45 stops it (which is good, because Gemma 3’s answer was a bread recipe). But the near-misses score 0.54 to 0.63, higher than several questions the blog actually answers, because they land on posts about the same topic. To block those, the threshold would have to reject 13 of the 30 real questions too.

So the gate only catches the obviously off-topic questions, and the rest is on the model. That’s the real reason size matters here: Gemma 4 and Haiku say “the passages don’t cover this”, and the small models never do.

Is it worth renting a GPU, or paying for an API?#

Usually, yes. At current pricing (yay competition) it’s almost never worth it, unless you have strong reasons not to (ie compliance, privacy, air gapped deployments, etc). As a thought exercise tho: the 3.1 GB Gemma 4 E2B matches Haiku on quality, so what would it cost to run it for readers instead of making them download it? The cheapest sensible cloud GPU for it, an NVIDIA L4 on Google Cloud, is about $0.71 an hour, or $516 a month to keep running. For the amount of readers this blog sees in a month, assuming each and everyone asks a question, that would be over a dollar per question.

Haiku on the other hand answered all 45 eval questions for $0.006. At 500 questions a month (a guy can dream) it would cost about seven cents. In other words, if you have a real use case, start with Haiku.

Shorter is safer#

If two sentences from Qwen3 are only right half the time, would letting it say more help? I ran a quick experiment with Qwen3-0.6B and Gemma 4 E2B, allowing for two, four or six sentences.

Faithful answers vs allowed answer lengthFaithful answers out of 30 when the model may answer in at most 2, 4 or 6 sentences. Gemma 4 E2B: 28, 26, 24. Qwen3-0.6B: 15, 9, 5.Gemma 4 E2BQwen3-0.6B0%25%50%75%100%Faithful answers (of 30)246Sentences the model may useGemma 4 E2B, at most 2 sentences: 28/30 faithful; invented an answer for 0/15 off-blog questionsGemma 4 E2B, at most 4 sentences: 26/30 faithful; invented an answer for 0/15 off-blog questionsGemma 4 E2B, at most 6 sentences: 24/30 faithful; invented an answer for 0/15 off-blog questions28/30Gemma 4 E2B · 24/30Qwen3-0.6B, at most 2 sentences: 15/30 faithful; invented an answer for 5/15 off-blog questionsQwen3-0.6B, at most 4 sentences: 9/30 faithful; invented an answer for 7/15 off-blog questionsQwen3-0.6B, at most 6 sentences: 5/30 faithful; invented an answer for 6/15 off-blog questions15/30Qwen3-0.6B · 5/30Faithful answers out of 30 when the model may use 2, 4 or 6 sentences.

Both get less faithful with more room, but the smaller model falls off a cliff: Qwen3 goes from 15 to 9 to 5 faithful answers out of 30. Every extra sentence is another chance to hallucinate. At six sentences it stops declining the GPU question altogether and helpfully summarises every passage instead, before concluding that “the answer would depend on whether the user needs a high-performance GPU or a standard one.”

Gemma 4 slips too (28, 26, 24), but it barely uses the extra room (about 2.8 sentences when allowed 6) and never makes up an answer to a question the blog doesn’t cover. Shoutout to Gemma 4 btw, this model family is pretty awesome. This is a great option for finetuning.

What’s next#

The next step would be making the small models less bad at this one job. A 300 MB model doesn’t need to know everything; it needs to read four passages and either summarise them or say they don’t answer the question. That sounds like a fine-tuning problem, and maybe a follow-up post.

The other idea is the homepage search box: if you’ve already loaded a model on “ask my blog”, the homepage could quietly use it to improve search, without ever downloading anything on its own. In the meantime: ask my blog something, and check the passages before you believe the summary.