专家称，Kimi K3 的强大能力并非仅靠利用 Anthropic 的 Fable 获得

白宫科技顾问 Michael Kratsios 表示，中国公司 Moonshot——最大可用开放权重 LLM Kimi K3 的开发商——通过复制 Anthropic 的 Fable LLM 构建了其模型，同时还使用了未经批准出口至中国的芯片。

“以窃取美国专有技术、破坏美国研究为目的的大规模秘密工业蒸馏，是不可接受的，”Kratsios 写道。此前有报道称，美国正讨论禁止中国开放权重模型，这一讨论已搅动 AI 行业。Moonshot 未回应有关其训练流程的问题，Kratsios 也没有披露其指控来源的更多细节。

Kratsios 的推文呼应了美国财政部长 Scott Bessent 的说法： “我们在许多中国模型中发现了美国大语言模型的水印，这不可接受。”目前尚不清楚这些水印具体是什么，财政部也未回应置评请求。

不过，专家怀疑，蒸馏——即通过查询 LLM 来探究其内部工作方式并复制其能力的过程——是否应为 Kimi K3 所展现的先进能力负责。

Laude Institute 研究员、Snorkel AI 联合创始人 Braden Hancock 向 TechCrunch 表示：“我不认为仅靠严格意义上的蒸馏，能够在 Fable 发布后这么短时间内获得如此强大的模型。坦率地说，根本没有足够时间，对吧？Fable 从 7 月 1 日起才公开可用。你不可能在两周内蒸馏出这么多数据、训练一个模型，然后将其发布。”

Allen Institute for AI 的 AI 研究员 Nathan Lambert 在昨天发布的一期播客中表示：“我的观点是，随着中国模型越来越接近前沿、训练机制转向[强化学习]，蒸馏的影响正在变得越来越小。假如情况并非如此，任何人都能轻易利用其数据进行蒸馏，从而追上 GLM 或 K3。但我们并没有、也不会仅凭监督微调看到这种结果。”

执行蒸馏要求实验室系统性地查询目标模型，以生成可用于后训练的数据。有时，这会明确要求模型阐述其思维链，以了解它如何解决问题。另一些时候，模型的提示词与回答会被用来训练新模型，这一过程被称为监督微调（supervised fine-tuning，SFT）。

正是这一微调过程，可能导致一个表面上由第三方创建的模型声称自己是 Claude。Lambert 认为，微调是“模型学会礼仪”的环节。

但 Lambert 表示，随着模型变得更加复杂，SFT 的收益正变得不那么重要。要蒸馏出类似 Fable 的能力，很可能需要强化学习技术。在许多情况下，这意味着让较大模型的 Agent 对较小模型的回答进行评分，并根据评分作出调整。

更先进的技术还需要更大规模的基础设施。大规模强化学习训练可能需要数千万个 Agent。通过前沿实验室的 API 来完成这件事“将极其昂贵，而且很可能成为时间瓶颈，因为这些模型相当慢；坦率说，甚至未必能带来性能提升”。

此前的前沿模型似乎很可能对 Kimi 有所贡献；Anthropic 今年早些时候曾公开指控 Moonshot、DeepSeek 和 MiniMax 系统性蒸馏其模型。Anthropic 表示，它通过 IP 地址及其他元数据，发现其模型与识别为上述公司用户之间存在数百万次交互。这些查询“不同于正常使用模式，反映出蓄意提取能力，而非合法使用”。Anthropic 未回应 TechCrunch 关于 Fable 蒸馏的询问。

然而，蒸馏被认为是 AI 公司中的普遍做法，并不仅限于中国。Elon Musk 今年早些时候作证，称他的公司 SpaceXAI 曾蒸馏 OpenAI 模型以开发 Grok，并称这种做法在业内很常见。例如，蒸馏与构建合成数据集之间的界限可能相当模糊。

Hancock 表示：“总体而言，美国人低估了这些中国团队的技术专长。Moonshot 的一位创始人曾是 CMU 的博士生。这些都是在扎实开展工作的真正研究人员和工程师。……如果美国模型的发展停滞，我认为中国的进展会放缓，但仍将继续。他们并不只是在搭便车。”

Kratsios 评论中的另一部分——即 Moonshot 获得了先进 Nvidia 芯片 Grace Blackwell 300，以及使用了位于泰国、配备 GB300 的服务器——同样很难与蒸馏问题切割开来。乔治城大学安全与新兴技术中心研究员 Sam Bresnick 表示，这些芯片被禁止出口到中国，但黑市确实存在。今年 5 月，美国服务器制造商 Supermicro 的创始人因向中国走私先进芯片而被起诉。

Bresnick 表示：“我支持在全球范围内为数据中心推行‘了解你的客户’法律。如果你允许一家公司在你的最先进硬件上进行大规模训练，就需要建立报告机制，说明这家公司是谁、在做什么。”

Joe Biden 政府的商务部曾于 2024 年提议针对数据中心制定联邦层面的‘了解你的客户’规则，但在 Donald Trump 任内似乎没有取得进一步进展。不过，向海外运送先进芯片的出口商，应当确保这些芯片仅被用于获批准的用途。

White House science advisor Michael Kratsios said that Moonshot, the Chinese company behind the Kimi K3, the largest available open-weight LLM, built its model by copying Anthropic’s Fable LLM while using chips that aren’t cleared for export to China.

“Large-scale, covert industrial distillation aimed at stealing proprietary U.S. technology and undermining American research is unacceptable,” Kratsios wrote, amid reported discussions about banning Chinese open-weight models that have roiled the AI sector. Moonshot did not respond to questions about its training process, and Kratsios did not share more details about the sources of his allegations.

Kratsios’ tweet echoed comments from Treasury Secretary Scott Bessent that “we are finding watermarks of our U.S. large language models on many of the Chinese models, and that that’s unacceptable.” It’s not clear what those watermarks consist of, and the Treasury Department did not respond to a query.

However, experts are skeptical that distillation—the process of querying an LLM to determine its inner workings and copy its capabilities—is responsible for the advanced capabilities that Kimi K3 displays.

“I don’t think you get a model this strong and this quickly on the heels of Fable doing strictly distillation,” Braden Hancock, a researcher at the Laude Institute and co-founder of Snorkel AI, told TechCrunch. “There’s just not even frankly time, right? Fable’s only been publicly available since July 1st. You can’t distill that much data, train a model, and release it in two weeks.”

“I’ve been of the opinion that distillation has becoming less and less impactful over time as the Chinese models get closer to the frontier and the training regime shifts to [reinforcement learning],” Nathan Lambert, an AI researcher at the Allen Institute for AI, said in a podcast released yesterday. “[I]f it were the case, everyone would be easily able to catch up to a GLM or to a K3 by using its data for distillation. But we have not, or we won’t see this, from supervised fine-tuning alone.”

Performing distillation requires a lab to systematically query its target model in order to generate data that can be used for post-training. Sometimes this explicitly involves asking the model to articulate its chain-of-thought to understand how it solves problems. Other times, the prompts and responses from a model are used to train a new model in a process called supervised fine-tuning, or SFT.

It’s this fine-tuning process that can result in a model ostensibly created by a third party claiming that it is Claude. Fine tuning is where, in Lambert’s view, the “model picks up its manners.”

But Lambert says that the benefits of SFT are becoming less important as models become more complex. To distill Fable-like capabilities would likely require reinforcement learning techniques. In many cases, that means having an agent of the larger model grade the smaller model’s responses, and adjusting based on the grade.

The more advanced techniques also require more significant infrastructure. Large reinforcement learning runs can require tens of millions of agents. Using a frontier lab’s API to do that “would be insanely expensive and potentially it would probably be a time bottleneck because these models are pretty slow and to be frank might not even give you a performance uplift.”

It seems likely that previous frontier models might have contributed to Kimi; Anthropic publicly accused Moonshot, DeepSeek and MiniMax of systematically distilling its models earlier this year. Anthropic said it discovered millions of exchanges between its models and users it identified at those companies through IP addresses and other meta data. Those queries were “distinct from normal usage patterns, reflecting deliberate capability extraction rather than legitimate use.” Anthropic didn’t respond to TechCrunch’s queries about Fable distillation.

However, distillation is seen as common among AI companies, not just in China. Elon Musk testified earlier this year that his company SpaceXAI distilled OpenAI models to develop Grok, and that the practice was common in the industry. The line between distillation and developing synthetic data sets, for example, can be fairly blurry.

“[I]n general, Americans are understating the technical expertise of these Chinese teams,” Hancock said. “One of the founders of Moonshot was a CMU PhD student. These are legitimate researchers and engineers doing solid work. …if American models ground to a halt, I think China’s progress would slow, but would still continue. They’re not just riding coattails here.”

It’s also hard to disentangle distillation from the second part of Kratsios’ comment — that Moonshot had obtained advanced Nvidia Chips, Grace Blackwell 300s, and also accessed GB300 equipped-servers in Thailand. Those chips are banned from export to China, but a black market exists, according to Sam Bresnick, a research fellow at Georgetown’s Center for Security and Emerging Technology. In May, the founder of Supermicro, a US server builder, was indicted for smuggling advanced chips into China.

“I am a proponent of know your customer laws for data centers across the world,” Bresnick said. “If you are letting a company conduct huge training runs on your state-of-the-art hardware, there needs to be a reporting mechanism for who that company is and what they’re doing.”

President Joe Biden’s Department of Commerce proposed federal know-your-customer rules for data centers in 2024, but no further progress appears to have been made under Donald Trump. Exporters shipping advanced chips abroad, however, are supposed to ensure they are only used for approved purposes.