AI News
官方更新 Meta Engineering Blog Mon, 19 Oct 2026 正文 ok 1389 words

GenRecs 分词技术：Meta 如何通过稳定模型词表重塑广告推荐

英文标题：GenRecs Tokenization: Transforming Ads Recommendation at Meta with Stable Model Vocabulary

Meta 介绍了其广告推荐系统的最新演进——生成式推荐器（Generative Recommenders, GenRecs）的分词技术（GenRecs Tokenization）。该技术彻底改变了传统为每个实体维护独立 ID 的方式，转而使用包含通用世界知识的稳定语义词表将广告、商品和主页等表示为离散 token，从而使模型具备真正的跨实体推理与泛化能力。在生产环境中，该技术与 Meta 的 GEM 广告大模型及自适应排序模型配合，为广告主带来了 +0.35% 的全场景转化率提升。这一从“单广告打分”到“语义推理”的范式转换为工业级推荐系统的大模型原生化（LLM-native）奠定了底层数据基石。
打开原文相关 3 · 机会 1
media8.4 KB media406 B media848.8 KB media134.0 KB media103.0 KB media187.2 KB media103.0 KB media209.9 KB media103.0 KB media258.1 KB media103.0 KB media42.7 KB media11.3 KB media136.8 KB media82.1 KB media79.3 KB media117.2 KB media116.7 KB media89.4 KB media13.9 KB

中文

  • 生成式推荐器（Generative Recommenders，简称 GenRecs）代表了一种范式转变：从对每条广告进行打分，转向联合推理广告内容和用户偏好，从而为广告主预测最佳结果。利用大语言模型（LLM）内置的世界知识，广告匹配变得更加智能和精准，为广告主带来复合收益。

GenRecs 改进了当今 Meta 广告推荐系统进行预测的方式。生成式推荐器（Generative Recommenders）（简称 GenRecs）将我们的方法从对每个单独的广告候选进行打分，转变为利用 LLM 架构同时深度推理广告内容和用户偏好。通过将预训练的世界知识与对广告实际代表内容的语义理解相结合，该系统超越了点击模式匹配，去理解潜在的用户偏好。这使模型能够为每个人预测最相关的广告，即使是面对没有任何历史产品、广告或公司互动记录的全新产品也是如此。

GenRecs 分词（GenRecs Tokenization）是使这一切成为可能的基础数据表示。语言模型并不是为世界上的每个概念维护一个专用 ID，而是由一个由语义 token 组成的小型词表来表示任何概念。由于 token 描述了物品（例如广告、商品、服务等）是什么，因此模型可以将从未产生过互动记录的物品与它熟知的物品联系起来。一款新推出的“越野跑鞋”会直接继承模型在该品类中已经学到的知识，而不是从没有任何信号的新表示开始。

GenRecs 分词为广告推荐引入了一种结构化词表方法，使 GenRecs 能够应用于系统中包括广告、商品、主页和像素（pixel）在内的所有物品类型的召回和排序。

Meta 最前沿的广告推荐系统通过学习单个物品及其表现，持续推动广告主效果的提升。LLM 为在多个广告用例中利用语义分词进一步提高推荐系统的性能提供了独特的机遇：

  • 加速新实体的学习： 语义分词帮助模型推断相关性，并通过共享概念建立联系。新实体包括新创建的广告、长尾利基商品以及互动有限的主页。
  • 实现跨营销活动迁移： 语义分词创建了稳定的词表，允许模型在不同广告活动之间为广告主迁移见解与洞察。
  • LLM 扩展红利： 语义分词使收益能够与 LLM 缩放曲线（Scaling Curve）的性能提升相叠加。
  • 注入世界知识： 语义分词为预训练语言模型提供了一个直接的接口，以对广告内容和用户意图进行推理。

一旦物品迁移到语义 token，模型就会像 LLM 一样从 token 模式中学习。在生产环境中，这带来了：

  • 生产环境预测准确率提升： 结合 Meta 的 GEM 和 自适应排序模型，GenRecs 分词实现了 +0.35% 的转化率提升¹。
  • 复合反馈闭环： 更优的长尾预测能从长尾广告中获得更好的经验教训，从而丰富训练数据，进一步改善模型对所有广告的预测，形成性能提升的正向反馈闭环。
  • 全新的扩展方式： 在生产排序和召回中，token 沿着序列长度、训练数据和模型架构改变了扩展曲线。

¹ Meta 内部数据，统计于 2026 年 2 月。

GenRecs 分词技术简介

预训练的多模态模型读取每个广告、商品和主页（包括其文本和图像），并将这些物品编码为携带世界知识的嵌入向量（embedding）。分词器将该内容信号与从用户对物品互动中学到的表征相结合，然后将结果量化为离散 token 的短序列。相似的物品会共享 token，就像相关单词共享词根一样。例如，上一季到当季的两款跑鞋广告之间存在重叠，因此新营销活动立即可受益于模型从上一季产品中学到的所有知识。

GenRecs 分词的核心是一个三阶段流水线：

  • 编码（Encodes） 每个物品到语义嵌入空间中。
  • 将该嵌入 分解（Decomposes） 为独立的子空间。
  • 通过将每个子空间映射到码本（codebook）中的条目来进行 量化（Quantizes）。

根据实体和特征语义的不同，最大化广告推荐质量需要这些阶段的多样化组合。例如，对于许多广告表示，GenRecs 分词采用了前缀 n-gram，它结合了学习编码器、残差分解、学习向量码本以及token 参数化（token parametrization）。

Token 质量可预测排序准确率

独立于下游排序收益，分词器模型质量可以通过以下内在指标进行评估：

  • 聚类纯度（Cluster purity）： 共享同一个编码的物品往往属于同一类别，为模型提供了每个 token 更纯净的信号；
  • 分布均衡度（Distribution balance）： 物品均匀分布在语义 ID 之间，而不是拥挤在少数几个 ID 中。
  • 码本利用率（Codebook utilization）： 包含至少一个物品的编码比例更大，从而使词表能够表达更多差异化信息。

在我们测试的八个分词器版本中，我们观察到这些内在质量指标与下游排序预测准确率之间存在强相关性。在这些指标上得分更高的分词器版本，在广告模型的 CTR 预测准确率上也表现更优。

分词飞轮效应（The Tokenization Flywheel）

分词通过挖掘庞大商品目录中更深层次的语义模式，最大化了高曝光量营销活动的效率。对于曝光较少的长尾广告，分词使广告推荐模型能够通过共享的内容结构对其进行泛化，从而解决冷启动问题并实现跨实体迁移。当一个运动服装品牌推出下一季的越野跑鞋时，分词甚至在任何人看到新广告之前，就让新产品及其广告建立在模型从上一季跑鞋中学到的知识之上。

通过提高长尾广告的预测质量，分词不仅让长尾广告能够被精准排序，而且还改进了训练数据，进而提高所有广告的预测质量。每一次通过这个自我强化的飞轮，都能提升模型预测的质量。

LLM 推荐的基石

分词器的改进并不局限于为其调优的单一模型。通过多样性损失（diversity loss）、死码重置（dead code resets）和对比学习目标优化码本质量，在用生成的 token 训练生成式推荐器时，可产生显著更优的下游性能。我们在 LLM 原生召回（LLM-native retrieval） 方面的工作展示了这一方向的巨大潜力。基于共享 LLM 编码器构建的双塔架构达到了最前沿的召回质量，同时保持了高效的服务能力，仅使用一小部分训练数据即可达到经典双塔模型的性能。在此基础上，我们现在正在相同的词表之上扩展 具备推理能力的排序（reasoning-aware ranking）。

GenRecs 分词的未来

我们正沿着三个方向继续推动 GenRecs 分词向 ROI 最优前沿迈进：

深化冷启动泛化能力： 长尾广告库存是语义 token 贡献最大、对广告主影响最显著的领域。我们正在扩大分词对所有物品的覆盖范围，为广告主带来更出色的效果。

将分词器质量作为一等公民目标： 互动对齐训练（engagement-aligned training）、统一跨实体词表以及码本改进是当前大力投入的领域。由于内在质量指标可以预测下游排序收益，我们可以在不等待完整模型训练周期的情况下迭代分词器。

高效支持 LLM 原生架构推理： 完全语言化的分词（fully verbalized tokenization）为模型提供了最丰富的广告视角，但需要广告规模下无法落地的超大上下文窗口。我们将压缩后的语言化 token 与语义 ID 相结合，在保持可接受延迟的同时保留推理质量。

展望未来，我们正在探索今天用于改进广告排序的同一套词表是否能够统一跨版位的表示。一个涵盖文本、图像、音频和视频等广告内容的共享 token 空间，将使模型能够通过统一的表示来推理用户的兴趣所在以及广告主的供给内容，而不是将单独学习到的嵌入拼凑在一起。

随着推荐系统从判别式打分演变为包含生成式推理，数据底座也受益于从短暂的 ID 演变为稳定的语义表示。GenRecs 分词提供了这一基础，将 Meta 广告生态系统的复杂性转化为生成式模型可以学习、推理和生成的词表。

致谢

我们要感谢 Weifeng Cui、Ling Leng、Luke Simon、Hadar Ohana、Jimmy Chen、Minghai Chen、Chiyu Zhang、Dinesh Ramasamy、Debajyoti Roy、Shawn Shi、Gajanan Golegaonkar、Xiao Yang、Ashkan Aleali、Anuj Gargava、Xizhou Feng、Liquan Huang、Frank Shyu、Xiaoyu Chen、Kungang Li、Chonglin Sun、Matt Steiner、Sandeep Pandey、Santanu Kolay、Anuj Madan、Tak Yan，以及 Meta 广告推荐系统中首个大规模语义分词项目开发与生产化背后的整个团队。

English / 原文

  • Generative Recommenders (GenRecs) represent a paradigm shift from scoring every ad to reasoning about ad content and user preferences together in order to predict the best outcome for advertisers. Leveraging the world knowledge built into LLMs makes ad matching more intelligent and precise, compounding gains for advertisers.
    • GenRecs Tokenization is the foundational data representation that makes this possible. Rather than maintaining a dedicated ID for every item, GenRecs Tokenization represents each item with a set of tokens from a stable semantic vocabulary.
    • Paired with Meta’s Generative Ads Model (GEM) and Meta’s Adaptive Ranking Model, GenRecs Tokenization has delivered a +0.35% lift in conversions across surfaces for advertisers.

GenRecs improves the way Meta’s ads recommendation systems make predictions today. Short for Generative Recommenders, GenRecs shifts our approach from scoring every individual ad candidate to leveraging LLM architecture to deeply reason about ad content and user preferences simultaneously. By pairing pre-trained world knowledge with a semantic understanding of what an ad actually represents, the system moves beyond click-pattern matching to understand underlying user preferences. This allows the model to predict the most relevant ad for each person, even when surfacing brand-new products with no past product, ad or company engagement history.

GenRecs Tokenization is the foundational data representation that makes this possible. Instead of maintaining a dedicated ID for every concept in the world, language models compose a small vocabulary of semantic tokens to represent any concept. Because tokens describe what an item (e.g., ads, products, services, etc.) is, the model can relate an item it has never seen engagement for, to items it knows well. A newly launched “trail shoe” inherits what the model has already learned about that category, rather than starting a new representation with no signal.

GenRecs Tokenization introduces a structured vocabulary approach for ad recommendations, enabling GenRecs for both retrieval and ranking across every item type in the system, including ads, products, pages, and pixels.

Meta’s state-of-the-art ads recommendation system drives ongoing advertiser performance improvements by learning about individual items and how they perform . LLMs present a unique opportunity to further improve the performance of recommendation systems by leveraging semantic tokenization in several ads use cases:

  • Accelerating learning from new entities: Semantic tokenization helps models to infer relevance and establish relationships through shared concepts. New entities include newly created ads, niche products, and pages with limited engagement.
  • Enabling cross-campaign transfer: Semantic tokenization creates a stable vocabulary that allows the models to carry insights across campaigns for advertisers.
  • LLM scaling benefits: Semantic tokenization allows the benefits to compound with the LLM scaling curve’s improving performance.
  • Infusing world knowledge: Semantic tokenization gives pre-trained language models a direct interface to reason about ad content and user intent.

Once items are migrated to semantic tokens, models learn from token patterns much like LLMs do. In production, this yields:

  • Prediction accuracy improvements in production: Paired with Meta’s GEM and Adaptive Ranking Model, GenRecs Tokenization delivered a +0.35% lift1 in conversions.
  • Compounding feedback loop: Better tail predictions yield better lessons from tail ads which enrich the training data to further improve predictions for all ads from the model, creating a performance improving feedback loop.
  • A new way to scale: Across production ranking and retrieval, tokens bend the scaling curve along sequence length, training data, and model architecture.

1 Meta Internal Data, measured in February 2026.

Introducing GenRecs Tokenization

Pre-trained multimodal models read each ad, product, and page, including its text and images, and encode the items into an embedding that carries world knowledge. The tokenizer combines that content signal with representations learned from how people engage with the items, then quantizes the result into a short sequence of discrete tokens. Similar items share tokens, much as related words share roots. For example, there’s overlap between two running shoe ads from the previous season to the current season, so the new campaign immediately benefits from everything the model has already learned about the previous season’s products.

At the heart of GenRecs Tokenization is a three-stage pipeline that:

  • Encodes each item into a semantic embedding space.
  • Decomposes that embedding into independent subspaces.
  • Quantizes by mapping each subspace to an entry in a codebook.

Maximizing ads recommendation quality requires diverse combinations of these stages depending on entity and feature semantics. For example, for many ad representations, GenRecs Tokenization uses prefix n-grams that combine a learned encoder, residual decomposition, learned vector codebooks, and token parametrization.

Token Quality Predicts Ranking Accuracy

Independently of downstream ranking gain, tokenizer model quality can be evaluated against intrinsic measures like:

  • Cluster purity: The items that share a code tend to share a category, giving the model a cleaner signal per token;
  • Distribution balance: Items spread evenly across semantic IDs instead of crowding into a few.
  • Codebook utilization: A larger share of codes holds at least one item, so the vocabulary expresses more distinctions.

Across the eight tokenizer versions we tested, we see a strong correlation between these intrinsic quality metrics and downstream ranking prediction accuracy. Tokenizer versions that scored better on these metrics also had better CTR prediction accuracy for ads models.

The Tokenization Flywheel

Tokenization maximizes the efficiency of high-volume campaigns by uncovering deeper semantic patterns across their vast product catalogs. For tail ads that accumulate fewer impressions, tokenization enables ads recommendation models to generalize across them via shared content structure, recovering cold-start performance and enabling cross-entity transfer. When a sportswear brand launches next season’s trail shoe, tokenization lets the new product and its ads build on what the model learned from last season’s shoe, even before anyone sees the new ad.

By improving tail ad prediction quality, tokenization not only lets tail ads be ranked accurately, but also improves the training data, which then improves prediction quality for all ads. Each pass through this self-reinforcing flywheel improves the quality of model predictions.

The Foundations of LLM Recommendation

Tokenizer improvements are not confined to the model they were tuned for. Optimizing codebook quality through diversity loss, dead code resets, and contrastive objectives produces measurably better downstream performance when the resulting tokens train generative recommenders. Our work on LLM-native retrieval shows how far this can go. A two-tower architecture built on a shared LLM encoder reaches state-of-the-art retrieval quality while remaining efficiently servable, matching classic two-tower performance on a fraction of the training data. Building on this, we are now extending reasoning-aware ranking on top of the same vocabulary.

The Future of GenRecs Tokenization

We are continuing to push GenRecs Tokenization toward the ROI-optimal frontier along three directions:

Deepening cold-start generalization: Tail ad inventory is where semantic tokens contribute most and where advertiser impact is largest. We are scaling the coverage of tokenization across all items to deliver better performance to advertisers.

Treating tokenizer quality as a first-class objective: Engagement-aligned training, unified cross-entity vocabulary, and codebook improvements are active areas of investment. Because intrinsic quality measures predict downstream ranking gains, we can iterate on the tokenizer without waiting on a full model training cycle.

Serving LLM-native architectures efficiently: Fully verbalized tokenization gives a model the richest view of an ad, but requires context windows that are impractical at ads scale. We are combining compacted verbalized tokens with semantic IDs to retain reasoning quality while maintaining acceptable latency.

Looking further out, we are exploring whether the same vocabulary that improves ads ranking today could unify representation across surfaces. A shared token space spanning ads content across text, image, audio, and video, would let models reason about what a person is interested in and what an advertiser has to offer with one representation, rather than stitching together separately learned embeddings.

As recommendation systems evolve from discriminative scoring to include generative reasoning, the data foundation benefits from evolving from ephemeral IDs to stable semantic representations. GenRecs Tokenization provides the foundation, translating the complexity of Meta’s ads ecosystem into a vocabulary that generative models can learn, reason about, and generate from.

Acknowledgements

We would like to thank Weifeng Cui, Ling Leng, Luke Simon, Hadar Ohana, Jimmy Chen, Minghai Chen, Chiyu Zhang, Dinesh Ramasamy, Debajyoti Roy, Shawn Shi, Gajanan Golegaonkar, Xiao Yang, Ashkan Aleali, Anuj Gargava, Xizhou Feng, Liquan Huang, Frank Shyu, Xiaoyu Chen, Kungang Li, Chonglin Sun, Matt Steiner, Sandeep Pandey, Santanu Kolay, Anuj Madan, Tak Yan, and the entire team behind the development and productionization of the first large scale semantic tokenization effort in Meta’s ad recommendation system.