EmbeddingGemma 2：将多模态语义搜索带到边缘设备

今天，Google DeepMind 发布了 EmbeddingGemma 2：一款在同等规模中表现领先的开放权重多模态嵌入模型，可原生将文本、图像、视频帧和音频映射到单一、统一的向量空间中。对于构建本地搜索和媒体检索体验的开发者而言，它降低了串联使用独立图像描述、语音转文本和文本嵌入模型所带来的延迟与内存开销。EmbeddingGemma 2 专为本地、隐私优先的应用设计，在紧凑的 7.4 亿参数规模内覆盖文本、视觉和音频模态；其模块化编码器在 Google Pixel 11 Pro 上运行纯文本权重时仅需约 191MB 活跃 RAM，运行完整多模态模型时约需 567MB。尤其值得注意的是，EmbeddingGemma 2 可以充当超低延迟的端侧决策引擎：无需任何训练数据或微调，它就能将用户输入直接与分类标签和描述进行匹配，在数毫秒内实现即时的零样本意图路由。有关模型架构和评测（包括多模态嵌入能力）的概览，请参阅 Google DeepMind 博客。

本文将介绍如何立即通过 Google AI Edge 的交互式演示、移动端 Google AI Edge Gallery 和 Mac 端 Google AI Edge Foresight 体验 EmbeddingGemma 2 的能力；还将说明如何利用该模型构建私有的端侧语义搜索、视觉关键帧检索和条件触发工作流。在未来数周内，我们还将通过 ML Kit 在 Android 上以服务形式提供该模型，其中包括利用 NPU 加速，进一步优化其在广泛设备上的性能。

Google AI Edge Gallery 是一款跨平台演示应用，旨在让用户和开发者在实际场景中体验并评估端侧 AI 模型及其能力。今天，我们将在 Google AI Edge Gallery 应用中新增两个由 EmbeddingGemma 2 驱动的交互式演示：即时媒体搜索和视频精彩瞬间查找器。

即时媒体搜索让用户能够通过自然语言或示例图像，在本地媒体中查找特定图像或视频。该系统会将输入查询和媒体内容都转换为端侧嵌入向量——后者存储在本地 SQLite 数据库中——再通过返回余弦相似度最高的内容实现相关内容检索。这让用户能够在输入时获得即时、交互式的搜索体验。

抱歉，您的浏览器不支持播放此视频

边输边搜与以图搜图

• 端侧图像嵌入计算与快速多模态检索： 无论是从应用提供的示例图像集开始，还是使用自己的媒体测试即时搜索，您都可以体验 EmbeddingGemma 2 在手机本地执行嵌入计算的速度，且无需互联网连接。
• 多模态检索与相似度匹配： 由于查询 token 嵌入和余弦相似度排序通过 MediaPipe UniversalEmbedder 和 SemanticRetriever 在设备端快速执行，搜索结果会随每次按键实时更新。当您刚输入 “Katze”（德语中的 “Cat”，即“猫”）时，网格会立即显示收藏中的所有猫科动物照片；继续输入 “Katze schläft auf Tastatur”（德语中的 “Cat sleeping on keyboard”，即“睡在键盘上的猫”）时，系统会动态重新排序，并实时将您心中那张特定照片置于顶部。除多种语言的文本查询外，用户还可以从手机中选择任意图片，或使用实时摄像头画面来检索在视觉和语义上相关的照片。

视频精彩瞬间查找器展示了 EmbeddingGemma 2 跨模态表征的另一种应用方式：用户无需转录音频或生成中间文本描述，即可在本地视频录制内容中定位特定的视觉瞬间。

抱歉，您的浏览器不支持播放此视频

在本地视频文件中搜索特定瞬间

• 端侧视频关键帧索引与自然语言搜索： 用户选择本地视频后（例如家庭视频或体育集锦），端侧引擎会为视觉内容及音频片段建立索引并生成嵌入。当用户输入诸如 “kids laughing”（“孩子们在笑”）、“dog catching a frisbee”（“狗接住飞盘”）或 “person blowing out birthday candles”（“有人吹灭生日蜡烛”）等描述性查询时，引擎会计算提示词嵌入向量，将其与视频帧嵌入进行比较，并立即突出显示匹配瞬间所在的准确时间戳。

从 Google Play Store 或 Apple App Store 下载最新版 Google AI Edge Gallery 应用，试用这些新演示。查看 Github 仓库，了解其实现方式，并开始使用 EmbeddingGemma 2 构建自己的体验。

Mac 版 Google AI Edge Foresight：提升生产力与上下文检索

我们很高兴宣布推出 Mac 版 Google AI Edge Foresight。这是一款实验性应用，可将具备上下文感知能力的会议助手直接带到您的设备上。Foresight 完全由本地 AI 处理驱动，可无缝协助记笔记、索引和调用您的对话转录稿及私有文件。它可直接集成系统音频和麦克风，因此开箱即用地兼容任何会议平台，即使完全离线也能使用。

抱歉，您的浏览器不支持播放此视频

自动完善速记笔记

抱歉，您的浏览器不支持播放此视频

AI 实时检测并回答问题

Foresight 由 EmbeddingGemma 2 和 Gemma 4 模型驱动，提供以下能力：

• 增强记笔记： 利用从当前会议对话中实时检索到的细节，丰富手动速记笔记。
• 跨模态检索： 通过自然语言轻松查找不同类型的媒体内容，包括图像、文档、转录稿和笔记。
• 端侧优势： 您的敏感信息始终安全；无论网络连接如何，生产力都不会中断；同时无需承担云端订阅费用。

通过将文本、音频和视觉数据映射到统一向量空间，Foresight 可在您的个人知识库中执行本地、安全的搜索。无论您是在会议中实时回答问题，还是撰写详尽笔记，所需细节始终触手可及，而您的敏感数据绝不会离开设备。

立即下载 Google AI Edge Foresight Mac 应用，为所有会议配备智能的本地助手。

使用 ML Kit 为 Android 开发

对于面向 Android 生态系统的开发者，EmbeddingGemma 2 是一款轻量级、任务专用模型，非常适合在设备端执行。未来数周内，我们将把这些统一搜索能力引入 ML Kit，为未来的生产级 Android 应用提供一条标准化、平台集成的接入路径。通过集成 ML Kit，开发者可以开箱即用地获得 EmbeddingGemma 的生产级质量，无需自行管理模型，也不必担心 APK 体积膨胀。ML Kit 将通过 NPU 加速（如可用）和自动模型更新提供极快性能。

使用 MediaPipe Tasks 进行跨平台开发

要集成与上述 Google AI Edge Gallery 和 Foresight 类似的本地搜索功能，开发者通常需要处理底层媒体预处理，以便将数据送入嵌入模型。为简化这一流程，MediaPipe Tasks 正在为现有的 Embedder 和 Semantic Retriever 增加对 EmbeddingGemma 2 的支持；后者是我们现有 RAG SDK 的新接口。

无论您的目标平台是 iOS、macOS、Windows、Linux 还是 Web，MediaPipe Tasks 都提供统一的开发体验，让您一次编写、部署至所有平台。开发者无需手动管理预处理和数据转换，Embedder 和 Semantic Retriever Tasks 提供了一套可直接使用的解决方案，完全抽象掉这些步骤。

• MediaPipe Universal Embedder Task： 自动处理图像缩放、张量归一化和多模态 token 化。直接传入原始图像或文本字符串，即可获得 768 维（或截断为 128d–512d）的归一化向量，无需编写样板代码。
• MediaPipe Semantic Retriever Task： 这是一款面向边缘设备优化的嵌入式向量搜索引擎，可在设备端为嵌入建立索引，并快速运行近似最近邻（ANN）搜索，在个位数毫秒内返回排序后的匹配结果。

EmbeddingGemma 2 还可通过 MediaPipe Decision Task 用作多模态实时决策引擎。这使开发者能够无需微调，轻松对图像、文本或音频输入进行分类，并在设备端即时作出决策，从而支持路由或动作预测等应用场景。下面的示例展示了 MediaPipe Decision Task 在国际象棋游戏中运行时，每回合评估 500 个选项的响应时间低于 100ms，体现了 EmbeddingGemma 2 作为完全端侧决策引擎的效率。

抱歉，您的浏览器不支持播放此视频

EmbeddingGemma 2 为端侧低延迟 MediaPipe Decision Task 提供支持

使用 LiteRT 和 EmbeddingGemma 2 构建跨平台应用

MediaPipe Tasks 提供了开箱即用的简便性；而对于需要精细控制处理过程和加速方式的跨平台应用，则可以直接基于 LiteRT——为 ML Kit、MediaPipe Tasks、Google AI Edge Gallery 和 Foresight 提供支持的成熟引擎——进行构建。借助在 CPU、GPU 和 NPU 间的统一加速，部署因卓越的可移植性而大幅简化：单个 **.litertlm**文件可在所有受支持的边缘平台上，开箱即用于 CPU 和 GPU。

在 Android 上，即将推出的 ML Kit API 将利用 LiteRT 为平台提供优化实现，让您无需担心模型生命周期管理，即可同时获得性能与简便性。

为实现 EmbeddingGemma 2 的最佳推理速度，我们针对视觉编码等计算密集型工作负载实施了专门优化。这解锁了高度灵敏的实时 AI 体验，例如 Google AI Edge Gallery 应用中的“边输边搜”：在 MacBook M5 Pro GPU 上，视觉嵌入耗时最低仅为 37.3 ms（每秒 26.9 张图像）。更多测量结果请参阅下表。

表：在一组具有代表性的设备上，EmbeddingGemma 2 视觉功能在 CPU、GPU 和 NPU 上的单张图像延迟。延迟列指视觉嵌入延迟，每张图像以最多 70 个视觉 token 的预算进行评估。

底层架构优化

要在内存受限的边缘设备上实现 EmbeddingGemma 2 的高性能执行，依赖于以下优化：

• 跨模态模块化： 仅加载所需编码器（例如，用于照片和视频搜索的“文本 + 视觉”编码器；或者在添加语音备忘录时动态激活音频编码器），以最大化边缘效率并管理系统内存压力。
• 量化感知训练（QAT）： 将模型权重压缩为 INT4 和 INT8，使中端消费级硬件也能够运行多模态向量搜索。
• 自适应输入与输出缩放： 根据文本和图像输入大小，我们会在运行时动态加载最优计算图，并在所有 CPU、GPU 和 NPU 后端上保持一致支持。这一设计可在不浪费计算资源的前提下适应可变文本长度，同时让开发者控制视觉 token 预算，以实现延迟和精度之间的理想平衡。对于输出，内置的 Matryoshka Representation Learning（MRL）切片支持即时截断向量维度，最多可将本地存储和索引占用缩减 8 倍。

LiteRT 在广泛的边缘设备和 **Web（WASM）**上提供稳健的跨平台性能。有关受支持硬件平台的完整列表和最新性能基准，请参阅我们的 Hugging Face 模型卡。您可以在 Google Cloud 上通过 LiteRT benchmark 工具使用更多实体设备进行测试，或查看我们的[免配置网页演示](https://google-ai-edge.github.io/LiteRT-LM/web_demos/em

Today, Google DeepMind launched EmbeddingGemma 2 , a best-in-class for its size, open-weight, multimodal embedding model that natively maps text, images, video frames, and audio into a single, unified vector space. For developers building local search and media retrieval experiences, it reduces the latency and memory overhead of chaining separate image captioning, speech-to-text, and text-embedding models. Designed specifically for local, privacy-first applications, EmbeddingGemma 2 covers text, vision, and audio modalities in a compact 740M parameter footprint, offering modular encoders that can run on as little as ~191MB active RAM for text-only weights and ~567MB for the full multimodal model on a Google Pixel 11 Pro. Most notably, EmbeddingGemma 2 can act as an ultra-low-latency on-device decision engine. Without requiring any training data or fine-tuning, it matches user inputs directly against classification labels and descriptions, delivering instant, zero-shot intent routing in a matter of milliseconds. For an overview of the model architecture and evals including multimodal embedding capabilities, check out the Google DeepMind blog.

In this post, we’ll share how you can experience the power of EmbeddingGemma 2 right away through Google AI Edge’s interactive showcases, Google AI Edge Gallery on mobile and Google AI Edge Foresight on Mac, and how you can leverage the model to build your own private, on-device semantic search, visual keyframe retrieval, and condition-trigger workflows. In the coming weeks, we will also make this model available as a service on Android, accessible through ML Kit, including using NPU acceleration to further optimize performance across a broad range of devices.

Google AI Edge Gallery is a cross-platform showcase application designed to allow users and developers to experience and evaluate on-device AI models and their capabilities in practical settings. Today, we are adding two interactive showcases powered by EmbeddingGemma 2 to the Google AI Edge Gallery app: Instant Media Search and Video Moments Finder.

Instant Media Search allows users to find specific images or videos in their local media using natural language or example images. By converting both input query and the media into on-device embedding vectors - with the latter stored in a local SQLite database - the system retrieves relevant content by returning those with the largest cosine similarity. This enables an instant, interactive search experience as the user types.

Sorry, your browser doesn't support playback for this video

Search-as-you-type and image-to-image retrieval

• On-device image embedding calculation and fast multimodal retrieval: Whether you are starting with the sample image set provided by the app or testing instant search on your own media, you can experience how quickly EmbeddingGemma 2 executes embedding computations locally on your phone, no internet connection required.
• Multimodal retrieval and similarity matching: Because query token embedding and cosine similarity ranking execute quickly on device using MediaPipe UniversalEmbedder and SemanticRetriever, search results update live on every keystroke. As soon as you type "Katze" ("Cat" in German), the grid immediately displays all feline photos in your collection; continuing to type "Katze schläft auf Tastatur" ("Cat sleeping on keyboard" in German) dynamically re-ranks and brings the specific photo you have in mind to the top in real time. Besides text queries in multiple languages, users can also select any image from their phone or use their live camera stream to retrieve visually and semantically related photos.

Video Moments Finder demonstrates another example of how EmbeddingGemma 2's cross-modal representations can be applied, enabling users to locate specific visual moments across local video recordings without transcribing audio or generating intermediate text captions.

Sorry, your browser doesn't support playback for this video

Search local video files for specific moments

• On-device video keyframe indexing and natural language search: After users select a local video (such as a family video or sports reel), the on-device engine indexes vision with audio chunks and generates embeddings for them. When users enter descriptive queries, such as "kids laughing" , "dog catching a frisbee" , or "person blowing out birthday candles," the engine computes the prompt embedding vector, compares it against the video frame embedding, and instantly highlights the exact timestamps where matching moments occur.

Download the latest version of Google AI Edge Gallery app from Google Play Store or Apple App Store to try these new demos. Check-out the Github repo to learn how it’s implemented and start building your own experiences using EmbeddingGemma 2.

Productivity boost and context retrieval in Google AI Edge Foresight on Mac

We are excited to announce the launch of Google AI Edge Foresight for Mac, an experimental app that brings a context-aware meeting companion directly to your device. Powered by fully local AI processing, Foresight seamlessly assists with note-taking, indexing, and recalling your conversation transcripts and private files. By integrating directly with your system audio and microphone, it works out-of-the-box with any meeting platform, even completely offline.

Sorry, your browser doesn't support playback for this video

Automatic enhancement of shorthand notes

Sorry, your browser doesn't support playback for this video

Questions detected and answered live by AI

Powered by EmbeddingGemma 2 and Gemma 4 models, Foresight delivers:

• Enhanced Note-taking: Enriches manual shorthand notes with details retrieved in real-time from your current meeting conversation.
• Cross-Modal Retrieval: Easily find different types of media (images, documents, transcripts, and notes) using natural language.
• On-device Benefits: Your sensitive information remains secure, along with uninterrupted productivity regardless of your network connection, and with no cloud subscription costs.

By mapping text, audio, and visual data into a single vector space, Foresight runs local, secure searches across your personal knowledge library. The details you need are always at your fingertips, whether you are answering live questions mid-meeting or drafting thorough notes, while your sensitive data never leaves your device.

Download the Google AI Edge Foresight Mac app today, and bring an intelligent, local companion to all your meetings.

Building For Android with ML Kit

For developers building for the Android ecosystem, EmbeddingGemma 2 is a lightweight, task-specific model perfectly suited for on-device execution. We are bringing these unified search capabilities to ML Kit in the coming weeks, providing a standardized, platform-integrated path for production Android apps in the future. By integrating with ML Kit developers can get production-grade quality for EmbeddingGemma out of the box without having to manage the model or worrying about APK bloat. ML Kit will feature blazing fast performance through NPU acceleration (when available) and automatic model updates.

Building For X-Platform with MediaPipe Tasks

Integrating local search features, similar to those in Google AI Edge Gallery and Foresight above, usually requires developers to handle low-level media pre-processing for the data to be fed into the embedding model. To simplify this, MediaPipe Tasks is adding support for EmbeddingGemma 2 to the existing Embedder and Semantic Retriever; the new interface to our current RAG SDK.

Whether you are targeting iOS, macOS, Windows, Linux or Web, MediaPipe Tasks provide a unified developer experience to write once and deploy across all platforms. Instead of developers manually managing pre-processing and data transformations, the Embedder and Semantic Retriever Tasks provide a turnkey solution that completely abstracts these steps.

• MediaPipe Universal Embedder Task: Automatically handles image resizing, tensor normalization, and multimodal tokenization. Pass raw images or text strings directly to receive 768-dimensional (or truncated 128d–512d) normalized vectors with zero boilerplate.
• MediaPipe Semantic Retriever Task: An edge-optimized embedded vector search engine indexes embeddings on-device and runs fast Approximate Nearest Neighbour (ANN) searches, returning ranked matches in single-digit milliseconds.

EmbeddingGemma 2 can also be used as a multimodal, real-time decision engine with the MediaPipe Decision Task. This allows developers to effortlessly classify image, text or audio inputs without the need for fine-tuning and make instant decisions on-device, enabling use cases like routing or action prediction. The example below shows the MediaPipe Decision task running with less than 100ms response time to evaluate 500 options each turn in the chess game, showcasing the efficiency of EmbeddingGemma 2 as the decision engine fully on-device.

Sorry, your browser doesn't support playback for this video

EmbeddingGemma 2 powers low-latency MediaPipe Decision task on-device

Building cross-platform apps using EmbeddingGemma 2 with LiteRT

While MediaPipe Tasks offers turnkey simplicity, cross-platform applications requiring fine-grained control over processing and acceleration can be built directly on LiteRT, the proven engine powering ML Kit, MediaPipe Tasks, Google AI Edge Gallery, and Foresight. Powered by unified acceleration across CPUs, GPUs, and NPUs, deployment is greatly simplified with exceptional portability: a single **.litertlm**file runs out of the box on both CPU and GPU across all supported edge platforms.

On Android, ML Kit APIs (coming soon) leverage LiteRT to provide optimized implementation for the platform, giving you both performance and simplicity without having to worry about model lifecycle management.

To provide the best inference speed for EmbeddingGemma 2, we implemented targeted optimizations for compute-intensive workloads like vision encoding. This unlocks highly responsive, real-time AI experiences, as “search-as-you-type” in the Google AI Edge Gallery app, with visual embeddings taking as little as 37.3 ms (26.9 images per sec) on a MacBook M5 Pro GPU. See table for additional measurements.

Table: Per image EmbeddingGemma 2 vision latency across CPU, GPU and NPU for a representative set of devices. Latency column specifies the vision embedding latency, evaluated with a maximum budget of 70 vision tokens per image.

Architectural optimizations under the hood

Achieving high-performance execution for EmbeddingGemma 2 on memory-constrained edge devices is enabled by the following optimizations:

• Modularity across modalities: Load only the encoders you need (e.g., Text + Vision for photo and video search, or dynamically activate the audio encoder when voice memos are added) to maximize edge efficiency and manage system memory pressure.
• Quantization-Aware Training (QAT): Compresses model weights to INT4 and INT8 , bringing multimodal vector search within reach of mid-range consumer hardware.
• Adaptive input and output scaling : Given the text and image input size, we dynamically load the optimal computational graph at runtime, consistently supported across all CPU, GPU, and NPU backends. This design accommodates variable text length without wasted compute while empowering developers to control vision token budgets for the ideal balance between latency and precision. For output, built-in Matryoshka Representation Learning (MRL) slicing allows on-the-fly dimension truncation, cutting local storage and index footprints by up to 8x.

LiteRT delivers robust, cross-platform performance across a broad spectrum of edge devices andweb (WASM). For a complete list of supported hardware platforms and the latest performance benchmarks, see our Hugging Face model card. You can try the LiteRT benchmark tool with more physical devices on Google Cloud or check out our zero-setup web demo for an instant, browser-based playground.

Get started today

EmbeddingGemma 2 and Google AI Edge provide the building blocks to bring fast, private, and offline multimodal retrieval directly into your applications. Whether you’re experimenting with instant image search, visual keyframe retrieval, or ambient condition triggers, all the tools you need are available today.

Acknowledgements

We'd like to extend a special thanks to our significant contributors for their work on this project:

Abhishek Jatram, Aditya Srivastava, Akshat Sharma, Alexander Kanaukou, Alice Zheng, Ami Kubota, Ander Dobo, Andrew Zhang, Brad Lassey, Chanchal Raj, Chandramouli Amarnath, Charlie Xu, Chenchen Tang, Chintan Parikh, Chirag Gupta, Cormac Brick, David Chou, Debapriya Maji, Denis Daletski, Fengwu Yao, Florian Kübler, Geonsun Lee, Gregory Karpiak, Henrique Schechter Vera, Himangshu Roy, Hriday Chhabria, Ian Ballantyne, Ivan Llanos, Jae Yoo, Jenn Lee, Jianing Wei, Jing Jin, Jingjiang Li, Jingtao Zhou, Jingxiao Zheng, Juhyun Lee, Julius Kammerl, Karthik Thirumalai, Kat Black, Kristen Quan, Kristen Wright, Lu Wang, Lutz Justen, Malini PV, Marissa Ikonomidis, Matthew Chan, Matthew Soulanille, Matthias Grundmann, Mogan Shieh, Na Li, Naina Singla, Olivier Lacombe, Priya Patel, Qidong Zhao, Queenie Zhang, Renjie Wu, Rishika Sinha, Rishubh Khurana, Ronald Wotzlaw, Sachin Kotwani, Sahil Dua, Sandeep Patil, Sebastian Russo, Sebastian Schmidt, Shengyi Lin, Shuangfeng Li, Somdatta Banerjee, Steven Toribio, Suril Shah, Sylvain Vignaud, Tenghui Zhu, Vinod Mamillapalli, Vladimir Kirilyuk, Wai Hon Law, Weiyi Wang, Xiaoming Hu, Xinan Cheng, Xu Chen, Yi-Chun Kuo, Yishuang Pang, Yu-hui Chen.