继我们在 发布博客 中宣布后，我们正式发布 Gemma 4 12B，一个稠密多模态模型，采用 统一的无编码器架构。

Gemma 4 12B 为本地 AI 带来了数个里程碑：

1. 多模态无编码器架构： 完全跳过繁重的多阶段视觉和音频编码器，多模态数据直接馈入 LLM 主干，降低多模态延迟。
2. 我们首个支持音频输入的中型模型： 在 Gemma 家族中，音频输入此前仅限于小型轻量边缘架构（例如 E4B）。Gemma 4 12B 是第一款能够原生摄入音频的中型模型。
3. 开发者友好尺寸：足够小，可在配备 16GB VRAM 或统一内存的专用 GPU 笔记本上本地运行。为最大化本地推理速度，我们还发布了一个专用的 多令牌预测（MTP） 模型。
4. 全新 MacOS 桌面体验：我们首次发布可下载的 macOS 桌面应用，让开发者在消费级设备上直接体验完全本地化的语音和视觉交互。

架构

传统多模态模型依赖于冻结的独立视觉编码器（例如，Gemma 4 边缘尺寸使用 1.5 亿参数视觉模型，中型尺寸使用 5.5 亿参数）和音频编码器（Gemma 4 E2B 与 E4B 为 3 亿参数）。在馈入 LLM 之前使用多个独立编码器处理多模态输入会导致延迟增加和内存占用碎片化。

Gemma 4 12B 通过使用单个解码器专用 Transformer 解决了这些问题，该 Transformer 包含与 Gemma 4 31B Dense 模型相同的先进解码器结构。

• 视觉嵌入器（35M 参数）： 替代其他中型 Gemma 4 模型的 27 层视觉 Transformer 层。原始 48×48 像素块通过单一矩阵乘法投影到 LLM 隐藏维度。因式分解坐标查找（X 和 Y 矩阵）将空间位置信息直接附加到输入。
• 音频波投影： 消除了单独的音频编码器（跳过 Gemma 4 E2B 和 E4B 中使用的 12 层 conformer 层）。原始 16 kHz 音频信号被切片为 40ms 帧（每帧 640 个浮点数），并线性投影到 LLM 输入空间。
• 统一微调优势： 由于视觉、音频和文本输入共享完全相同的权重，您不再需要联合调整独立的冻结编码器。下游适配器（如 LoRA）或全量微调自然地在一次传递中更新整个多模态令牌循环（通过 Hugging Face 或 Unsloth）。

有关该无编码器架构工作原理的更深入概述，请查看 Gemma 4 12B 视觉指南。

能力

Gemma 4 12B 取得了卓越的性能，具备自动语音识别、智能体推理、说话人分离、视频理解、编码等能力。

以下示例展示了该模型的智能体和多模态能力：

示例 1：Gemma 4 12B 创建一个使用自身处理图像的本地应用

凭借其智能体与多模态理解能力，Gemma 4 12B 可以轻松搭载现有的智能体框架，例如 OpenCode。本示例中，我们使用 llama.cpp 在本地提供了该模型，并利用 gemma-skills 编码了一个 Gradio 应用，帮助用户处理图像。该应用正是由构建它的同一个 Gemma 4 12B 模型驱动的！

示例 2：以 1 FPS 处理含音频的 5 分钟视频

我们使用 Gemma 4 12B 分析了 Google IO 主题演讲片段（5 月 19 日），具体为 00:15:32 到 00:20:45 的 5 分钟。为此，我们提取了该片段的所有帧（1 FPS）、提示词以及视频中的音频：

提示词：

1. 313 帧（1FPS，图像缩放到 视觉令牌预算 70）
2. "what happens when the man takes a selfie?"
3. 视频音频及下方的提示词

当视频中的男子“自拍”或展示将智能手机举在面前时，这是一种巧妙的方式，用来形象地表现 AI 模型如何能够接收现有媒体——例如个人照片或视频片段——并“重新构想”它。在这些特定片段中，模型拍摄一张自拍照并将其作为基础，生成各种场景（如空间站中的人或穿越森林的人）。

本质上，该男子并不是真的在自拍；而是表演一个视觉隐喻，展示 AI 能够接收特定输入（“自拍照”）并基于它生成全新内容世界的能力。 这是 Gemini Omni 模型“交换”与“构建世界”演示的一部分，展现了其进行复杂、多模态推理和创意生成的能力。

本地与桌面服务：由 LiteRT-LM 驱动

与 Gemma 4 12B 发布同步，我们正式推出由 LiteRT-LM 驱动的强大本地设备开发者集成，将零延迟的本地 AI 执行原生带入标准桌面环境：

1. 原生 MacOS 应用：移动版 Google AI Edge Gallery 正式扩展到桌面平台，可在 Apple Silicon GPU 上离线、原生运行 Gemma 4 12B。它包含一个安全的沙盒化 Python 执行循环，能够在聊天气泡中编写、执行并绘制科学图表。与此同时，Mac 上的 Google AI Edge Eloquent 应用也上线了对 Gemma 12B 的支持，用于驱动语音编辑对话输入。

2. 即插即用的本地 API 服务器（litert-lm serve）： 使用新的 litert-lm serve CLI 命令 将 Gemma 4 12B 运行为本地、兼容 OpenAI 的 API 服务器。无缝对接标准集成（例如 Continue、Aider、OpenClaw、Hermes 或 OpenCode），利用内存中的无状态前缀缓存匹配上下文历史，即时跳过预填充延迟。

Shell litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm gemma-4-12B-it.litertlm gemma4-12b

Start the OpenAI-compatible server

litert-lm serve

在 Google AI Edge Gallery 博客 中可以找到相关深入解读。

立即开始

准备好使用 Gemma 家族首个无编码器架构构建本地多模态智能体了吗？以下是今天的上手方式：

• 亲自尝试：在 LM Studio、Ollama、Google AI Edge Gallery 应用、Google AI Edge Eloquent 应用以及 LiteRT-LM CLI 中点击几次即可体验。
• 下载权重：直接从 Hugging Face 和 Kaggle 下载预训练和指令微调检查点。
• 集成与学习：查阅 开发者文档 和 快速开始笔记本。
• 使用您喜爱的开发工具：使用 Hugging Face Transformers、llama.cpp、MLX、SGLang 和 vLLM 实现本地推理流水线，或使用 Unsloth 高效微调。
• 利用 Gemma Skills 解锁智能体开发：为支持智能体利用最新的 Gemma 进展进行构建，我们发布了官方 Skills 仓库。这是一个专门设计的技能库，旨在让智能体能够使用 Gemma 模型进行构建。
• 按您的方式部署：使用 Google Cloud 在生产环境中启动端点。通过 Gemini Enterprise Agent Platform Model Garden、Cloud Run 和 GKE 按需部署。

Following the announcement in our launch blog, we are releasing Gemma 4 12B , a dense multimodal model with a unified, encoder-free architecture.

Gemma 4 12B introduces several milestones for local AI:

1. A multimodal encoder-free architecture: Bypassing heavy multi-stage vision and audio encoders entirely, multimodal data is fed straight into the LLM backbone, reducing multimodal latency.
2. Our first medium-sized model with audio input: In the Gemma family, audio inputs were restricted to small, lightweight edge architectures (e.g. E4B). Gemma 4 12B is the first medium-sized model capable of natively ingesting audio.
3. Developer-friendly size : Small enough to run locally on dedicated GPU laptops with 16GB VRAM or unified memory. To maximize local inference speeds, we are additionally releasing a dedicated multi-token prediction (MTP) model.
4. New MacOS desktop experience : For the first time, we are releasing downloadable macOS desktop applications, letting developers experience fully local spoken and visual interaction directly on consumer-grade devices.

The Architecture

Traditional multimodal models rely on frozen, separate vision encoders (e.g., Gemma 4 uses a 150M parameter vision model for edge sizes and 550M for medium-sized models) and audio encoders (300M parameters for Gemma 4 E2B and E4B). Processing multimodal inputs with multiple separate encoders before feeding them to the LLM leads to increased latency and fragmented memory footprints.

Gemma 4 12B solves these issues by utilizing a single decoder-only transformer containing the same advanced decoder structure as the Gemma 4 31B Dense model.

• Vision embedder (35M parameters): Replaces the 27 vision transformer layers of the other medium-sized Gemma 4 models. Raw 48x48 pixel patches are projected to the LLM hidden dimension with a single matmul. A factorized coordinate lookup (X and Y matrices) attaches spatial location information directly to the input.
• Audio wave projection: Eliminates the separate audio encoder (skipping the 12 conformer layers used in Gemma 4 E2B and E4B). Raw 16 kHz audio signals are sliced into 40ms frames (640 floats each) and projected linearly to the LLM input space.
• Unified fine-tuning advantage: Because vision, audio, and text inputs share the exact same weights, you no longer have to co-tune separate frozen encoders. Downstream adapter (e.g. LoRA) or full tuning naturally update the entire multimodal token loop in a single pass (via Hugging Face or Unsloth).

For a more in-depth overview of how this encoder-free architecture works, check out A Visual Guide to Gemma 4 12B.

Capabilities

Gemma 4 12B achieves outstanding performance, with capabilities such as automatic speech recognition, agentic reasoning, diarization, video understanding, coding, and more.

See below examples for a demonstration of the model's agentic and multimodal capabilities:

Example 1: Gemma 4 12B creates a local image processing app that uses Gemma 4 12B

Thanks to its agentic and multimodal understanding capabilities, Gemma 4 12B can be easily used with existing agent harnesses like OpenCode. In this example, we served it locally using llama.cpp using the gemma-skills to code a Gradio app that helped the user process images. This app was powered by the same Gemma 4 12B model that built it!

Example 2: Processing 5 minutes of Video at 1 FPS with audio

We used Gemma 4 12B to analyse a part from the Google IO Keynote moment from May 19, specifically the 5 minutes between 00:15:32 and 00:20:45. To do that, we extracted all the frames of that segment (at 1 FPS), plus the prompt and the audio from the video:

Prompt:

1. 313 frames (at 1FPS, images resized to visual token budget 70)
2. "what happens when the man takes a selfie?"
3. the Video's audio + the prompt below

When the man takes a "selfie" or is shown holding a smartphone in front of his face in these demonstration videos, it is a clever way to visually represent how the AI models can take existing media—like a personal photo or a video clip—and "reimagine" it. In these specific clips, the model is taking a selfie and using it as a base to generate various scenarios (like a person in a space station, or walking through a forest).

Essentially, the man is not actually taking a selfie; rather, he is acting out a visual metaphor for the AI's capability to take one specific input (a "selfie") and generate a whole world of new content based on it. This is part of the "Swap" and "Build worlds" demonstrations of the Gemini Omni model, showing its ability to perform complex, multi-modal reasoning and creative generation.

On-Device & Desktop Serving: Powered by LiteRT-LM

In tandem with the Gemma 4 12B launch, we are officially introducing powerful on-device developer integrations powered by LiteRT-LM, bringing zero-latency local AI execution natively to standard desktop environments:

1.Native MacOS Apps : The mobile Google AI Edge Gallery is officially expanding to desktop platforms, running Gemma 4 12B offline, natively on Apple Silicon GPUs. It comes with a secure sandboxed Python execution loop to write, execute, and plot scientific charts inside the chat bubble. In parallel, the Google AI Edge Eloquent app on Mac launches support for Gemma 12B to power Voice Edit conversational inputs.

2. Drop-in Local API Servers (litert-lm serve): Run Gemma 4 12B as a local, OpenAI-compatible API server using the new litert-lm serve CLI command. Seamlessly connect standard integrations (e.g., Continue, Aider, OpenClaw, Hermes or OpenCode), leveraging stateless prefix caching in memory to match context history and instantly bypass prefill latency.

litert-lm import --from-huggingface-repo=litert-community/gemma-4-12B-it-litert-lm  gemma-4-12B-it.litertlm gemma4-12b

# Start the OpenAI-compatible server
litert-lm serve

Shell

Find a deep dive about it on the Google AI Edge Gallery blog.

Getting Started Today

Ready to build local multimodal agents with the first encoder-free architecture of the Gemma family? Here is how you can jump in today

• Try it yourself : Experiment with a couple of clicks in LM Studio, Ollama, Google AI Edge Gallery App, the Google AI Edge Eloquent app and the LiteRT-LM CLI
• Download the weights : Download the pre-trained and instruction-tuned checkpoints directly from Hugging Face and Kaggle.
• Integrate & learn: Review the developer documentation and the quick start notebook.
• Use your favorite development tools : Implement local inference pipelines with Hugging Face Transformers, llama.cpp, MLX, SGLang, and vLLM, or fine-tune with efficiency using Unsloth.
• Unlock Agentic Development with Gemma Skills: To support agents to build with the latest Gemma advancements, we are releasing our official Skills Repository. This is a library of skills designed specifically to enable agents to build with Gemma models.
• Deploy your way: Spin up endpoints in production using Google Cloud. Deploy your way through Gemini Enterprise Agent Platform Model Garden, Cloud Run and GKE.