Meta 携 Muse Glimmer 回归：本地化、Agent 化、多模态且开源！

来自开源 LLM 元老团队的好消息！今日发布的 Muse Glimmer 是 Meta 的全新多模态模型，专为本地 Agent 使用场景设计。它由 Muse 蒸馏而来，参数量缩减至 300 亿，并以 Apache 2.0 许可证发布；无论是为保护隐私进行本地部署、降低成本，还是单纯用于探索实验，它都很合适。其目标应用包括编程、文档分析、个人助手，以及类似 Claw 或 Hermes 的、注重隐私的应用配置。

为庆祝发布，我们已在 transformers、llama.cpp、vLLM、Inference Endpoints 及其他库中提供 Meta 发布首日支持。我们构建了一些很酷的项目，并会在本文中说明相关发现。**请查看下方演示以获取灵感。**你可以在 Hugging Face Hub 上找到 Muse Glimmer。

基准测试

基准测试结果

分数按已发布结果报告。粗体表示比较模型中的最佳结果；↓ 表示越低越好。

类别	基准测试	Muse Glimmer-30B	Gemma4-31B 推理模式	Qwen3.6-27B 推理模式
通用 Agent	MCP Atlas	75.5	54.2	62.5
通用 Agent	DeepSearch QA	74.6	61.7	71.1
通用 Agent	τ³-Banking	23.5	15.1	16.7
通用 Agent	WildClawBench	47.6	37.6	43.2
通用 Agent	GDPval-AA	953	811	1141
通用 Agent	GAIA2	43.3	36.4	40.0
通用 Agent	SkillsBench（使用 Skills）	44.3	32.4	46.6
通用 Agent	OSWorld-Verified	65.9	58.5	75.6
Agent 编程	SWE-Bench Pro	51.2	36.9	50.2
Agent 编程	SWE-Bench Verified	76.0	66.6	77.2
Agent 编程	TerminalBench 2.1	51.7	43.4	60.7
Agent 编程	SciCode	43.6	43.4	39.8
多模态	Charxiv Reasoning	78.8	77.7	78.4
多模态	ScreenSpot Pro	75.4	75.9	76.1
多模态	OmniDocBench v1.5	75.8	72.5	77.8
多模态	MMMU Pro	74	73	75
安全	CI Memories	违规（↓）：26.4覆盖率：64.8	违规（↓）：12.1覆盖率：53.0	违规（↓）：53.4覆盖率：66.9
安全	Siren AgentDojo	攻击成功率（↓）：28.4效用：94.2	攻击成功率（↓）：25.6效用：90.8	攻击成功率（↓）：40.3效用：92.7
通用能力与推理	IFBench	77.0	76.0	70.8
通用能力与推理	AIME 2026	94.7	89.2	94.1
通用能力与推理	GPQA Diamond	83.5	85.7	84.2
通用能力与推理	Humanity’s Last Exam（纯文本 + 无工具）	22.0	23.6	23.1
通用能力与推理	AA-LCR	80.0	68.3	73.3
通用能力与推理	Beam 128K	65.1	58.2	63.0

架构

Muse Glimmer 是一个稠密的 300 亿参数模型，由以下部分组成：

• 用于视觉的 20 亿参数 ViT 风格编码器（Perception Encoder）
• 280 亿参数文本解码器

除主 VLM 外，还有一个基于 DFlash 实现的推测解码草稿模型。该模块的使用是可选的，代价是增加部分内存占用，但可以显著加快生成速度。我们发现，该草稿模型尤其适合编程等结构化内容生成。

文本解码器

该语言模型采用以下架构组件：

• **混合注意力：**在使用旋转位置嵌入的三个滑动窗口层（窗口大小为 2,048 个 token）之后，接入第四个使用全注意力和 NoPE（无位置嵌入）的层。因此，其模式为 (SWA, SWA, SWA, Full)，重复 13 次，共形成 52 层。这使模型能够利用 RoPE 保留相对顺序和距离信息，并利用 NoPE 在全局范围保留信息。
• **门控分组查询注意力：**每个键值头由 16 个查询头共享，这将 KV-cache 内存降低 16 倍，使生成更快、成本更低。
• **带额外查询缩放的 Q-K 归一化：**在计算注意力之前，Muse Glimmer 对每个查询头和键头应用 RMS 归一化，以保持注意力 logits 稳定。之后，查询会乘以一个缩放因子，在归一化后设定目标 logit 缩放。额外的查询缩放在 softmax 层面表现得类似于逆温度。

感知编码器

Muse Glimmer 使用一个图像编码器同时处理图像和视频。不同于其他 VLM 所使用的相对较小的视觉编码器，这是一个规模达 20 亿参数、采用 Perception Encoder 架构设计的类 ViT 模型。Meta 此前曾将 Perception Encoder 作为多种下游空间与多模态任务的骨干网络进行介绍。

该编码器会将图像切分为形状为 2 帧 x 3 通道 x 14 x 14 的 patch，并通过线性层进行投影。随后，会向这些嵌入添加从已学习的位置表插值得到的绝对位置嵌入。之后，这些嵌入被送入由 50 层和 GELU MLP 组成的视觉塔。与语言模型类似，其注意力模式由三个窗口注意力层和一个全注意力层组成。在注意力层内部，对查询和键应用 2D RoPE。

经过 Transformer 后，pixel shuffle 会拼接相邻空间 token 的 2x2 分组，在不丢弃通道的情况下将图像 token 数量减少 4 倍。合并后的特征随后被投影至与文本解码器共享的嵌入空间。

视频会逐帧经过同一个编码器处理，每帧都被转换为 patch（形状为 [batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14]）。处理器目标采样率为每秒 2 帧，并将视频片段上限设为从视频中均匀采样的 96 帧。处理器会创建带时间戳的视频占位符，将文本与帧交错排列，例如 Time: 0.0s <|video|> x N；在最终投影层之前，这些占位符会被替换为最终的视频嵌入。

Transformers

将 transformers 升级至最新版本，以便使用 Muse Glimmer。

pip install --upgrade transformers accelerate

Muse Glimmer 在 transformers 中提供发布首日支持，主模型和推测解码草稿模型均已支持。你可以使用 AutoModelForMultimodalLM 和 AutoProcessor 类加载模型及处理器。

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "meta-models/Muse-Glimmer-30B"


processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto"
)

相同的代码片段无需修改即可在 NVIDIA（CUDA）、AMD（ROCm）和 Intel（XPU）GPU 上运行；device_map="auto" 会将模型部署到可用的加速器上。

纯文本推理

加载模型后，可以按如下方式进行纯文本推理。

messages = [
    {"role": "user", "content": "Write a short joke about saving RAM."},
]


inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)
input_len = inputs["input_ids"].shape[-1]


outputs = model.generate(**inputs, max_new_tokens=1024)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
print(response)

使用图像和文本提示模型

要使用图像和文本，需要安装 torchvision。

pip install torchvision

Muse Glimmer 接受图像作为输入，示例如下：

messages = [
    {
        "role": "user", "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "What is shown in this image?"}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)
input_len = inputs["input_ids"].shape[-1]


outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
print(response)

多模态工具调用

Muse Glimmer 可以进行多模态工具调用，以下是具体方法。在下方示例中，我们要求模型根据图像中的城市调用天气工具。

import json
import re

tools = [
    {
        "type": "function",
        "function": {
            "name": "weather.get",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                },
                "required": ["city"],
            },
        },
    }
]


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tools=tools,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

parsed = processor.tokenizer.parse_response(response)

目标检测

你可以使用 Muse Glimmer 对图像进行开放式目标检测，方法如下。

import json

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
        {
            "type": "text",
            "text": (
                "Detect the bridge. Return only the detection in the model's "
                "native object-detection format, with no explanation."
            ),
        },
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

detections = json.loads(response.removesuffix("<|eot|>"))
print(detections)



xyxy = (
round(box["x_min"] / 1000 * width),
round(box["y_min"] / 1000 * height),
round(box["x_max"] / 1000 * width),
round(box["y_max"] / 1000 * height),
)

视频推理

要处理视频，我们建议在环境中安装 torchcodec。

pip install torchcodec

Muse Glimmer 可以在没有音频的情况下回答有关视频的复杂问题。你可以按如下方式进行视频推理；下面是来自 VideoMME2 的一个示例，该基准是最受欢迎的视频问答基准。

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/IMG_8137.mp4"},
            {"type": "text", "text": "Describe what happens in this video."},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low",
    processor_kwargs={"num_frames": 96},
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=1024)

response = processor.decode(
    outputs[0, input_len:],
    skip_special_tokens=False,
)

parsed = processor.parse_response(
    response,
    prefix=inputs["input_ids"],
)
print(parsed)

Llama.cpp

Muse Glimmer 在 llama.cpp 中提供发布首日支持。Meta 已在此仓库发布经过校准的量化版本，Uunsloth 也将发布优化量化版本。同时支持 DFlash 推测解码。你可以使用预构建的 llama 二进制文件启动 llama 服务器或 CLI。要安装 llama.cpp，请运行：

curl -LsSf https://llama.app/install.sh | sh

然后可按如下方式启动服务器。

llama serve meta-models/Muse-Glimmer-30B-GGUF

服务器启动后，可访问 localhost:8080，通过内置 WebUI 进行对话。

TODO：插入使用该模型的 WebUI 视频

你也可以按如下方式查询服务器。

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Write a limerick about python exceptions"}
        ]
    }'

你还可以使用

Meta is back with Muse Glimmer: local, agentic, multimodal, and open source!

Great news from the OGs of open source LLMs! Muse Glimmer, released today, is Meta’s new multimodal model, especially designed for local agentic use cases. Distilled from Muse to

30B

parameters, and released under the

Apache 2.0 license

, it’s ideal deploying locally for privacy, reducing costs, or just hacking around. It’s intended for privacy-aware applications such as coding, document analysis, personal assistants, Claw- or Hermes-like setups.

To celebrate, we are shipping with Meta day-0 support in transformers, llama.cpp, vLLM, Inference Endpoints, and other libraries. We built a few cool things and explain our findings in this blog. Check out the demos below for inspiration. You can find Muse Glimmer on the Hugging Face Hub.

Benchmarks

Benchmark results

Scores are reported as published. Bold indicates the best result among the compared models; ↓ indicates lower is better.

Category | Benchmark | Muse Glimmer-30B
High Reasoning | Gemma4-31B
Thinking Mode | Qwen3.6-27B
Thinking Mode
---|---|---|---|---
General Agentic | MCP Atlas | 75.5 | 54.2 | 62.5
General Agentic | DeepSearch QA | 74.6 | 61.7 | 71.1
General Agentic | τ³-Banking | 23.5 | 15.1 | 16.7
General Agentic | WildClawBench | 47.6 | 37.6 | 43.2
General Agentic | GDPval-AA | 953 | 811 | 1141
General Agentic | GAIA2 | 43.3 | 36.4 | 40.0
General Agentic | SkillsBench (With Skills) | 44.3 | 32.4 | 46.6
General Agentic | OSWorld-Verified | 65.9 | 58.5 | 75.6
Agentic Coding | SWE-Bench Pro | 51.2 | 36.9 | 50.2
Agentic Coding | SWE-Bench Verified | 76.0 | 66.6 | 77.2
Agentic Coding | TerminalBench 2.1 | 51.7 | 43.4 | 60.7
Agentic Coding | SciCode | 43.6 | 43.4 | 39.8
Multimodal | Charxiv Reasoning | 78.8 | 77.7 | 78.4
Multimodal | ScreenSpot Pro | 75.4 | 75.9 | 76.1
Multimodal | OmniDocBench v1.5 | 75.8 | 72.5 | 77.8
Multimodal | MMMU Pro | 74 | 73 | 75
Safety | CI Memories | Violation (↓): 26.4
Coverage: 64.8 | Violation (↓): 12.1
Coverage: 53.0 | Violation (↓): 53.4
Coverage: 66.9
Safety | Siren AgentDojo | Attack Success Rate (↓): 28.4
Utility: 94.2 | Attack Success Rate (↓): 25.6
Utility: 90.8 | Attack Success Rate (↓): 40.3
Utility: 92.7
General Capabilities and Reasoning | IFBench | 77.0 | 76.0 | 70.8
General Capabilities and Reasoning | AIME 2026 | 94.7 | 89.2 | 94.1
General Capabilities and Reasoning | GPQA Diamond | 83.5 | 85.7 | 84.2
General Capabilities and Reasoning | Humanity’s Last Exam (Text + No Tools) | 22.0 | 23.6 | 23.1
General Capabilities and Reasoning | AA-LCR | 80.0 | 68.3 | 73.3
General Capabilities and Reasoning | Beam 128K | 65.1 | 58.2 | 63.0

Architecture

Muse Glimmer is a dense 30B parameter model consisting of:

• 2B ViT-style encoder for vision (Perception Encoder)
• 28B parameter text decoder

In addition to the main VLM, there’s also a speculative decoding drafter implemented on DFlash. Usage of this module is optional, and it can provide much faster generation in exchange for some memory cost. We found this drafter to be particularly well suited to structured content generation such as coding.

Text Decoder

The language model uses the following architecture components:

• Hybrid attention: Alternating between three sliding window layers (of 2,048 tokens) using rotary position embedding, followed by a fourth layer that uses full attention and NoPE (no positional embedding). The pattern is therefore (SWA, SWA, SWA, Full), repeated 13 times to a total of 52 layers. This allows the model to retain relative order and distance information with RoPE and preserve information globally with NoPE.
• Gated Grouped-Query Attention: Each key-value head is shared by 16 query heads, which reduces KV-cache memory by 16x and makes generation faster and cheaper.
• Q-K normalization with extra query scaling: Before computing attention, Muse Glimmer applies RMS normalization to every query and key head to keep attention logits stable. After this, queries are multiplied by a scale factor to set the target logit scale after normalization. The extra query scaling behaves like an inverse temperature at the softmax level.

Perception Encoder

Muse Glimmer uses one image encoder to handle both images and videos. Unlike the relatively small vision encoders used in other VLMs, this is a sizable 2B ViT-like model designed after the Perception Encoder architecture. Perception Encoder was previously introduced by Meta as a backbone for various downstream spatial and multimodal tasks.
The encoder patchifies images to a shape of 2 frames x 3 channels x 14 x 14, and passes them through a linear layer for projection. An interpolated absolute position embedding from a learned position table is then added to these embeddings. These are then sent to the vision tower which consist of 50 layers and GELU MLPs. Similar to the language model, the attention pattern consists of three window attention layers followed by one full attention layer. Inside the attention layers, 2D RoPE is applied to the queries and keys.

After transformer, pixel shuffle concatenates 2x2 groups of neighboring spatial tokens which reduces the number of image tokens 4x without discarding their channels. The merged features are then projected to the shared embedding space of the text decoder.

Videos go through the same encoder frame by frame, where each frame is converted into patches (of shape [batch, temporal groups, grid height, grid width, 2 frames, 3 channels, 14, 14]). The processor targets 2 frames per second and caps the clip at 96 frames sampled evenly across video. The processor creates timestamped video placeholders, interleaving text with frame e.g. “Time: 0.0s <|video|> x N” in which the final video embeddings are replaced before the final projection layer.

Transformers

Upgrade transformers to the latest version to be able to use Muse Glimmer.

pip install --upgrade transformers accelerate

Muse Glimmer comes with day-0 support in transformers, both for the main model and the speculative decoding drafter. You can use AutoModelForMultimodalLM and AutoProcessor classes to load the model and the processor.

from transformers import AutoProcessor, AutoModelForMultimodalLM

MODEL_ID = "meta-models/Muse-Glimmer-30B"


processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    device_map="auto"
)

The same snippet runs unchanged on NVIDIA (CUDA), AMD (ROCm) and Intel (XPU) GPUs, device_map="auto" places the model on whichever accelerator is available.

Text-only Inference

After loading the model, you can do text-only inference with it as follows.

messages = [
    {"role": "user", "content": "Write a short joke about saving RAM."},
]


inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)
input_len = inputs["input_ids"].shape[-1]


outputs = model.generate(**inputs, max_new_tokens=1024)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
print(response)

Prompting the model with images and text

We would need torchvision to be able to use images and text.

pip install torchvision

Muse Glimmer accepts images as input, as demonstrated here:

messages = [
    {
        "role": "user", "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "What is shown in this image?"}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)
input_len = inputs["input_ids"].shape[-1]


outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
print(response)

Multimodal tool calling

Muse Glimmer can do multimodal tool calling, here’s how you can do it. In the example below, we ask the model to call the weather tool based on the city in the image.

import json
import re

tools = [
    {
        "type": "function",
        "function": {
            "name": "weather.get",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                },
                "required": ["city"],
            },
        },
    }
]


messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "I'm going to the city in this picture. What clothes should I wear?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tools=tools,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low"
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

parsed = processor.tokenizer.parse_response(response)

Object Detection

You can use Muse Glimmer to do open ended object detection in images as follows.

import json

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
        {
            "type": "text",
            "text": (
                "Detect the bridge. Return only the detection in the model's "
                "native object-detection format, with no explanation."
            ),
        },
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=128)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

detections = json.loads(response.removesuffix("<|eot|>"))
print(detections)



xyxy = (
round(box["x_min"] / 1000 * width),
round(box["y_min"] / 1000 * height),
round(box["x_max"] / 1000 * width),
round(box["y_max"] / 1000 * height),
)

Video Inference

To work with videos we recommend installing torchcodec into the environment.

pip install torchcodec

Muse Glimmer can answer complex questions about videos without audio. You can do video inference as follows, here’s an example from VideoMME2, which is the most popular video question answering benchmark.

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {
        "role": "user",
        "content": [
            {"type": "video", "video": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/IMG_8137.mp4"},
            {"type": "text", "text": "Describe what happens in this video."},
        ],
    },
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    reasoning_strength="low",
    processor_kwargs={"num_frames": 96},
).to(model.device)

input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=1024)

response = processor.decode(
    outputs[0, input_len:],
    skip_special_tokens=False,
)

parsed = processor.parse_response(
    response,
    prefix=inputs["input_ids"],
)
print(parsed)

Llama.cpp

Muse Glimmer comes with day-0 llama.cpp support. Meta has distributed calibrated quants in this repo, and Uunsloth is releasing optimized quants as well. DFlash speculative decoding is supported as well. You can use a pre-built llama binary to start a llama server or a CLI. To install llama.cpp, run

curl -LsSf https://llama.app/install.sh | sh

Then you can start the server as follows.

llama serve meta-models/Muse-Glimmer-30B-GGUF

Once the server has started, you can head to localhost:8080 to chat with the built-in WebUI.

TODO: Insert webui video with this model

You can also query the server as follows.

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Write a limerick about python exceptions"}
        ]
    }'

You can also use llama server with coding agents like Pi.

Speculative Decoding

DFlash uses a lightweight block-diffusion drafter model to provide same output with extra speed-ups in decoding phase. Transformers and llama.cpp ship support for DFlash drafter of Muse Glimmer day-0.

Below you can see how speculative decoding can speed-up generation in realistic setups. The video shows llama.cpp webui with DFlash on the left and regular generation on the right.

Speculative Decoding with transformers

You can load the drafter and model as follows, and infer like how you would with base model with an additional parameter (shown in the upcoming snippets).

import torch
from transformers import AutoProcessor, MuseGlimmerAssistantModel, MuseGlimmerForConditionalGeneration

model_id = "meta-models/Muse-Glimmer-30B"
target = MuseGlimmerForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
assistant = MuseGlimmerAssistantModel.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "user", "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/merve/vl-test-suite/resolve/main/SF.png"},
            {"type": "text", "text": "What is shown in this image?"}
        ]
    }
]

out = target.generate(**inputs, assistant_model=assistant, speculation_type="dflash", max_new_tokens=64, do_sample=True)
print(processor.batch_decode(out)[0])

Speculative Decoding with llama.cpp

You can start llama server using following command. --spec-draft-n-max argument controls how many future tokens DFlash proposes during each speculative-decoding step. Muse Glimmer’s DFlash model was trained with a block size of 16, one anchor token plus 15 proposed tokens, so any value above 15 will be clamped to 15.

llama serve -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash --spec-draft-n-max 15

You can also use llama cli with speculative decoding drafter as follows.

llama cli -hf meta-models/Muse-Glimmer-30B-GGUF --spec-type draft-dflash

Support for Muse Glimmer vLLM with transformers backend

For this release, we ship support for vLLM with transformers backend.

vllm serve meta-models/Muse-Glimmer-30B --model-impl transformers --tensor-parallel-size 4


curl -s http://127.0.0.1:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "meta-models/Muse-Glimmer-30B",
      "messages": [
        {"role": "user", "content": "Explain tensor parallelism briefly."}
      ],
      "temperature": 0.0,
      "max_tokens": 256
    }'

Fine-tuning with TRL

You can use TRL to fine-tune Muse Glimmer using various methods from SFT to Async GRPO. We have run two experiments on bf16 with Hopper-class GPUs with 80GB VRAM each.

Workload	Practical minimum
Inference / eval, BF16	1×80 GB H100
LoRA SFT, BF16	1×80 GB H100, microbatch 1 + checkpointing
Full SFT, BF16	8×80 GB H100 with FSDP/ZeRO-3
LoRA GRPO, Transformers rollouts	1×80 GB H100, but slow/tight
LoRA GRPO, separate vLLM rollout server	8×H100: 4 rollout + 4 training
Full-finetune GRPO	8 GPUs is usually insufficient

As part of this release, we ship an example to fine-tune Muse Glimmer on small split of MolmoWeb dataset. This shows how to make model generate structured outputs and how to fine-tune on images.

We also experimented with running the model on OpenCode with AsyncGRPO example. Model shows strong coding capabilities, so we encourage you to try training with coding environments in OpenEnv and TRL.

Demos

Here are some fun ways to try out Muse Glimmer. In our opinion, the coolest thing about this model is that it is a local scale personal assistant that can code. That means you can make it do things like, quantize itself, find quantized weights on the Hub, deploy itself to inference endpoints, and even optimize itself for specific hardware! Let’s go team local 🚀

Connect OpenClaw to Muse Glimmer

Assume the Inference Endpoint exposes an OpenAI-compatible /v1 API.

Set HF_TOKEN in the OpenClaw gateway environment, then add this to ~/.openclaw/openclaw.json:

OpenClaw configuration

{
  models: {
    mode: "merge",
    providers: {
      muse: {
        baseUrl: "https://YOUR-ENDPOINT.endpoints.huggingface.cloud/v1",
        apiKey: {
          source: "env",
          provider: "default",
          id: "HF_TOKEN"
        },
        api: "openai-completions",
        authHeader: true,
        models: [{
          id: "meta-models/Muse-Glimmer-30B",
          name: "Muse Glimmer",
          reasoning: false,
          input: ["text", "image"],
          contextWindow: 32768,
          maxTokens: 8192
        }]
      }
    }
  },
  agents: {
    defaults: {
      model: { primary: "muse/meta-models/Muse-Glimmer-30B" }
    }
  }
}

Restart OpenClaw:

openclaw gateway restart

Validate from a fresh session:

openclaw agent --message "Reply with: muse-ready"

Use the exact model ID returned by the endpoint’s /v1/models response if it differs.

Hey Muse Glimmer, quantize yourself

If we hook up Muse Glimmer to the Hugging Face MCP and update its AGENTS.md we give it the capability to find a quantized version of itself on the hub and run locally. This is handy if you want to work on something private, or just cut costs.

If you do this a second time, Muse Glimmer will find the cached weights and switch to them, so feel free to add a convenient command like /spawn.

Muse Glimmer inspects the machine and Hub, selects or creates a Q4_K_M GGUF, launches llama-server, and validates model discovery and chat completion. The result is a smaller local build behind an OpenAI-compatible API. Here’s the prompt we added to AGENTS.md.

https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/Muse%20Glimmer%20Quantisation%20Demo%20-%20explained.mp4?download=true

By adding this to AGENTS.md openclaw or hermes will be able to solve the rest.

Local quantization prompt

## Local model deployment

When asked to deploy locally, perform the work; do not give instructions.

1. Inspect hardware and the Hugging Face cache.
2. Search the Hub for compatible GGUF weights using `apps=llama.cpp`; confirm exact filenames through the model-tree API.
3. Prefer an existing suitable GGUF, normally `Q4_K_M`. Treat `mmproj-*.gguf` as projector weights.
4. If no GGUF exists, download the source weights, convert with `convert_hf_to_gguf.py`, then quantize with `llama-quantize`.
5. Preserve source weights and record the repository, revision, filenames, and quantization.
6. Start `llama-server` with an `onyx` alias and an OpenAI-compatible endpoint.
7. Validate `/v1/models` and `/v1/chat/completions`, requiring non-empty, correct content.
8. Report concise progress and logs. Claim completion only after validation passes.

Hey Muse Glimmer, deploy yourself

Muse Glimmer can also take care of the opposite. Let’s get Glimmer to deploy itself on Hugging Face Inference Endpoints. Which is useful if you want to speed up on some cutting edge hardware.

N.B. You can also just deploy Muse Glimmer to Inference Endpoints directly and connect your agent.

Muse Glimmer pins the model revision, deploys it to a protected Hugging Face Inference Endpoint, and verifies health, model discovery, and chat completion. It then connects the Claw agent with secrets and rollback preserved. Here’s the prompt we added to AGENTS.md. Muse glimmer will also need the Hugging Face MCP and/or the Hugging Face CLI and Skills.

Inference Endpoint deployment prompt

## Hugging Face Inference Endpoint deployment

When asked to deploy on Hugging Face Inference Endpoints, perform the work; do
not give instructions.

1. Inspect Hugging Face authentication, the current model repository, and any
   existing endpoints.
2. Confirm the exact model repository and immutable revision through the Hub
   API; inspect its architecture, configuration, and chat template.
3. Confirm that the model is supported by vLLM, then deploy or update a
   protected Inference Endpoint using the managed native vLLM engine.
4. Choose an available region and the smallest suitable accelerator. Use one
   replica and enable scale-to-zero when supported.
5. Preserve the previous endpoint configuration for rollback. Do not expose
   tokens, publish private weights, or replace an unrelated endpoint.
6. Wait for the endpoint to become ready. If startup fails, inspect the logs
   and report the actual blocker rather than repeatedly changing settings.
7. Validate `/health`, `/v1/models`, and `/v1/chat/completions`, requiring the
   expected model and non-empty, correct content. When agent use is required,
   also validate a real structured tool call.
8. Configure the Claw agent to use the endpoint's OpenAI-compatible `/v1` URL,
   storing credentials as secrets and retaining the previous provider as
   rollback. Test the connection in a fresh session.
9. Report concise progress and finish with the repository, revision, engine,
   hardware, endpoint URL, scaling state, and validation results. Claim
   completion only after every required check passes.

Hey Muse Glimmer, optimize yourself

Finally, let’s get Muse Glimmer to do some light RSI. We can instruct our agent to optimize its own inference engine for specific hardware, in this case a Nvidia H100. To do this, the agent will need to use another inference engine, like Inference Endpoints above.

Muse Glimmer benchmarks its own single-H100 serving stack, testing one reversible change at a time while holding the workload fixed. It keeps only correctness-passing gains and finishes with the fastest reproducible configuration. Here’s the prompt we added to AGENTS.md. Muse glimmer need the Hugging Face MCP and the Hugging Face CLI and Skills.

Self-optimization prompt

You are Muse Glimmer acting as an autonomous inference-optimization engineer for your own serving stack.

Goal: maximize valid single-H100 aggregate completion throughput in tokens/second.

Protocol:
1. Establish a correctness-passing baseline.
2. Test one reversible optimization at a time.
3. Keep the prompt, concurrency, sampling, request count, warm-up, and decode length fixed.
4. Reject results that fail correctness or prefix checks.
5. Record every experiment chronologically with its configuration, raw throughput, correctness, and delta.
6. Keep improvements and revert regressions.
7. Stop after six consecutive regressions or when the experiment budget is exhausted.
8. Report the best valid configuration and exact reproduction command.

Create a minimal scientific animation of the results:
- white background;
- raw tokens/second—never normalize;
- one point revealed per experiment;
- connect every point chronologically;
- begin with the lowest valid result;
- stop at the best result;
- export as a GIF.

Never fabricate, interpolate, or count correctness-failing measurements.

https://huggingface.co/buckets/huggingface/muse-glimmer-assets/resolve/onyx-optimization-progress.gif?download=true

Hey Muse Glimmer, research the Hub

Try Muse Glimmer as a Hugging Face research agent. The Gradio Space sends each model request to a private Hugging Face Inference Endpoint through its OpenAI-compatible API. It also connects to the official Hugging Face MCP server, giving the agent read-only tools to search and inspect Hub repositories, models, datasets, Spaces, documentation, and papers.

Wrapping Up

We are happy to welcome Muse Glimmer to the Hugging Face Hub. Try Muse Glimmer with your local coding setups today!