Marin：面向基础模型研发的开源框架与社区

英文标题：🔥 marin-community/marin

Marin 是一个面向基础模型研究与开发的开源研究计划、软件平台和社区，覆盖数据整理、预训练、后训练与评估等完整 LLM 开发流程。项目强调“开放开发”，承诺实时记录流程、实验、决策及失败实验，并公开构建模型所需的过程知识。其当前重点是从零开始训练并后训练一个总参数量超过 5000 亿、训练计算量达 5e24 model-FLOPs 的 MoE 模型，同时通过 Delphi 扩展套件研究从 3e18 到 1e23 FLOPs 的扩展规律。对于开发者而言，Marin 提供了以依赖步骤为核心的实验框架、教程和可复用组件，可用于从 TinyStories 小模型训练扩展至大规模 GPU 或 TPU 集群训练。

4.4 KB 948 B 1.3 KB 61.2 KB 6.3 KB 13.2 KB 426 B 11.5 KB 15.9 KB 56.9 KB

中文

Marin

“我不害怕风暴，因为我正在学习如何驾驭我的船。”
—— Louisa May Alcott

Marin 是一个用于研究和开发基础模型的研究计划、软件平台与社区。

Marin 关注大语言模型的训练，包括数据整理、转换、过滤、分词、预训练、后训练和评估。除这些模型背后的产物、软件和基础设施外，Marin 还致力于开放分享构建这些模型所需的全部过程知识。

Marin 的核心价值是**开放开发**。我们在流程、实验和决策发生的同时对其进行记录。从原始数据到最终模型的每一步都会被记录下来，失败实验也是这份记录的一部分。

Marin 还曾被用于构建音频-文本模型、DNA模型和蛋白质模型。我们通过将 Marin 作为库使用，并在 marin/experiments 中开展实验，来鼓励这类工作。

前沿混合专家模型

我们当前的重点是：从零开始预训练并后训练一个大型混合专家模型。该模型的训练计算量为 5e24 model-FLOPs，总参数量超过 5000 亿，目标是在对科学家和研究人员重要的任务上取得成功。

Delphi 是 Marin 的开放扩展套件，旨在将 LLM 配方从 3e18 FLOPs 扩展至 1e23 FLOPs，灵感来自 Pythia。它由三部分构成：将计算预算映射至模型配置的扩展配方；依据该配方在 Google TPU Research Cloud 上训练的扩展套件；以及利用较小 Delphi 模型预测较大模型表现的扩展定律。

我们发布了：

进展通过 GitHub issue #1337 跟踪。

更多汇总后的经验可见Open Athena 博客。以下为部分内容：

此前，我们使用 Marin 训练了一个 80 亿参数模型，其在我们的基础模型基准测试套件上优于 Llama 3.1 8B。你可以查看训练脚本，或阅读回顾报告。我们还训练了 Marin 32B。

深入了解与使用 Marin

Marin 的文档可通过 ReadTheDocs 获取，也可在 docs/ 文件夹中查看。

开始使用 Marin：

Marin 实验被定义为一组可相互依赖的步骤，并按照拓扑顺序执行，类似于 Makefile。

下面是一个简要示例，展示如何使用 Marin 在 TinyStories 上训练一个微型模型。更多细节可查看完整脚本。

from fray.cluster import ResourceConfig
from levanter.optim import AdamConfig
from marin.execution.lazy import lower
from marin.execution.step_runner import StepRunner
from marin.experiment.data import tokenized
from marin.experiment.train import train_lm

from experiments.llama import llama_nano
from experiments.marin_tokenizer import marin_tokenizer

# 1. Tokenize the dataset as a lazy handle — nothing downloads yet.
tinystories_tokenized = tokenized(
    name="tokenized/tinystories",
    source="roneneldan/TinyStories",
    tokenizer=marin_tokenizer,
    sample_count=1000,  # cap at 1 000 samples per shard to keep the tutorial fast
)

# 2. Train the model — depends on the tokenized dataset above.
nano_tinystories_model = train_lm(
    name="checkpoints/marin-nano-tinystories",
    version="v1",
    model=llama_nano,
    optimizer=AdamConfig(learning_rate=6e-4, weight_decay=0.1),
    # Steps can depend on other steps: nano_tinystories_model depends on tinystories_tokenized
    datasets={tinystories_tokenized: 1.0},
    batch_size=4,
    seq_len=2048,
    num_train_steps=100,
    z_loss_weight=None,
    evals=None,  # no point evaluating such a tiny model
    resources=ResourceConfig.with_cpu(),
)

if __name__ == "__main__":
    StepRunner().run([lower(nano_tinystories_model)])

这里我们创建了两个步骤：一个用于对数据集进行分词，另一个用于训练模型。训练步骤依赖于已分词的数据集步骤，因此会在分词步骤完成后执行。

经过少量修改，你可以将其扩展为在更大数据集上训练更大模型，或训练数据集混合体，甚至扩展至非常大的 GPU 或 TPU Pod（或多切片 TPU）。

• 有关项目工作流程，请参阅 CONTRIBUTING.md。
• 可加载的 Agent 技能位于 .agents/skills/（以及 .claude/skills/）中。例如，.agents/skills/add-dataset/ 提供了添加新数据集的分步指南。

Marin 的核心协作者来自 Stanford CRFM 和 Open Athena。

Marin 的研究得益于合作伙伴的慷慨支持。

用于 TRC 加速器	用于 GPU 集群
用于支持开发	用于支持开发

English / 原文

Marin

"I am not afraid of storms, for I am learning how to sail my ship. "
– Louisa May Alcott

Marin is a research program, software platform, and community for the research and development of foundation models.

Marin's concern is training large language models. This includes data curation, transformation, filtering, tokenization, pretraining, posttraining, and evaluation. Beyond the artifacts, software, and infrastructure, behind these models, Marin is committed to openly sharing all of the process knowledge required to build these models.

Marin's core value is open development. We document our processes, experiments, and decisions as they happen. Every step, from raw data to the final model, is recorded. Failed experiments are part of that record.

Marin has also been used for building audio-text models, DNA, and protein models. We encourage this work through the use of Marin as a library, in marin/experiments.

Frontier mixture-of-experts

Our current focus is pretraining, from scratch, and posttraining a large (5e24 model-FLOPs, 500 billion+ total parameters) mixture-of-experts model to succeed on tasks of importance to scientists and researchers.

Delphi is Marin's open scaling suite scaling a LLM recipe from 3e18 to 1e23 FLOPs, inspired by Pythia. It has three parts: a scaling recipe that maps compute budgets to model configurations, a scaling suite trained from that recipe on the Google TPU Research Cloud, and a scaling law that uses the smaller Delphi models to predict the larger ones.

We released:

Progress was tracked in GitHub issue #1337.

Some additional consolidated learnings can be found on the Open Athena blog. A selection, below:

Previously, we used Marin to train an 8B parameter model that outperformed Llama 3.1 8B on our base-model benchmark suite. You can see the training script or read the retrospective. We also trained Marin 32B.

Learning more & using Marin

The documentation for Marin is available on ReadTheDocs or in the docs/ folder.

To get started with Marin:

Marin experiments are defined as a set of steps that can depend on each other and are executed in a topological order, like a Makefile.

As a brief example of how you can use Marin, here is a complete script for training a tiny model on TinyStories. You can check out the full script for more details.

from fray.cluster import ResourceConfig
from levanter.optim import AdamConfig
from marin.execution.lazy import lower
from marin.execution.step_runner import StepRunner
from marin.experiment.data import tokenized
from marin.experiment.train import train_lm

from experiments.llama import llama_nano
from experiments.marin_tokenizer import marin_tokenizer

# 1. Tokenize the dataset as a lazy handle — nothing downloads yet.
tinystories_tokenized = tokenized(
    name="tokenized/tinystories",
    source="roneneldan/TinyStories",
    tokenizer=marin_tokenizer,
    sample_count=1000,  # cap at 1 000 samples per shard to keep the tutorial fast
)

# 2. Train the model — depends on the tokenized dataset above.
nano_tinystories_model = train_lm(
    name="checkpoints/marin-nano-tinystories",
    version="v1",
    model=llama_nano,
    optimizer=AdamConfig(learning_rate=6e-4, weight_decay=0.1),
    # Steps can depend on other steps: nano_tinystories_model depends on tinystories_tokenized
    datasets={tinystories_tokenized: 1.0},
    batch_size=4,
    seq_len=2048,
    num_train_steps=100,
    z_loss_weight=None,
    evals=None,  # no point evaluating such a tiny model
    resources=ResourceConfig.with_cpu(),
)

if __name__ == "__main__":
    StepRunner().run([lower(nano_tinystories_model)])

Here, we create two steps, one for tokenizing the dataset and one for training the model. The training step depends on the tokenized dataset step, so it will be executed after the tokenization step is completed.

With slight modifications, you can extend this to train a larger model on a larger dataset, a mixture of datasets, even scaling to very large GPU or TPU pods (or multislice TPUs!).

• See CONTRIBUTING.md for project workflow.
• See .agents/skills/ (also .claude/skills/) for loadable agent skills. For example, .agents/skills/add-dataset/ has a step-by-step guide to adding new datasets.

Marin's core collaborators come from Stanford CRFM and Open Athena.

Marin's research is made possible by the generous support of our partners.

for TRC accelerators	for GPU clusters
for supporting development	for supporting development