开放权重模型如今已强大到足以处理多步骤 Agent 工作流、高级推理和长程编码等复杂任务。然而，随着这些模型能力提升，其规模也随之增长；托管万亿级参数架构需要专门构建的基础设施、高端 GPU 算力和经过优化的推理服务框架。2026 年 7 月 27 日，Moonshot AI 发布了 Kimi K3：这是一款拥有 2.8 万亿参数的混合专家（Mixture of Experts，MoE）模型，是首个达到 3 万亿参数量级的开放权重系统。Kimi K3 提供前沿级智能能力，同时公开其模型权重，使组织能够在自身基础设施上自托管目前能力最强的模型之一。

本文介绍两种在 AWS 上部署 Kimi K3 的方法：Amazon SageMaker HyperPod 和 Amazon Elastic Kubernetes Service（Amazon EKS）集群。

关于 Kimi K3

Kimi K3 基于差异化架构构建，采用 Kimi Delta Attention（KDA）、Gated Multi Head Latent Attention（MLA）以及 Stable LatentMoE 框架。该模型将其 2.8 万亿参数分布在 896 个专用专家中，每个 token 仅激活其中 16 个。这意味着在任何一次前向传播期间，约有 1040 亿参数处于激活状态，相较其前代产品 Kimi K2，扩展效率提升了 2.5 倍。

属性	值
总参数量	2.8 万亿
每个 Token 的激活参数量	1040 亿
架构	混合专家（MoE）
专家数量	896（每个 token 激活 16 个）
上下文窗口	100 万 Token
模态	原生多模态（文本 + 视觉）
发布日期	2026 年 7 月 27 日

Kimi K3 擅长长程编码任务、Agent 工作流和复杂推理。它支持原生工具调用、结构化输出，以及用于多步骤问题求解的常开思考模式。

模型可用性与格式

Kimi K3 的开放权重可通过 Hugging Face 获取，模型标识符为 moonshotai/Kimi-K3。其权重以 MXFP4（Microscaling Floating Point 4-bit，微缩放 4 位浮点）格式发布，为大规模推理部署中的模型质量和内存效率提供了有效平衡。

鉴于该模型的架构和规模，提供 Kimi K3 服务需要使用面向 Kimi K3 的 vLLM day-0 推理容器。截至本文撰写时，Kimi K3 对应的 vLLM 提交版本位于 vllm/vllm-openai:kimi-k3。我们预计这些提交将在后续版本中合并至主 vLLM 容器。vLLM 原生支持 MoE 架构、张量并行和 MXFP4 量化格式，因此是该模型推荐的推理引擎。

基础设施要求

部署如此规模的模型需要大量 GPU 算力。Kimi K3 需要使用 p6-b300 实例（ml.p6-b300.48xlarge），该实例提供 8 张 NVIDIA B300 Blackwell Ultra GPU，并配备高带宽互连，以支持跨完整专家池进行高效的张量并行推理。

AWS 提供两种获取该容量的主要机制：

1. 灵活训练计划（适用于 SageMaker HyperPod）： 提供可分配给 HyperPod 集群的承诺容量预留，确保 GPU 资源可用于持续的推理工作负载。
2. 容量区块（Capacity Blocks）： 允许你在特定时间段内预留 EC2 GPU 实例，在无需长期承诺的情况下确保获得 p6-b300 容量。Amazon EKS 工作负载可通过定位到预留容量来使用这些预留。

方案 1：在 Amazon SageMaker HyperPod 上部署

配合 Inference Operator，Amazon SageMaker HyperPod 为部署 Kimi K3 提供了最简单的路径。Inference Operator 会在创建集群时自动安装，并将容器编排、模型加载和端点管理的复杂性抽象化。

前提条件

在部署模型前，请完成以下两个前提步骤以设置基础设施。

步骤 1：创建采用 EKS 编排的 SageMaker HyperPod 集群

在部署模型前，必须先预置一个 HyperPod 集群。请进入 Amazon SageMaker AI 控制台，并按照集群创建流程操作：

1. 打开 SageMaker AI 控制台，选择 HyperPod Clusters > Cluster Management > Create HyperPod cluster。
2. 从列表中选择 Orchestrated by Amazon EKS。
3. 选择 Quick setup，以使用默认网络、存储和 IAM 资源预置集群；或者选择 Custom setup，以集成现有的 VPC、子网和安全组。
4. 在 Orchestration 下，创建新的 EKS 集群或附加现有 EKS 集群。确认已选择 Use default Helm charts and add-ons 选项，以便自动安装 Inference Operator 及其他所需 Operator。
5. 在 Instance groups 下，添加一个配置为 ml.p6-b300.48xlarge 实例类型的工作节点组。
6. 检查配置，然后选择 Submit 开始预置。

如需完整操作指南，请参阅使用 Amazon EKS 编排创建 SageMaker HyperPod 集群文档。

步骤 2：使用灵活训练计划获取容量

ml.p6-b300.48xlarge 实例类型需要预留容量。灵活训练计划可为 Blackwell GPU 节点提供承诺容量预留，确保集群能够使用 p6-b300 实例，而不会与通用按需资源池产生竞争。进入 SageMaker 控制台，根据你的时间安排和实例数量选择 FTP 区块。要创建或附加训练计划：

1. 在实例组配置中，选择 Training plan 作为容量来源。
2. 选择一个涵盖 ml.p6-b300.48xlarge 容量的现有计划，或创建一个新预留，指定所需的实例数量和持续时间。
3. 将 Target Availability Zone 设置为与你的训练计划容量所在可用区相匹配。

当集群达到 Active 状态且 p6-b300 节点健康时，即可部署模型。

部署模型

要在 HyperPod 上部署 Kimi K3，请将以下 InferenceEndpointConfig 清单应用到集群：

apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: kimik3
spec:
  modelName: Kimi-K3
  instanceType: ml.p6-b300.48xlarge
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    huggingFaceModel:
      modelId: moonshotai/Kimi-K3
    modelSourceType: huggingface
  worker:
    image: vllm/vllm-openai:kimi-k3
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      mountPath: /opt/ml/model
      name: model-weights
    resources:
      limits:
        nvidia.com/gpu: 8
      requests:
        nvidia.com/gpu: 8
    args:
      - "--model"
      - "moonshotai/Kimi-K3"
      - "--trust-remote-code"
      - "--load-format"
      - "fastsafetensors"
      - "--enable-prefix-caching"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser"
      - "kimi_k3"
      - "--reasoning-parser"
      - "kimi_k3"
      - "--served-model-name"
      - "Kimi-K3"
      - "--moe-backend"
      - "auto"
      - "--tensor-parallel-size"
      - "8"
      - "--no-enable-flashinfer-autotune"
    environmentVariables:
      - name: "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION"
        value: "1"

该 YAML 文件也已在 GitHub 仓库中提供。使用以下命令应用此配置：kubectl apply -f kimi-k3.yaml

Inference Operator 会负责从 Hugging Face 下载模型、容器调度、运行状况检查和端点就绪管理。端点切换至就绪状态后，会在已配置的调用路径上提供兼容 OpenAI 的 API。

方案 2：在 Amazon EKS 上部署

对于希望自行管理 Kubernetes 基础设施的团队，可以在独立的 Amazon EKS 集群上部署 Kimi K3，并通过 EC2 Capacity Blocks 获取 GPU 容量。该服务支持在特定期限内预留 p6-b300 实例，无需长期承诺。

关键部署步骤

AI on EKS 项目提供了一个推理就绪集群方案，可自动完成端到端预置。从高层面看，部署包含以下阶段：

1. 预置 EKS 集群

使用提供的 Terraform 模块创建 GPU 优化的 EKS 集群。这包括 VPC 网络、托管节点组，以及 GPU 工作负载所需的 IAM 角色和策略。

2. 使用 Capacity Blocks 预留 GPU 容量

在目标可用区中为 p6-b300.48xlarge 实例创建 Capacity Block 预留。Capacity Blocks 保证所请求的 GPU 节点在预留时间窗口内可用。预留生效后，实例将作为工作节点加入 EKS 集群。

3. 安装 GPU 驱动程序和设备插件

该方案会在节点组上安装 NVIDIA 设备插件和 GPU 驱动程序，使 Kubernetes 可以发现可用 GPU 并据此进行调度。

4. 部署 vLLM 推理服务器

通过 Helm chart 或 Kubernetes 清单部署带有 Kimi K3 专用参数的 vLLM 容器，包括张量并行大小为 8、MXFP4 加载格式和 MoE 后端配置。模型标识符指向 Hugging Face 仓库；或者，也可以将模型权重同步至 Amazon Simple Storage Service（Amazon S3），以加快模型加载速度。推理服务参数与上文 HyperPod 配置中展示的参数一致。

5. 暴露推理端点

Kubernetes Service（类型为 LoadBalancer，或通过 Ingress 控制器）会在 8000 端口暴露 vLLM 服务器，为应用提供兼容 OpenAI 的 /v1/chat/completions 端点。

6. 验证

通过向端点发送测试请求并验证模型成功返回响应，确认部署已完成。

如需包含 Terraform 模块、Helm values 和分步说明的完整部署指南，请参阅 AI on EKS Kimi K3 方案。

调用端点

部署完成后，Kimi K3 端点会暴露兼容 OpenAI 的 chat completions API。你可以使用 OpenAI Python SDK 或简单的 curl 命令进行调用。

使用 OpenAI Python SDK

from openai import OpenAI
client = OpenAI(
    base_url="http://<ENDPOINT_URL>:8000/v1",
    api_key="not-needed"
)


response = client.chat.completions.create(
    model="Kimi-K3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.choices[0].message.content)

使用 curl

curl -X POST http://<ENDPOINT_URL>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
  }'

将 <ENDPOINT_URL> 替换为由 HyperPod Inference Operator 或 EKS Ingress 配置暴露的服务端点。

清理

为避免持续产生费用，在不再需要时请删除本指南中创建的资源。对于 SageMaker HyperPod 部署：

1. 运行 kubectl delete -f kimi-k3.yaml 删除 InferenceEndpointConfig。
2. 在 SageMaker AI 控制台中，进入 HyperPod Clusters，选择你的集群，然后选择 Delete。
3. 如果不再需要，释放或取消灵活训练计划预留。

对于 Amazon EKS 部署：

4. 删除 vLLM 部署及关联的 Kubernetes 服务。
5. 终止 GPU 节点组，或使用 Terraform（terraform destroy）删除 EKS 集群。
6. 如果 Capacity Block 预留尚未到期，则释放该预留。
7. 有关 p6-b300 实例和 Capacity Block 预留的定价详情，请参阅 Amazon EC2 定价页面。

结论

Kimi K3 代表了开放权重模型能力的新前沿，而 AWS 提供了大规模部署该模型所需的基础设施和托管服务。无论选择简化的 HyperPod Inference Operator 路径，还是自管 EKS 集群的灵活方案，p6-b300 GPU 实例、vLLM 推理服务和 MXFP4 量化权重的组合，都可为这一全球最大的开放模型提供内置运行状况检查、自动恢复和端点就绪验证的部署能力。以下是一些入门链接：

1. SageMaker HyperPod Kimi K3 示例
2. AI on EKS Kimi K3 方案
3. [创建 SageMaker H

Open weight models have become powerful enough to handle complex tasks such as multi-step agentic workflows, advanced reasoning, and long-horizon coding. However, as these models grow in capability, they also grow in size and hosting multi-trillion parameter architectures requires purpose-built infrastructure, high-end GPU compute, and optimized serving frameworks. On July 27, 2026, Moonshot AI released Kimi K3, a 2.8 trillion parameter Mixture of Experts (MoE) model that represents the first open-weight system to reach the 3 trillion parameter class. Kimi K3 delivers frontier-level intelligence while making its weights publicly available, so that organizations can self-host one of the most capable models in existence on their own infrastructure.

This post walks through deploying Kimi K3 on AWS using two approaches: Amazon SageMaker HyperPod, and Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

About Kimi K3

Kimi K3 is built on a differentiated architecture featuring Kimi Delta Attention (KDA), Gated Multi Head Latent Attention (MLA), and a Stable LatentMoE framework. The model distributes its 2.8 trillion parameters across 896 specialist experts, activating only 16 per token. This means approximately 104 billion parameters are active during any single forward pass, yielding a 2.5x improvement in scaling efficiency over its predecessor, Kimi K2.

Attribute	Value
Total Parameters	2.8 Trillion
Active Parameters per Token	104 Billion
Architecture	Mixture of Experts (MoE)
Expert Count	896 (16 activated per token)
Context Window	1 Million Tokens
Modality	Native Multimodal (Text + Vision)
Release Date	July 27, 2026

Kimi K3 excels at long-horizon coding tasks, agentic workflows, and complex reasoning. It supports native tool calling, structured output, and an always-on thinking mode for multi-step problem solving.

Model availability and format

The open weights for Kimi K3 are available on Hugging Face under the model identifier moonshotai/Kimi-K3. The weights are distributed in MXFP4 (Microscaling Floating Point 4-bit) format, which provides an effective balance between model quality and memory efficiency for large-scale inference deployments.

Given the model’s architecture and size, serving Kimi K3 requires a vLLM day-0 inference container for Kimi K3. At the time of writing, vllm commits for Kimi K3 are in vllm/vllm-openai:kimi-k3. We expect these to be merged to the main vllm container in the upcoming releases. vLLM provides native support for MoE architectures, tensor parallelism, and the MXFP4 quantization format, making it the recommended serving engine for this model.

Infrastructure requirements

Deploying a model of this scale requires substantial GPU compute. Kimi K3 requires a p6-b300 instance (ml.p6-b300.48xlarge), which provides 8 NVIDIA B300 Blackwell Ultra GPUs with high-bandwidth interconnects necessary for efficient tensor-parallel inference across the full expert pool.

AWS offers two primary mechanisms to procure this capacity:

1. Flexible Training Plans (for SageMaker HyperPod): Provide committed capacity reservations that can be allocated to your HyperPod cluster, so that GPU resources are available for sustained inference workloads.
2. Capacity Blocks: Allow you to reserve EC2 GPU instances for a defined period, providing guaranteed access to p6-b300 capacity without long-term commitments. Amazon EKS workloads consume these reservations by targeting the reserved capacity.

Option 1: Deploying on Amazon SageMaker HyperPod

Amazon SageMaker HyperPod with the Inference Operator provides the simplest path to deploying Kimi K3. The Inference Operator is installed automatically as part of cluster creation, and it abstracts away the complexity of container orchestration, model loading, and endpoint management.

Prerequisites

Before deploying the model, complete the following two prerequisite steps to set up your infrastructure.

Step 1: Create a SageMaker HyperPod cluster with EKS orchestration

Before deploying the model, you must provision a HyperPod cluster. Navigate to the Amazon SageMaker AI console and follow the cluster creation workflow:

1. Open the SageMaker AI console and select HyperPod Clusters > Cluster Management > Create HyperPod cluster.
2. Choose Orchestrated by Amazon EKS from the list.
3. Select Quick setup to provision a cluster with default networking, storage, and IAM resources, or choose Custom setup to integrate with existing VPC, subnets, and security groups.
4. Under Orchestration , either create a new EKS cluster or attach an existing one. Verify that the Use default Helm charts and add-ons option is selected so that the Inference Operator and other required operators are installed automatically.
5. Under Instance groups , add a worker group configured with the ml.p6-b300.48xlarge instance type.
6. Review the configuration and choose Submit to begin provisioning.

For the complete walkthrough, refer to the Creating a SageMaker HyperPod cluster with Amazon EKS orchestration documentation.

Step 2: Procure capacity using a Flexible Training Plan

The ml.p6-b300.48xlarge instance type requires reserved capacity. A Flexible Training Plan provides a committed capacity reservation for your Blackwell GPU nodes, guaranteeing that p6-b300 instances are available to your cluster without contention from the general on-demand pool. Go to SageMaker Console, select a FTP block based on your timeline and instance count. To create or attach a training plan:

1. In the instance group configuration, choose Training plan as the capacity source.
2. Select an existing plan that covers ml.p6-b300.48xlarge capacity, or create a new reservation specifying the instance count and duration that you want.
3. Set the Target Availability Zone to match the zone where your training plan capacity is allocated.

Once the cluster reaches an Active state with healthy p6-b300 nodes, you are ready to deploy the model.

Deploying the model

To deploy Kimi K3 on HyperPod, apply the following InferenceEndpointConfig manifest to your cluster:

apiVersion: inference.sagemaker.aws.amazon.com/v1
kind: InferenceEndpointConfig
metadata:
  name: kimik3
spec:
  modelName: Kimi-K3
  instanceType: ml.p6-b300.48xlarge
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    huggingFaceModel:
      modelId: moonshotai/Kimi-K3
    modelSourceType: huggingface
  worker:
    image: vllm/vllm-openai:kimi-k3
    modelInvocationPort:
      containerPort: 8000
      name: http
    modelVolumeMount:
      mountPath: /opt/ml/model
      name: model-weights
    resources:
      limits:
        nvidia.com/gpu: 8
      requests:
        nvidia.com/gpu: 8
    args:
      - "--model"
      - "moonshotai/Kimi-K3"
      - "--trust-remote-code"
      - "--load-format"
      - "fastsafetensors"
      - "--enable-prefix-caching"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser"
      - "kimi_k3"
      - "--reasoning-parser"
      - "kimi_k3"
      - "--served-model-name"
      - "Kimi-K3"
      - "--moe-backend"
      - "auto"
      - "--tensor-parallel-size"
      - "8"
      - "--no-enable-flashinfer-autotune"
    environmentVariables:
      - name: "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION"
        value: "1"

This yaml is also provided in the GitHub repository. Apply this configuration with:kubectl apply -f kimi-k3.yaml

The Inference Operator handles model download from Hugging Face, container scheduling, health checks, and endpoint readiness. Once the endpoint transitions to a ready state, it exposes an OpenAI compatible API at the configured invocation path.

Option 2: Deploying on Amazon EKS

For teams that prefer to manage their own Kubernetes infrastructure, you can deploy Kimi K3 on a standalone Amazon EKS cluster and procure GPU capacity through EC2 Capacity Blocks , which allow you to reserve p6-b300 instances for a defined duration without long-term commitments.

Key deployment steps

The AI on EKS project provides an inference-ready cluster recipe that automates the end-to-end provisioning. At a high level, the deployment involves the following stages:

1. Provision the EKS cluster

Use the provided Terraform modules to create a GPU-optimized EKS cluster. This includes VPC networking, managed node groups, and the necessary IAM roles and policies for GPU workloads.

2. Reserve GPU capacity with Capacity Blocks

Create a Capacity Block reservation for p6-b300.48xlarge instances in your target Availability Zone. Capacity Blocks guarantee that the requested GPU nodes will be available for the reserved time window. Once the reservation becomes active, the instances join your EKS cluster as worker nodes.

3. Install GPU drivers and device plugin

The recipe installs the NVIDIA device plugin and GPU drivers on the node group, so that Kubernetes can discover and schedule against the available GPUs.

4. Deploy the vLLM inference server

A Helm chart or Kubernetes manifest deploys the vLLM container with Kimi K3 specific arguments, including tensor-parallel size of 8, the MXFP4 load format, MoE backend configuration. The model identifier points to the Hugging Face repository, or as an alternative, you can sync the model weights to Amazon Simple Storage Service (Amazon S3) for faster model loading. The serving arguments mirror those shown in the HyperPod configuration above.

5. Expose the inference endpoint

A Kubernetes Service (type LoadBalancer or via an Ingress controller) exposes the vLLM server on port 8000, providing the OpenAI compatible /v1/chat/completions endpoint to your applications.

6. Validate

Confirm the deployment by sending a test request to the endpoint and verifying a successful model response.

For the full deployment walkthrough, including Terraform modules, Helm values, and step-by-step instructions, refer to the AI on EKS Kimi K3 recipe.

Invoking the endpoint

Once deployed, the Kimi K3 endpoint exposes an OpenAI compatible chat completions API. You can invoke it using the OpenAI Python SDK or a simple curl command.

Using the OpenAI Python SDK

from openai import OpenAI
client = OpenAI(
    base_url="http://<ENDPOINT_URL>:8000/v1",
    api_key="not-needed"
)


response = client.chat.completions.create(
    model="Kimi-K3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.choices[0].message.content)

Using curl

curl -X POST http://<ENDPOINT_URL>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Kimi-K3",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
  }'

Replace <ENDPOINT_URL> with the service endpoint exposed by your HyperPod Inference Operator or EKS ingress configuration.

Clean up

To avoid ongoing charges, delete the resources you created during this walkthrough when you no longer need them.For SageMaker HyperPod deployments:

1. Delete the InferenceEndpointConfig by running kubectl delete -f kimi-k3.yaml.
2. In the SageMaker AI console, navigate to HyperPod Clusters, select your cluster, and choose Delete.
3. Release or cancel your Flexible Training Plan reservation if it is no longer needed.

For Amazon EKS deployments:

4. Delete the vLLM deployment and associated Kubernetes services.
5. Terminate the GPU node group or delete the EKS cluster using Terraform (terraform destroy).
6. Release your Capacity Block reservation if it has not yet expired.
7. For pricing details on p6-b300 instances and Capacity Block reservations, refer to the Amazon EC2 pricing page.

Conclusion

Kimi K3 represents a new frontier in open-weight model capabilities, and AWS provides the infrastructure and managed services to deploy it at scale. Whether you choose the streamlined HyperPod Inference Operator path or the flexibility of a self-managed EKS cluster, the combination of p6-b300 GPU instances, vLLM serving, and MXFP4 quantized weights delivers a deployment with built-in health checks, auto-recovery, and endpoint readiness verification for the world’s largest open model.To get started, here are some links

1. SageMaker HyperPod Kimi K3 Example
2. AI on EKS Kimi K3 Recipe
3. Creating a SageMaker HyperPod Cluster (AWS Documentation)
4. Kimi K3 on Hugging Face
5. vLLM Documentation

---

About the authors

Andrew Smith is a Sr. Cloud Support Engineer in the SageMaker, Vision & Other team at AWS, based in Sydney, Australia. He supports customers using many AI/ML services on AWS with expertise in working with Amazon SageMaker. Outside of work, he enjoys spending time with friends and family as well as learning about different technologies.

Erez Zarum is a Senior Startups Solutions Architect at AWS. Erez is passionate about Containers and the AI/ML landscape, and his unique approach empowers Startups to accelerate AI/ML workloads on Amazon EKS.

Vivek Gangasani is a Worldwide Leader for Solutions Architecture, SageMaker Inference. He leads Solution Architecture, Technical Go-to-Market (GTM) and Outbound Product strategy for SageMaker Inference. He also helps enterprises and startups deploy and optimize a GenAI models and build AI workflows with SageMaker and GPUs. Currently, he is focused on developing strategies and content for optimizing inference performance and use-cases such as Agentic workflows, RAG etc. In his free time, Vivek enjoys hiking, watching movies, and trying different cuisines