Agent Substrate：面向大规模 Agent 部署的高密度运行时系统

注意：这不是 Google 官方支持的产品。本项目不符合 Google 开源软件漏洞奖励计划的参与资格。

Agent Substrate 为大规模 Agent 部署提供高性能、高密度的运行时环境。其 Agent Substrate 控制平面为 Agent 沙箱提供完整的生命周期管理，支持亚秒级的 Agent 恢复/挂起操作，并允许在同一计算基础设施上对 Agent 进行高度复用。它支持包括微虚拟机和 gVisor 在内的多种沙箱技术，从而为各类沙箱提供一致的生命周期操作。

Agent Substrate 的核心机制是：将更多的“Actor”（如 Agent 等应用）映射到更少的就绪“Worker”上。它利用 Agent 类应用大部分时间处于空闲状态这一特点，实现高程度复用。系统提供管理 Actor 生命周期的能力（例如创建/销毁、挂起/恢复），可实时将 Actor 分配给 Worker，并将入站流量路由至相应 Actor。

Agent Substrate 被设计为一个低约束、低预设立场的系统。它所管理的工作负载不必是字面意义上的 AI Agent，但 AI Agent 是最能说明其适用场景的一类应用。它不是用于构建 Agent 的 SDK，而是一个用于大规模运行 Agent 的系统。

Agent Substrate 使用 Kubernetes 进行基础设施预置和 Worker 生命周期管理（Kubernetes Pod）。它构建于 Pod、Pod 自动扩缩容等 Kubernetes 功能之上，同时由 Agent Substrate 提供面向 Agent 的调度与控制，以实现更低延迟。以 Kubernetes 作为底层系统，可为端到端 Agent 化部署所需的所有工作负载类型提供一致的基础设施管理，并支持针对横跨 Agent、推理和训练周期的 RL 场景进行整体基础设施优化。

视频：Agent Substrate 演示

观看 Agent Substrate 集群仅通过 8 个物理 Pod 复用承载约 250 个有状态 Actor。

该演示突出展示了 Substrate 的核心开发者体验和“Agentic Infrastructure（Agent 化基础设施）”能力：

1. Actor 即时迁移： 可将 Actor 高性能地挂起并恢复至资源池中任意可用 Worker，激活时间低于 1 秒。
2. 状态持久化： 通过完整状态快照，在休眠周期之间完整保留持久化工作内存（易失性 RAM）和文件系统状态。
3. Agent 群体复用： 通过在小型共享物理 Pod 池上“调度周转”大量有状态 Actor 注册表，展示超过 30 倍的超额订阅能力。

如要在自己的集群中复现此演示，请参阅详细的 Counter Demo 操作指南。

如需更多视频和操作指南，请访问我们的 YouTube 频道：agent-substrate。

框架无关性与兼容性

Agent Substrate 采用框架和 Agent Harness 无关的设计。由于它在内核层面管理标准 OCI 容器（通过 gVisor），因此可以托管使用任意技术栈构建的 Agent。

• Agent Development Kit (ADK)： 原生支持与 ADK 兼容的 Actor 身份及持久化工作内存。
• LangChain： 为长时间运行、有状态的 LangChain Agent 以及沙箱化工具调用提供理想执行环境。
• Claude Code 与 CodeX： 支持高密度、有状态的编程环境，可跨会话保留终端和文件系统状态。
• Model Context Protocol (MCP)： 将安全、沙箱化的 MCP 服务器部署为 Substrate Actor，为任何 LLM 提供持久化工具。

Agent Substrate 目前仍处于早期开发阶段，尚未准备好用于生产环境，其 API 几乎肯定会发生变化。当前阶段不对向后兼容性作出任何保证，项目中的一切内容都可能被修改。

支持的 Kubernetes 版本

目前，我们的目标是支持 Kubernetes 的最新稳定版本以及此前的一个次要版本。

社区

如需获取公告、参与技术讨论或获得社区支持，请加入 ate-dev Google Group。

我们每周四太平洋标准时间上午 10:00 至 11:00 举办社区例会。

我们也在 CNCF Slack 中设有频道；如果你尚无访问权限，可在此申请邀请。

有关为项目贡献代码的指南，请参阅 CONTRIBUTING.md。我们欢迎各种形式的贡献，但该项目仍然非常早期。我们当前的重点是完善核心系统和演示，因此近期可能无法审查或合并与这些目标不一致的贡献。

要快速搭建完整环境：

1. 确保你的开发机器已安装并配置 Go、kubectl 和 docker。我们会通过 Go 自动管理其他依赖，包括 kind。

2. 运行以下步骤：

# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
hack/create-kind-cluster.sh

# install ate, valkey, rustfs
hack/install-ate-kind.sh --deploy-ate-system

# install counter demo
hack/install-ate-kind.sh --deploy-demo-counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create an atespace (required before creating actors), then a counter actor in it
kubectl ate create atespace demo
kubectl ate create actor my-counter-1 -a demo --template=ate-demo-counter/counter

# port-forward the network router to bind to local port `8000`
kubectl port-forward -n ate-system svc/atenet-router 8000:80

3. 在另一个终端中，发送 HTTP 请求以递增计数器：

curl -X POST -H "Host: my-counter-1.demo.actors.resources.substrate.ate.dev" -i http://localhost:8000/

GKE 快速入门（开发环境）

1. 创建并配置环境文件：

   cp hack/ate-dev-env.sh.example .ate-dev-env.sh
   
   # Edit .ate-dev-env.sh to match your project and preferences, then source it:
   source .ate-dev-env.sh
   

2. 为 gcloud 启用应用默认凭证：

   gcloud auth application-default login --project=${PROJECT_ID}
   

3. 预置所需的 GCP 资源（GKE 集群、Redis、GCS 和 IAM 绑定）：

   go run ./tools/setup-gcp bootstrap
   

4. 将 Agent Substrate 系统部署到集群：

   ./hack/install-ate.sh --deploy-ate-system
   

5. 随后即可部署示例应用。详细操作指南请参阅 demos/counter/README.md 或 demos/sandbox/README.md。

   ./hack/install-ate.sh --deploy-demo-counter
   

自定义设置与部署

你可以按需运行各个独立的设置步骤来创建 GCP 资源。可通过 go run ./tools/setup-gcp --help 查看可用选项。例如：

go run ./tools/setup-gcp create cluster
go run ./tools/setup-gcp create bucket

同样，你可以使用安装脚本部署或清理指定的 Agent Substrate 组件。通过 ./hack/install-ate.sh --help 查看全部选项。

# Re-deploy only ate-apiserver of the ATE system
./hack/install-ate.sh --deploy-ate-apiserver

# Delete everything (core system and all demos)
./hack/install-ate.sh --delete-all

拆除资源（GCP）

如果需要删除由设置脚本创建的资源，可以使用提供的 hack/teardown.sh 脚本。该脚本会以与创建相反的顺序删除资源，并可妥善处理部分失败的情况。

或者，也可以按需运行单独的拆除步骤（可通过 ./hack/teardown.sh 查看可用选项）。

拆除本地 kind 资源

如果需要删除本地 kind 集群及其镜像仓库（前提是它们由 hack/create-kind-cluster.sh 创建）：

./hack/delete-kind-cluster.sh

我们提供了多个用于展示 Agent Substrate 能力的示例应用：

1. Counter Demo ：一个有状态 Go HTTP 服务器，展示跨挂起/恢复操作的状态保留，以及动态 CRD 路由。
2. Sandbox Demo (Antigravity) ：安全的沙箱化执行环境（运行 Alpine Linux），允许任意 Shell 执行，同时可跨会话保留文件系统状态。
3. Claude Code Multiplex ：展示如何通过在有限 Worker 池上复用多个 Claude Code Agent，实现对物理硬件的超额订阅。
4. Multi-Template ：两个运行不同二进制文件的 ActorTemplate，跨三个命名空间共享一个 WorkerPool。
5. Request Parking ：一个超额订阅的资源池，路由器会在 Worker 可用前暂存入站请求，而不是返回 503。
6. Autoscaled WorkerPool ：使用由 prometheus-adapter 提供指标的 HPA，根据已分配 Worker 数量对 WorkerPool 进行扩缩容。

• 架构：控制平面、节点监督器和网络栈如何协同工作。

• API 配置指南：配置 WorkerPool、ActorTemplate、Secret 和 Volume 的详细参考。

• 完整 CLI 文档：kubectl-ate 的安装与使用说明。

• 术语表：核心术语（Actor、Atespace、ActorTemplate、WorkerPool、Worker、ate-api-server、atenet、atelet、ateom）及其关系。

• 集成仓库：集成代码所在位置、仓库命名方式，以及修复如何回流至核心项目。

• 可观测性指南：Actor 日志、指标和分布式追踪指南。

• 身份验证指南：配置受信任的 JWT 提供方和人工用户凭证。

• 请求暂存：路由器如何在 WorkerPool 短暂饱和期间暂存请求。

• 威胁模型：信任边界、假设条件和已知风险。

• 路线图：当前限制及后续计划。

• 基准测试指南：基于 Locust 的负载测试、监控栈和编排式基准测试工具。

• cmd/ateapi：核心控制平面 API 服务器，通过 gRPC 端点管理 Actor 和 Worker 生命周期。

• cmd/atelet：节点级 DaemonSet，负责监督物理 Worker Pod、协调快照操作并管理状态传输。

• cmd/atecontroller：用于协调 WorkerPool 和 ActorTemplate 自定义资源的 Kubernetes 控制器。

• cmd/atenet：组合式网络控制器，提供 DNS、Envoy 路由和代理 Sidecar。

• cmd/ateom-gvisor：运行在沙箱化 Worker Pod 内的 Pod 内部辅助程序，用于执行 runsc checkpoint 和 restore 命令。

• cmd/ateom-microvm：ateom-gvisor 的微虚拟机对应组件，以 cloud-hypervisor VM 的形式运行 Actor。

• cmd/podcertcontroller：一个“polyfill”，提供最终将以不同名称进入上游 Kubernetes 的 Pod Certificate 签名器。

• cmd/kubectl-ate：用于管理 Agent Substrate 资源的 CLI 工具。参阅其 README。

• cmd/benchmarking：负载测试使用的合成工作负载，包括可按需消耗 RAM、磁盘和文件描述符的 glutton。

• tools/setup-gcp：用于设置必要 GCP 基础设施资源（GKE、GCS、IAM）的预置工具。

• demos/：展示 Agent Substrate 能力的示例应用。

NOTE: This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.

Agent Substrate delivers a performant, high density runtime environment for large scale agent deployments. The agent substrate control plane provides full lifecycle management for agent sandboxes, delivering sub-second agent resume/suspend operations, and allows heavy multiplexing of agents onto the same computer infrastructure. It supports multiple sandbox technologies including microVMs and gVisor, enabling consistent lifecycle operations for all sandbox types.

At its core, Agent Substrate maps a larger set of “actors” (applications such as agents) onto a smaller set of ready “workers”, relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing. It provides functionality to manage an actor’s lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real time, and to route incoming traffic to them.

Agent Substrate is intended to be a low-opinion system. The workloads it manages don't have to be literal AI agents, but those are the best example of the kind of applications it is designed for. It is not an SDK for building agents, but rather a system for running them at scale.

Agent Substrate leverages Kubernetes for the infrastructure provisioning and worker lifecycle management (Kubernetes Pods). It builds on top of Kubernetes features like Pods and Pod autoscaling, while Agent Substrate provides agent-specific scheduling and control to achieve lower latency. Using Kubernetes as the underlying system enables consistent infrastructure management across all workloads types that are required for end to end agentic deployments and allows holistic infrastructure optimizations for RL scenarios that span agentic, inference and training cycles.

Watch the Agent Substrate cluster multiplex ~250 stateful actors across just 8 physical pods.

This demo highlights the core developer experience and "Agentic Infrastructure" capabilities of Substrate:

1. Instant Actor Teleport: High-performance suspend and resume of actors onto any available worker in the pool with sub-second activation.
2. State Persistence: Persistent working memory (volatile RAM) and filesystem state preserved perfectly across hibernation cycles via full-state snapshots.
3. Agent Swarm Multiplexing: Demonstrates 30x+ oversubscription by "juggling" a large registry of stateful actors onto a small pool of shared physical pods.

To reproduce this demo in your own cluster, please refer to the detailed walkthrough in the Counter Demo.

For more videos and walkthroughs, visit our YouTube channel: agent-substrate.

Framework Agnostic & Compatibility

Agent Substrate is designed to be framework and agent harness agnostic. Because it manages standard OCI containers at the kernel level (via gVisor), it can host agents built on any stack.

• Agent Development Kit (ADK): Native support for ADK-compatible actor identity and persistent working memory.
• LangChain: Ideal execution environment for long-running, stateful LangChain agents and sandboxed tool-calling.
• Claude Code & CodeX: Support for high-density, stateful coding environments that preserve terminal and filesystem state across sessions.
• Model Context Protocol (MCP): Deploy secure, sandboxed MCP servers as Substrate Actors to provide durable tools for any LLM.

Agent Substrate is currently in early development. It is not ready for production use, and the APIs are almost guaranteed to change. We are not making any guarantees about backward compatibility at this stage, and everything in this project may be changed.

Supported Kubernetes Releases

Currently we aim to support the latest stable release of Kubernetes, and the previous minor release.

Community

For announcements, technical discussions, and community support, please join the ate-dev Google Group.

We host a weekly community meeting every Thursday from 10:00am - 11:00am PST.

We also have channels in the CNCF slack; request an invite here if you don't have access.

Please see CONTRIBUTING.md for guidelines on contributing to the project. We welcome contributions of all kinds, but the project is VERY young. Our immediate focus is on building out the core system and demos, so we may not be able to review or merge contributions that don't align with those goals in the near term.

To quickly set up the complete environment:

1. Make sure you have Go, kubectl, and docker installed and configured on your dev machine. We will automatically manage other dependencies via Go, including kind.

2. Run the following steps:

# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)
hack/create-kind-cluster.sh

# install ate, valkey, rustfs
hack/install-ate-kind.sh --deploy-ate-system

# install counter demo
hack/install-ate-kind.sh --deploy-demo-counter

# install kubectl-ate
go install ./cmd/kubectl-ate

# create an atespace (required before creating actors), then a counter actor in it
kubectl ate create atespace demo
kubectl ate create actor my-counter-1 -a demo --template=ate-demo-counter/counter

# port-forward the network router to bind to local port `8000`
kubectl port-forward -n ate-system svc/atenet-router 8000:80

3. In a separate terminal , send an HTTP request to increment the counter:

curl -X POST -H "Host: my-counter-1.demo.actors.resources.substrate.ate.dev" -i http://localhost:8000/

GKE Quickstart (Development)

1. Create and configure your environment file:

   cp hack/ate-dev-env.sh.example .ate-dev-env.sh
   
   # Edit .ate-dev-env.sh to match your project and preferences, then source it:
   source .ate-dev-env.sh
   

2. Enable application-default credentials for gcloud:

   gcloud auth application-default login --project=${PROJECT_ID}
   

3. Provision the required GCP resources (GKE cluster, Redis, GCS, and IAM bindings):

   go run ./tools/setup-gcp bootstrap
   

4. Deploy the Agent Substrate system to your cluster:

   ./hack/install-ate.sh --deploy-ate-system
   

5. You can then deploy the sample applications. See demos/counter/README.md or demos/sandbox/README.md for detailed walkthroughs.

   ./hack/install-ate.sh --deploy-demo-counter
   

Custom Setup and Deployment

You can run individual setup steps to create GCP resources as needed. See go run ./tools/setup-gcp --help for available options. For example:

go run ./tools/setup-gcp create cluster
go run ./tools/setup-gcp create bucket

Similarly, you can deploy or cleanup specific Agent Substrate components using the installation script. See ./hack/install-ate.sh --help for all options.

# Re-deploy only ate-apiserver of the ATE system
./hack/install-ate.sh --deploy-ate-apiserver

# Delete everything (core system and all demos)
./hack/install-ate.sh --delete-all

Tearing down resources (GCP)

If you need to delete the resources created by the setup script, you can use the provided script hack/teardown.sh. This script will delete resources in the reverse order of creation and handles partial failures gracefully.

Or run individual teardown steps as needed (see ./hack/teardown.sh for available options).

Tearing down local kind resources

If you need to delete the local kind cluster and its registry (if it was created by hack/create-kind-cluster.sh):

./hack/delete-kind-cluster.sh

We provide several sample applications demonstrating Agent Substrate's capabilities:

1. Counter Demo : A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and dynamic CRD routing.
2. Sandbox Demo (Antigravity) : A secure, sandboxed execution environment (running Alpine Linux) that allows arbitrary shell execution while preserving filesystem state across sessions.
3. Claude Code Multiplex : Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers.
4. Multi-Template : Two ActorTemplates running different binaries share one WorkerPool, across three namespaces.
5. Request Parking : An oversubscribed pool where the router holds inbound requests until a worker frees up, instead of returning 503.
6. Autoscaled WorkerPool : Scales a WorkerPool on its assigned-worker count with an HPA fed by prometheus-adapter.

• Architecture: How the control plane, node supervisor, and networking stack fit together.

• API Configuration Guide: Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes.

• Full CLI Documentation: Installation and usage for kubectl-ate.

• Glossary: Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateom) and how they relate.

• Integration Repositories: Where integrations live, how their repositories are named, and how fixes flow back to core.

• Observability Guide: Guide to actor logging, metrics, and distributed tracing.

• Authentication Guide: Configure trusted JWT providers and human credentials.

• Request Parking: How the router parks requests through transient worker-pool saturation.

• Threat Model: Trust boundaries, assumptions, and known risks.

• Roadmap: Current limitations and what is planned next.

• Benchmarking Guide: Locust-based load tests, monitoring stack, and the orchestrated benchmark harness.

• cmd/ateapi: The core control plane API server exposing gRPC endpoints to manage actor and worker lifecycles.

• cmd/atelet: A node-level DaemonSet that supervises physical worker pods, coordinates snapshotting, and manages state transfers.

• cmd/atecontroller: A Kubernetes controller that reconciles WorkerPool and ActorTemplate custom resources.

• cmd/atenet: A combined networking controller providing DNS, Envoy routing, and proxy sidecars.

• cmd/ateom-gvisor: An interior-pod helper running inside sandboxed worker pods to execute runsc checkpoint and restore commands.

• cmd/ateom-microvm: The micro-VM peer of ateom-gvisor, running actors as cloud-hypervisor VMs.

• cmd/podcertcontroller: A "polyfill" that provides Pod Certificate signers that will eventually ship in upstream Kubernetes (with different names).

• cmd/kubectl-ate: A CLI tool for managing Agent Substrate resources. See its README.

• cmd/benchmarking: Synthetic workloads used by the load tests, including glutton, which consumes RAM, disk, and file descriptors on demand.

• tools/setup-gcp: A provisioning utility to set up the necessary GCP infrastructure resources (GKE, GCS, IAM).

• demos/: Sample applications demonstrating Agent Substrate capabilities.