构建超过 100 万个云端 Agent 后，我们学到了什么

所谓“云端 Agent”，是指其运行时托管在远程环境中，并且能够独立于本地机器上的任何进程运行的 Agent。

这类 Agent 具有多种形态1：Meta Muse、Claude Tag、OpenAI Codex Cloud、Sierra、ElevenLabs Agents、Higgsfield Supercomputer。

组织应当能够针对自身的使用场景，构建和部署这类 Agent，并获得所需的能力与安全保障。

我们发现，相较于单体式解决方案或 SaaS 产品，灵活、可组合的基础设施组件更适合作为云端 Agent 的基础。

过去一年里，我们构建并经过实战检验了这些基础设施组件，并将其中之一——Unreal Agent harness——开源。今天，我们想讨论云端 Agent 技术栈的其余部分。

在 Unreal Labs，我们帮助企业针对其使用场景构建、部署和评估云端 Agent。欢迎联系：contact@unreallabs.ai

把玩 Agent 作为一种爱好很有趣。你买一台 Mac mini，将 OpenAI 订阅连接到 Hermes 之类的 Agent，点击完成基础设置，瞧，你就可以与这个帮你处理各种事务的小家伙互动了。

但为真实企业构建 Agent 就没那么轻松了。你需要考虑智能 AI 可能做出的所有边界情况、成本控制和评估，这会让人头昏脑涨。别担心，今天我们会一步一步讲清楚。

首先，你是否预计系统中会有不止一名用户与 Agent 交互？如果是，那么你需要将不同 Agent 相互隔离。最佳方式是使用沙箱2。

如今已有许多沙箱解决方案。本文不会深入介绍它们，但有几个高层面的问题值得思考：

• 你是将 Agent harness 放在沙箱内部，还是放在沙箱外部？
• 如果 harness 位于沙箱外部，你的沙箱提供商是否允许暂停沙箱以降低成本？基于 Firecracker microVM 的沙箱通常支持这一点。根据工作负载不同，这可以降低运行 Agent 的成本。例如，可以在推理期间暂停沙箱。

沙箱为 Agent 提供专属工作空间，但这个空间可能是临时的。组织的上下文、技能和持久记忆，需要在每次 Agent 启动时始终可用。

我们见过多种 Agent 文件存储方案，主要包括持久卷和 S3 快照。还有一种看似被忽视、但我们认为非常适合许多内部部署的方式：通过 NFS 挂载子目录。我们将这类存储视为用于读取的大型缓存，以及用于写入的暂存区域。

这样可以快速准备沙箱，同时，挂载和权限能够在不同 Agent 的工作区之间提供基础隔离。

例如，对话型 Agent 需要访问此前的消息。我们将消息历史以只读目录的形式提供，使 Agent 可以查阅这些内容，而不能修改底层记录。数据库仍然是对话历史的唯一事实来源。

Agent 应当跨运行周期记住什么？对于部分 Agent，维护者会选择并管理可供 Agent 使用的持久文档。另一些 Agent 则自行维护记忆，并需要一种方式根据既往工作更新这些记忆。

对于自行管理记忆的持久化 Agent，另一个独立的后台 Agent 可以审查日志和已有记忆，再更新组织的共享记忆。这个过程通常被称为“做梦”（dreaming）。我们更倾向于称之为“冥想”（meditation）：这个名称更能体现其回顾过去经验、整理 Agent 所学内容的过程。

Agent 需要访问 Google Drive 等外部服务，以及 CMS 等内部服务。这些请求通常需要凭证（认证信息），由此带来一个问题：如何让 Agent 使用某个账户，却不把该账户的秘密信息交给它？

出口代理（egress proxy）可以处理这一问题。所有离开沙箱的流量都会经过它，无论目标是外部 API，还是系统中的其他服务。该代理可以：

• 拦截请求、检查请求目标，并注入所需凭证。
• 阻止 Agent 无权访问的目标请求。
• 检查和修改请求正文。

出于安全等原因，最好将凭证管理与存储同代理服务分离。代理会针对每个请求从 Credential Broker 获取凭证；Credential Broker 会检查请求方是否有权发起该类型调用。凭证会在请求离开沙箱后才被添加，因此 Agent 永远不会接触这些凭证。

Credential Broker 负责管理账户连接。它处理账户接入，检查谁可以使用每个连接，并在适用情况下在后台刷新认证令牌。

将它与出口代理分离的另一个原因是，账户连接会在整套技术栈中共享。代理需要凭证来转发 Agent 请求，其他服务则需要凭证来完成诸如通过 Slack 通信等工作。专用 Broker 能让这些组件共享连接和访问规则，而不必各自实现凭证管理。

对于 API 密钥和令牌存储，我们发现 OpenBao——Linux Foundation 对 Vault 的一个分支——既可靠，又易于部署和管理。但存储机密只是其中一部分工作。Credential Broker 还会追踪由谁连接了每个账户、连接状态，以及凭证存储在何处。我们将这些记录保存在 Postgres 中。

消息传递是这些共享账户连接的使用场景之一。已连接的 Slack 账户可以让系统进行通信，但我们仍需决定哪些消息需要 Agent 关注，以及 Agent 的回复应发送到哪里。

Messaging Gateway 通过统一接口处理 Slack、iMessage、Telegram、WhatsApp、内部聊天、Instant Bloomberg 或任何其他消息服务。其职责包括：

• 传入消息：将需要 Agent 关注的消息，例如 Slack 提及，路由至 Agent Orchestrator。
• 传出消息：将 Agent 的响应发送到指定频道或线程，并检查目标是否存在。
• 对话存储：持久化保存并检索属于每个组织或用户的对话。
• 消息权限：执行关于 Agent 可以向哪些频道发送消息的规则。

一条传入消息到达时，可能遇到空闲 Agent、已经在工作的 Agent，或者根本不存在正在运行的沙箱。Agent Orchestrator 负责处理这些情况：它准备环境、管理执行，并在 Agent 就绪后投递排队的消息。

其主要职责包括：

• 创建可访问所需文件和服务的沙箱
• 准备 Agent、启动 Agent 并管理其生命周期
• 保存执行轨迹并流式传输对话输出

它还负责管理计划任务、跟踪执行状态，并清理空闲沙箱。

传入消息会先经过一个队列，因此即使 Agent 正在工作，新请求也能到达。当有人发送后续指令，或多个人在共享频道中与 Agent 交谈时，这一点尤为重要。Messaging Gateway 决定哪些消息进入队列，编排器则在 Agent 准备就绪时将其投递。

自定义聊天界面是可选的。一些 Agent 完全通过 Slack 等消息平台与用户交互。另一些则需要在 Web 或移动应用中提供自己的界面。对于后者，编排器可以存储对话输出并流式推送实时更新，让用户能够重新打开对话、回顾结果，并实时跟进消息、工具调用和推理过程。

以 Web 聊天为例。在这类部署中，PostgreSQL 为对话历史提供持久化存储，而 Redis Streams 则承载活跃对话及其实时更新。这使得 Web 界面能够加载历史记录并跟进正在进行的工作，而不会让较慢的存储成为瓶颈。

在构建 Web 界面时，assistant-ui 是一个不错的选择。它的 external-store runtime 可连接到应用自身的后端，而其消息格式也足够灵活，能够支持不同客户端的需求。

围绕 Agent 的基础设施依赖 harness 来执行工具并报告运行情况。

harness 负责运行 Agent 循环：调用模型、执行工具并维护对话状态。

两家提供商都提供了第一方 Agent SDK：Anthropic 提供 Claude Agent SDK，OpenAI 则提供 OpenAI Agents SDK 和 Codex SDK。

根据我们的经验，这些 SDK 仍在持续演进。随着 SDK 版本和模型变化，以及新功能启用，维护集成需要持续投入。我们遇到了几项挑战：

• **模型兼容性：**Claude Agent SDK 是为 Claude 和 Anthropic 的 API 构建的。通过 Fireworks 的 Anthropic 兼容端点运行 Kimi K3 时，我们发现了工具调用 ID 冲突问题。直接测试表明，尽管工具成功执行，重复使用的 ID 仍会导致 Claude Code 丢弃后续模型请求的结果。API 兼容性本身并不保证 harness 兼容性。
• **保持模型与运行时同步：**较新的模型可能无法很好地配合较旧版本的 harness 和 SDK，因此必须升级。此类升级也可能要求修改集成逻辑：例如，后台任务需要新的完成逻辑，以避免过早退出和挂起。
• **工具限制：**禁用某项工具时，也应移除要求使用该工具的指令。在 Claude Agent SDK 中，阻止 AskUserQuestion 会移除其定义，但规划工具仍然会要求模型调用它。遵循这些指令可能导致工具调用被拒绝；由于 harness 自身存在冲突指令，这既浪费 token，也增加成本。

这些 SDK 均为开源项目，我们已向 Claude Agent SDK 贡献了多项修复。

这些经验，加上我们对于 harness 应如何运作的想法，促使我们开发了 Unreal Agent。请在我们的博客中了解更多关于 Unreal Agent 的信息。

我们介绍了用于构建云端 Agent 的基础设施，以及在生产环境运行它们时遇到的一些问题。不同 Agent 需要这套技术栈中的不同部分。我们相信，拥有灵活且可在必要时复用的组件，会更容易应对这些差异。

如果你正在构建云端 Agent，我们很乐意与你合作！在 Unreal Labs，我们帮助企业根据其使用场景构建、部署和评估 Agent。欢迎联系：contact@unreallabs.ai。

By “cloud agent,” we mean an agent whose runtime is hosted remotely and can operate independently of any process on your local machine.

These agents take many forms1: Meta Muse, Claude Tag, OpenAI Codex Cloud, Sierra, ElevenLabs Agents, Higgsfield Supercomputer.

Organizations should be able to build and deploy agents like these for their own use cases, with the capabilities and security guarantees they need.

We found that flexible, composable infrastructure components provide a better foundation for cloud agents than a monolithic solution or SaaS product.

Over the past year, we have built and battle-tested these infrastructure components and open-sourced one of them: the Unreal Agent harness. Today, we want to discuss the rest of the cloud agent stack.

At Unreal Labs, we help companies build, deploy and evaluate cloud agents for their use cases. Get in touch: contact@unreallabs.ai

Tinkering with agents as a hobby is fun. You purchase a Mac mini, connect your OpenAI subscription to an agent like Hermes, click through a basic setup, and voila, you can interact with the little guy (or gal!) helping you with whatever.

Building agents for real businesses is less fun. Thinking through all the edge cases of what smart AIs can do, cost controls, and evals makes your head spin. Don’t worry, today we’re going through all this step by step.

First off, do you expect more than one user interacting with agents in your system? Great, then you need to isolate the agents from one another. The best way to do that is to use sandboxes2.

There are many sandbox solutions available today. We won’t cover them in depth in this post, but here are a few high-level thoughts worth entertaining:

• Are you putting an agent harness inside the sandbox or outside of it?
• If the harness lives outside the sandbox, does your sandbox provider allow you to pause the sandbox and reduce costs? Sandboxes based on Firecracker microVMs usually allow you to do so. Depending on the workload, you can reduce the cost of running agents. For example, pausing the sandbox during inference.

A sandbox gives an agent a dedicated space to work, but that space may be temporary. The organization’s context, skills, and persistent memory need to remain available whenever the agent starts.

We’ve seen several approaches to file storage for agents. The main ones are persistent volumes and S3 snapshots. One approach that seems overlooked, but that we think suits many internal deployments, is mounting subdirectories over NFS. We treat this storage as a large cache for reads and a staging area for writes.

This makes sandbox preparation fast, while mounts and permissions provide baseline isolation between agents’ workspaces.

For example, conversational agents need access to previous messages. We expose a view of message history as a read-only directory, so agents can consult it without modifying the underlying records. The database remains the source of truth for conversation history.

What should an agent remember across runs? For some agents, maintainers choose and manage the persistent documents available to the agent. Others maintain their own memory and need a way to update it from past work.

For persistent agents that manage their own memory, a separate background agent can review logs and existing memories, then update the organization’s shared memory. This process is often called dreaming. We prefer meditation: the name better captures reflecting on past experience and organizing what the agent has learned.

The agent needs to access external services such as Google Drive and internal services such as a CMS. These requests often require credentials (auth), which poses a problem: how do we let the agent use an account without giving it the account’s secrets?

An egress proxy handles this. All traffic leaving the sandbox passes through it, whether the destination is an external API or another service in your system. The proxy can:

• Intercept requests, check their destination, and inject the required credentials.
• Block requests to destinations the agent isn’t allowed to access.
• Inspect and modify request bodies.

It’s best to keep credential management and storage separate from the proxy service, partly for security reasons. The proxy retrieves the credentials for each request from the Credential Broker, which checks whether the requester is permitted to make that type of call. Credentials are added after the request leaves the sandbox, so the agent never handles them.

The Credential Broker manages account connections. It handles connecting accounts, checks who can use each connection, and refreshes auth tokens in the background, where applicable.

Another reason to keep it separate from the egress proxy is that account connections are shared across the stack. The proxy needs credentials to forward agent requests, while other services need them for work such as communicating through Slack. A dedicated broker lets these components share connections and access rules without each implementing its own credential management.

For storing API keys and tokens, we’ve found OpenBao (a Linux Foundation fork of Vault) reliable and easy to deploy and manage. But storing secrets is only part of the job. The Credential Broker also tracks who connected each account, the connection’s status, and where its credentials are stored. We keep those records in Postgres.

Messaging is one of the places where these shared account connections are used. A connected Slack account lets the system communicate, but we still need to decide which messages require the agent’s attention and where its replies should go.

The Messaging Gateway handles this through a common interface for Slack, iMessage, Telegram, WhatsApp, internal chat, Instant Bloomberg, or any other messaging service. Its responsibilities include:

• Incoming messages. Routes messages that require the agent’s attention, such as a Slack mention, to the Agent Orchestrator.
• Outgoing messages. Sends the agent’s responses to the intended channel or thread, checking that the destination exists.
• Conversation storage. Durably saves and retrieves conversations belonging to each organization or user.
• Messaging permissions. Enforces rules about which channels an agent can send messages to.

An incoming message may find an idle agent, an agent already working, or no running sandbox at all. The Agent Orchestrator handles these cases: it prepares the environment, manages execution, and delivers queued messages when the agent is ready.

Its main responsibilities are to:

• Create a sandbox with access to the required files and services
• Prepare an agent, start it and manage its lifecycle
• Save execution traces and stream conversation output

It also manages scheduled runs, tracks execution status, and cleans up idle sandboxes.

Incoming messages go through a queue, allowing new requests to arrive while an agent is working. This matters when someone sends follow-up instructions or several people talk to an agent in a shared channel. The Messaging Gateway decides which messages enter the queue, and the orchestrator delivers them when the agent is ready.

A custom chat interface is optional. Some agents interact with users entirely through messaging platforms such as Slack. Others need their own interface in a web or mobile app. For these agents, the orchestrator can store conversation output and stream live updates, letting users reopen conversations, revisit results, and follow messages, tool calls, and reasoning as they appear.

Take web chat as an example. In these deployments, PostgreSQL provides durable storage for conversation history, while Redis Streams serves active conversations and their live updates. This lets the web interface load history and follow ongoing work without bottlenecks on the slow storage.

For building the web interface, assistant-ui has been a good fit. Its external-store runtime connects to the application’s own backend, and its message format has been flexible enough to support different client requirements.

The infrastructure around the agent depends on the harness to execute tools and report what is happening.

A harness runs the agent loop: it calls the model, executes tools, and maintains conversation state.

First-party agent SDKs are available from both providers: Anthropic offers the Claude Agent SDK, while OpenAI offers the OpenAI Agents SDK and the Codex SDK.

In our experience, these SDKs are still evolving. Maintaining integrations requires ongoing work as SDK versions and models change and new features are enabled. We’ve encountered several challenges:

• Model compatibility: The Claude Agent SDK is built for Claude and Anthropic’s API. Running Kimi K3 through Fireworks’ Anthropic-compatible endpoint exposed tool-call ID collisions. Direct tests showed that reused IDs caused Claude Code to drop results from subsequent model requests, despite successful tool execution. API compatibility alone does not guarantee harness compatibility.
• Keeping models and runtimes aligned: Newer models can work poorly with older harness and SDK versions, making upgrades necessary. These upgrades can also require integration changes: background tasks, for example, needed new completion logic to prevent early exits and hangs.
• Tool restrictions: Disabling a tool should also remove instructions to use it. In the Claude Agent SDK, blocking AskUserQuestion removed its definition, but planning tools still told the model to call it. Following those instructions can lead to a rejected tool call, wasting tokens and adding cost because of the harness’s own conflicting instructions.

These SDKs are open source, and we’ve contributed multiple fixes to the Claude Agent SDK.

These experiences, together with our own ideas about how a harness should work, led us to develop Unreal Agent. Read more about Unreal Agent on our blog.

We’ve covered the infrastructure we use to build cloud agents and some of the problems we’ve encountered running them in production. Different agents need different parts of this stack. We believe having flexible and sometimes reusable components makes it easier to build for those differences.

If you’re building a cloud agent, we’d love to work together! At Unreal Labs, we help companies build, deploy, and evaluate agents for their use cases. Get in touch: contact@unreallabs.ai.