🔶 文件优于工具：我们如何用虚拟文件系统和 bash 构建 Agent
英文标题：🔶 Files over tools: how we built our agent with a virtual filesystem and bash
2026 年 3 月，Knock 发布了一款基于虚拟文件系统和 bash 的 AI Agent，能够管理所有客户消息资源。该 Agent 采用 Elixir 移植的 just-bash 实现文件系统上下文发现，避免直接暴露大量 API 工具，从而降低上下文窗口压力。这种架构更接近 Claude Code 等编码代理，支持多轮会话和惰性加载，为开发者提供了一种将领域映射到文件系统来增强 Agent 上下文的高效模式。目前该 Agent 已通过 API、Slack 和 MCP 服务器在生产环境中处理复杂任务，并配套了完善的评估与可观测性体系。
105.7 KB
332.9 KB
25.1 KB
56.7 KB
1.2 KB
31.2 KB
118.2 KB
中文
2026 年 3 月，我们发布了 Knock Agent：一个用于管理 Knock 中所有客户消息资源的 AI Agent。它能创建和更新工作流、模板和受众，帮助细分用户，并回答有关客户消息表现的问题。该 Agent 可以从 Knock 仪表板、已连接的 Slack 工作区、API 或我们的 MCP 服务器调用。
在这篇文章中，我们将深入幕后，了解我们如何使用 bash、虚拟文件系统和我们的管理 API 来架构和构建 Agent。
定义我们 Agent 的愿景
我们的愿景是创建一个能管理 Knock 仪表板中所有可访问内容的 Agent。我们希望 Agent 能创建使用公司设计系统、匹配品牌语气并理解数据建模方式的消息。
为了实现这一愿景，我们需要创建一个丰富的 Agent 运行环境，能够访问 Knock 账户中的全部数据，并在创建消息和回答问题时使用这些上下文。
我们最初的原型仅针对工作流运行，这是一个充满细微差别的深度领域。我们采用了一种“按类型划分工具”的模式，每个工具都暴露我们管理 API 的一个原语，例如向工作流添加延迟步骤，或使用可视化块语言构建邮件模板。工具描述会编码使用该工具的特质并提供示例，而工具输入则描述了字段类型和必填项。
这种方法虽然有效且效果良好，但我们意识到它无法扩展：我们必须为管理 API 中的每个资源暴露一个工具，这会膨胀 Agent 的上下文窗口，除非我们实现更复杂的工具路由层或为 Agent 引入脚本语言。
我们发现了 Vercel 团队的一篇博文，“How to build agents with filesystems and bash”，他们在其中描述了为内部数据工具 d0 构建 Agent。他们解释了文件系统是 Agent 发现所需上下文的绝佳方式，以及如何将领域映射到文件系统来驱动这一上下文层。
在 Knock，我们长期以来拥有 CLI，它能让团队拉取 Knock 资源（如消息模板和组件）并在本地通过文件系统操作。
这里的想法很简单：与其暴露镜像管理 API 的工具，我们为 Agent 提供一个包含账户内容的文件系统，以及使用 bash 在该文件系统上编写脚本的能力。这样，Agent 可以探索文件系统来收集上下文，并使用 bash 根据需要高效编写脚本。
当 Agent 想要编辑工作流或模板时，它会就地修改文件系统中的文件，并_回调_到 Knock 来持久化更改。
这种模式比其他产品内助手更接近 Claude Code 和其他编码 Agent，符合我们通过 CLI 和本地优先方法维护客户消息的既定理念。
用于 Agent 的 bash——基于 Elixir
在 2025 年的假期，Vercel 的 Malte 发布了 just-bash。其想法很简单：让 Agent 拥有 Unix 文件系统和 bash 环境，而无需启动 Linux 镜像。实际上，这意味着在 TypeScript 中创建一个虚拟 bash 实现和虚拟文件系统，供 Agent 调用。
Knock 是一家深深扎根于 Elixir 生态系统的公司。我们希望拥有自己的虚拟化 bash 解释器。我们的选择是：1) 放弃我们在 Agent 上已做的工作，用 TypeScript 重建，或 2) 将该库移植一个版本到 Elixir。
幸运的是，像 just-bash 这样的库是 Agent 的可重现目标。它具有明确定义的表面区域和覆盖其支持命令（jq、ls、cat 等）的完备测试套件——这些固定装置被我们的 Elixir 移植版逐字重用。
在一次有史以来最津津乐道的“书呆子诱惑”中，我们说服了痴迷 Agent 的 Knock 之友 Ivar Vong 来尝试。在许多 token 之后，我们拥有了一个完整的 Elixir 版 bash 解释器和虚拟文件系统，他也将其命名为 just-bash，并带有广泛的测试套件和健壮的安全模型。
为什么不用沙箱？
一个显而易见的问题：为什么不通过沙箱为 Agent 提供一个_完整_的计算机，而不是虚拟文件系统？毕竟有许多供应商 可选。
我们认为，对于我们的简单需求，完整沙箱——在现阶段——是杀鸡用牛刀。与虚拟的内存选项相比，启动沙箱会带来性能成本，并引入更难的内容同步问题，以及我们暂时想要回避的应用外更改。
我们会在此复杂性天平倾斜时重新评估这一决定。为 Agent 增加 bash 之外的完整脚本语言（如 Python）可能是决定因素。我们遵循将“手”与“大脑”解耦的原则，因此迁移到_真正的_沙箱应该只是将调用从虚拟沙箱切换到真实沙箱。
连接一切
配备 Elixir 的 just-bash 和所有构建块，我们组装了基于文件系统的 Agent。
我们直接在 Anthropic API 之上运行典型的 Agent 循环，根据当前任务使用不同模型。该 Agent 循环由持久化工作流支持，能安全地重试步骤，使用 Elixir 中出色的 Oban 库，并以 Postgres 作为状态存储。
在 Agent 会话中，我们附加一个包含 just-bash 实例和完整虚拟文件系统的沙箱。对我们而言，沙箱只是运行在 Elixir 集群中的另一个进程。沙箱会在请求首次需要使用文件系统时惰性启动。这确保了不需要文件系统的请求可以快速回答，且占用最小。
当我们没有已运行的沙箱进程时，文件系统会先用账户中对象的_元数据_进行引导。这确保 Agent 拥有所需的完整资源目录，但我们可以在 Agent 操作它们之前_惰性加载_完整的较大资源。
这是该文件系统的样子：
|- account.json # account context
|- channels.json # the set of channels available
|- workflows.json # the full list of workflows on the account
|- ... # lots of other resources
当 Agent 想要探索特定工作流或模板时，它可以通过调用 load_resources 工具请求将它们批量加载到文件系统中。
workflows/
some-workflow/
workflow.json # the workflow configuration (steps, settings etc)
steps/
email_1/
template.html # the message template for the email step
in_app_1/
template.md # the message template for the in-app step
沙箱进程是长期存在的，因此后续轮次将让这些内容加载完毕并准备好操作。这对于多轮对话非常理想，你可以与 Agent 来回交流，进行一系列工作流或消息模板的更新。
我们为 Agent 公开了其他几个工具：
bash– 执行 bash 命令。read_file– 从文件系统读取文件，优于cat。edit_file– 对文件进行针对性编辑（将字符串 x 替换为 y）。write_file– 将整个文件写入文件系统。upsert_resource– 将文件系统上资源的更改持久化回 Knock。
这里唯一值得一提的非文件操作工具是 upsert_resource 工具。它通过读取存储在虚拟文件系统上的 bundle 并调用与管理 API 相同的内部方法，将文件系统的更改持久化回 Knock 账户。
我们的 Agent 循环会发出完整的事件日志，包括收到的提示、执行的操作和发送的响应。该事件日志在内存中缓冲，并在 Agent 执行时异步写入 Postgres 存储。事件日志会为后续轮次进行压缩和格式化以保留上下文。我们将其格式化并通过 NDJSON 流传输给连接的客户端。
教授 Agent 新技能
我们从 Agent 可以管理的单一领域——工作流和模板——开始，但我们知道我们希望我们的 Agent 能够处理 Knock 中的所有内容。
一开始，我们将所有关于工作流和模板的产品信息放入系统提示中，但我们知道这无法扩展。相反，我们将特定于资源的信息移到 Agent 可以“学习”的技能中，并引导 Agent 在编辑资源之前阅读这些技能。
每个技能包含 Agent 理解操作该资源所需的细微信息以及要避免的常见陷阱。技能在文件系统上可用，并在调用时烘焙到系统提示中，以便 Agent 了解可用的全部技能。
skills/
workflows/
SKILL.md # top-level skill definition
references/
template-editing.md # specific skill reference for template editing
broadcasts/
...
guides/
...
当 Agent 想要学习新技能时，它会调用 skill 工具。这确保技能的完整元数据被检索并添加到 Agent 的上下文窗口，并为我们提供了一种方便的方式来评估和监控 Agent 是否遵循其技能。
我们还以相同方式引入了客户定义的技能，因此技能可以是系统和客户定义的混合，每个都加载到系统提示中以供发现，并加载到文件系统中以供访问。
并非 Knock 账户中的所有数据都适合表示为静态文件。以 Knock 生成的日志数据为例。这最好通过 Agent 在运行时执行的查询来服务，以获取一页范围内的日志。
我们没有引入更多工具来向 Agent 暴露 Knock 中的每个资源，而是引入了一个自定义命令 (knock) 形式的 CLI，注册到我们的 just-bash 实例。
现在，当 Agent 想要查询 API 日志（或用于调试的工作流运行日志）时，它可以调用 knock api_logs list 或 knock messages list。我们通过调试技能教 Agent 如何操作 CLI，但由于 CLI 为每个资源暴露了 --help 命令，Agent 很容易学会以最少的引导和高成功率进行操作。
可观测性与评估
像任何生产系统一样，我们希望清楚地了解 Agent 在做什么以及如何运作，并拥有一个测试套件来验证其行为。
在可观测性方面，我们从 Agent 循环中以标准 OpenTelemetry 格式发出跟踪。我们将这些跟踪转发到 Honeycomb 和 Braintrust。Braintrust 作为工具可用于对 Agent 会话进行更深入的分析，而 Honeycomb 是我们的服务跟踪首选。
在评估方面，我们构建了一个自主开发的评估系统，基于 Anthropic 优秀文章 “Demystifying evals for AI agents” 中分享的见解。我们的评估衡量轨迹，检查结果是否具备一个或多个预期属性或形状，并使用 LLM-as-judge 对输出进行评分。我们每晚运行评估，并在 pass^k 上测量以确保 Agent 在常见任务上的准确性。
未来改进
我们很高兴能在这个基于文件系统的基础之上继续投资于我们的 Agent。我们正在考虑的一些未来改进包括：
- 为长时间运行的 Agent 会话引入更好的压缩机制。 我们目前有一些天真的机制，但希望优化会话摘要逻辑，以便我们的 Agent 能够处理长周期任务。
- 提高评估的广度和深度。 我们最初主要关注最常见的操作，但我们正着眼于向评估中添加更多案例，特别是针对更复杂的工作流构建和编辑。
- 探索用于专注工作的子 Agent。 我们正在评估子 Agent 如何处理上下文收集、分析以及主 Agent 循环之外的其他离散任务。
我们基于文件系统的 Agent 为我们提供了坚实的基础，以便随着 Agent 能力的扩展进行迭代。使用文件系统作为高效的上下文检索机制，使我们能够交付一个智能 Agent，能够执行我们在生产中看到客户要求的复杂任务。我们已经将 Agent 的调用面扩展到 API、Slack 和 MCP 服务器，并期待在未来为 Agent 添加更多功能。
English / 原文
In March 2026, we shipped the Knock Agent: an AI agent for managing all your customer messaging resources in Knock. It can create and update workflows, templates, and audiences, help segment users, and answer questions about how your customer messaging is performing. The agent can be invoked from the Knock dashboard, a connected Slack workspace, the API, or our MCP server.
In this post, we'll take a look behind the scenes at how we architected and built our agent using bash, a virtual file system, and our management API.
Defining the vision for our agent
Our vision was to create an agent that could manage everything that can be accessed from the Knock dashboard. We wanted the agent to create messaging that uses your company's design system, matches the tone of voice, and understands how your data is modeled.
To bring this vision to life, we needed to create a rich agent harness that could access the full breadth of data in your Knock account and use that context while creating your messaging and answering any questions.
We built our first prototype to operate exclusively on workflows, which is a deep domain with plenty of nuance around how it works. We used a tool-per-type pattern, where each tool exposed a primitive from our management API, such as adding a delay step to a workflow, or building an email template using our visual blocks language. The tool description encoded the idiosyncrasies of working with that tool and gave examples, while the tool input described the field types and what was required.
While this worked and led to good results, we realized that this approach would not scale: we'd be exposing a tool per resource in the management API, which would bloat the context window of the agent without us implementing a more sophisticated tool routing layer or introducing a scripting language for the agent to use.
We found a blog from the team at Vercel, "How to build agents with filesystems and bash", where they describe building an agent for d0, their internal data tool. They explain how the filesystem is a great way for an agent to discover the context it needs, and how you can map a domain to a filesystem to power this context layer.
At Knock, we've long had a CLI which enables teams to pull down their Knock resources, such as messaging templates and components, and operate on them locally via the filesystem.
The idea here was simple: instead of exposing tools that mirror our management API, we give the agent a filesystem with the contents of your account, and the ability to use bash to script on top of that filesystem. That way, the agent can explore the filesystem to gather context, using bash to efficiently write scripts as needed.
When the agent wants to make an edit to a workflow or template, it modifies a file in place in the filesystem and calls back to Knock to persist that change.
This pattern is much closer to Claude Code and other coding agents than other in-product assistants, and it fits the philosophy we've already established with our CLI and local-first approach to maintaining your customer messaging.
A bash for agents, in Elixir
Over the 2025 holidays, Malte from Vercel released just-bash. The idea was simple: let agents have a unix filesystem and a bash environment, without booting a Linux image. In practice, that meant creating a virtual bash implementation in TypeScript with a virtual filesystem that an agent could call.
Knock is a company firmly rooted in the Elixir ecosystem. We wanted a virtualized bash interpreter of our own. Our options were to 1) forgo the work we had done on our agent and rebuild it in TypeScript, or 2) port a version of the library to Elixir.
Fortunately, a library like just-bash is a reproducible target for an agent. It has a well-defined surface area and a thorough test suite covering the commands it supports (jq, ls, cat, etc.) — fixtures our Elixir port reuses verbatim.
In one of the all-time nerd snipes, we convinced agent-pilled friend of Knock, Ivar Vong, to give it a go. Many tokens later, we had a complete bash interpreter and virtual filesystem in Elixir, which he also called just-bash, with an extensive test suite and a robust security model.
Why not a sandbox?
The obvious question: why not give the agent a full computer via a sandbox rather than a virtual filesystem? After all, there are a number of vendors we could have picked to power it.
We believe a full sandbox is—at this stage—overkill for our simple needs. Starting a sandbox comes at a performance cost compared to the virtual, in-memory option, and introduces harder synchronization problems for content and changes made outside our app which we wanted to punt on for now.
We will revisit this decision should the complexity scale tip. Adding a full scripting language (like Python) outside of bash for the agent to write might be the deciding factor. We follow the principle of decoupling the hands from the brain, so moving to a real sandbox should just mean swapping calls to the virtual one for the real one.
Wiring it together
Armed with just-bash in Elixir and all of the building blocks, we assembled our filesystem-backed agent.
We run a typical agent loop on top of the Anthropic API directly, using different models for the task at hand. That agent loop is backed by a durable workflow that can safely retry steps, using the excellent Oban library in Elixir that backs its state with Postgres.
Within an agent session, we attach a sandbox which includes the just-bash instance and the full virtual filesystem. A sandbox for us is just another process that runs in our Elixir cluster. Sandboxes are started lazily whenever a request first needs to use the filesystem. This ensures that requests that don't need the filesystem return quick answers with a minimal footprint.
When we don't have an existing sandbox process running, the filesystem is bootstrapped with the metadata of the objects in the account. This ensures that the agent has a full catalog of resources that it needs, but that we can lazily load the full, larger resources before the agent needs to operate on them.
Here's what that filesystem looks like:
|- account.json # account context
|- channels.json # the set of channels available
|- workflows.json # the full list of workflows on the account
|- ... # lots of other resources
When the agent wants to explore specific workflows or templates, it can request that they be loaded into the filesystem in bulk by calling the load_resources tool.
workflows/
some-workflow/
workflow.json # the workflow configuration (steps, settings etc)
steps/
email_1/
template.html # the message template for the email step
in_app_1/
template.md # the message template for the in-app step
The sandbox process is long-lived, so subsequent turns will have this content loaded and ready to operate on. This is perfect for multi-turn conversations where you go back and forth with the agent to make a series of updates to a workflow or message template.
We expose a few other tools for the agent to work with:
bash– executes bash commands.read_file– reads a file from the filesystem, preferred overcat.edit_file– makes a targeted edit to a file (replace string x with y).write_file– writes an entire file to the filesystem.upsert_resource– persists changes made on a resource in the filesystem back to Knock.
The only tool of note here outside of file operations is the upsert_resource tool. It persists filesystem changes back to the Knock account by reading a bundle stored on the virtual filesystem and calling the same internal methods as our management API.
Our agent loop emits a full event log of the prompt it received, actions it performed, and the response it sent. This event log is buffered in memory and written asynchronously to a Postgres store as the agent executes. The event log is compacted and formatted for subsequent turns to preserve context. We format and stream the log to connected clients using an NDJSON stream.
Teaching the agent new skills
We started with a single domain that the agent could manage—workflows and templates—but we knew we wanted our agent to be able to work with everything in Knock.
At first, we put all the product information about workflows and templates into the system prompt, but we knew that wouldn't scale. Instead, we moved resource-specific information into skills that the agent could "learn," and steered the agent to read those skills before it edits a resource.
Each skill includes the necessary information the agent needs to understand the nuances of working with that resource, as well as common pitfalls to avoid. Skills are available on the filesystem and are baked into the system prompt at invocation time so the agent knows about the full breadth of skills available to it.
skills/
workflows/
SKILL.md # top-level skill definition
references/
template-editing.md # specific skill reference for template editing
broadcasts/
...
guides/
...
When the agent wants to learn a new skill, it calls the skill tool. This ensures that the full metadata of the skill is retrieved and added to the agent's context window, and gives a convenient way for us to evaluate and monitor that the agent is following its skills.
We also introduced customer-defined skills in the same manner, so skills can be a mix of system and customer-defined, each loaded into the system prompt for discovery, and into the filesystem for access.
Not all of the data within your Knock account is well-suited to be represented as static files. Take the logging data Knock generates. This is best served by a query the agent can run at execution time to fetch a page of scoped logs.
Instead of introducing even more tools to expose every resource within Knock to the agent, we introduced a CLI to the agent as a custom command (knock), registered to our just-bash instance.
Now, when the agent wants to query API logs (or workflow run logs to aid with debugging), it can call knock api_logs list or knock messages list. We teach the agent how to work with the CLI via a debugging skill, but because the CLI exposes a --help command for each resource, it's easy for the agent to learn to operate with minimal steering and a high degree of success.
Observability and evals
Like any production system, we wanted good visibility into what the agent is doing and how it operates, as well as a test suite to verify its behavior.
For observability, we emit traces from the agent loop in standard OpenTelemetry format. We forward these traces to both Honeycomb and Braintrust. Braintrust is useful as a tool to do deeper analysis of agent sessions, while Honeycomb is our go-to for our service traces.
On the evals side, we built a homegrown evals system based on the insights shared in the excellent post by Anthropic, "Demystifying evals for AI agents". Our evals measure trajectory, check the outcome has one or more expected properties or shape, and score the output using an LLM-as-judge. We run our evals nightly and measure on a pass^k to ensure accuracy of our agent on common tasks.
Future improvements
We're excited to continue to invest in our agent, on top of this filesystem-based foundation. Some future improvements we're considering include:
- Introducing better compaction for long-running agent sessions. We currently have some naive mechanisms here but want to refine our session summarization logic so that our agent can work on long horizon tasks.
- Improving the breadth and depth of our evals. We've focused primarily on the most common operations to start, but we're looking to add more cases to our evals, especially for more complex workflow building and editing.
- Exploring subagents for focused work. We're evaluating how subagents could handle context gathering, analysis, and other discrete tasks outside of the main agent loop.
Our filesystem-backed agent provides a solid foundation for us to iterate on as we expand the capabilities of what our agent can do. Using the filesystem as an efficient context retrieval mechanism has enabled us to ship an intelligent agent that can perform the complex tasks we're seeing customers ask in production. We've already expanded the agent surface to be invoked from the API, Slack, and our MCP server and look forward to adding more capabilities to the agent in the future.