NVIDIA-NeMo Switchyard：面向 LLM 流量的 Rust 路由代理与库
英文标题：🔥 NVIDIA-NeMo/Switchyard
Switchyard 是 NVIDIA-NeMo 推出的 Rust 代理与库，用于在不同模型提供商之间路由 LLM 请求，并在 OpenAI 与 Anthropic API 格式之间进行转换。它可让 Claude Code、Codex 等编码 Agent 保持原生 API 调用方式，同时将请求转发至 vLLM、NVIDIA NIM、Ollama 或任意兼容 OpenAI 的端点。项目提供随机路由、LLM 分类路由、基于信号的阶段路由及自定义算法，并通过 Prometheus 提供请求、错误、延迟、Token 和路由开销指标。该项目目前仍处于快速演进的 pre-alpha 阶段，不建议用于生产环境。
1.2 MB
69.4 KB
中文
Switchyard 是一个面向 LLM 流量的 Rust 代理和库。它可在不同提供商之间路由请求，在 OpenAI 与 Anthropic API 之间转换格式，记录运行指标，并提供具有类型约束、可组合的路由算法。
为什么选择 Switchyard？ 你可以将 Claude Code 或 Codex 等编码 Agent 指向开源模型。Switchyard 可在 OpenAI Chat、Anthropic Messages 和 OpenAI Responses 格式之间转换，因此 Agent 能继续使用其原生 API，而请求则由 vLLM、NVIDIA NIM、Ollama 或任何兼容 OpenAI 的端点处理。同一个代理还可将流量分散到多个模型，用于 A/B 基准测试；应用信号驱动的阶段路由；或运行你自行编写的自定义算法。
- 协议转换：在 OpenAI Chat、Anthropic Messages 和 OpenAI Responses 格式之间转换
- 多后端路由：随机路由、LLM 作为分类器的路由、信号驱动的阶段路由器，或你自己的算法
- 运行指标：Prometheus 指标覆盖请求、错误、延迟、Token 和路由开销
Switchyard 是快速演进中的 pre-alpha 软件。在达到 v1.0 前，其 API 和算法预计将发生重大变化。
警告
实验性软件，不适用于生产环境。
选择启动器路径，可通过 Switchyard 运行 Claude Code、Codex CLI 或 OpenClaw。选择服务器路径，可将 Switchyard 作为独立代理运行。选择库路径，可在你自己的 Rust 应用中嵌入路由功能。
如果尚未安装，请先安装 uv，然后安装已发布的 Switchyard 工具：
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"
你启动的编码 Agent 也必须已安装并存在于你的 PATH 中。此操作不会安装独立的 switchyard-server 二进制文件；请使用服务器路径进行安装。
设置 OpenRouter 密钥，并针对打包部署启动：
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard launch claude --model switchyard
switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard
如需使用自己的原生 TOML 部署，请传入其路由 ID 和配置：
switchyard launch claude --model my-route --config routes.toml
使用此路径安装并运行独立 Rust 代理。安装带有 Cargo 的 Rust，然后安装已发布的二进制文件：
cargo install --locked switchyard-server
switchyard-server --help
默认情况下，Cargo 会构建发布版二进制文件，并将其安装到 ~/.cargo/bin。
按照 入门指南创建 routes.toml，然后验证配置并启动服务器：
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
在另一个终端中验证代理：
curl http://localhost:4000/health
如需完整配置和测试请求，请参阅入门指南。
switchyard-libsy 可将路由算法嵌入到你自己的 Rust 应用中。它本身绝不会调用模型：算法决定应使用哪个目标，并将每一次模型调用都交回给你处理，因此它可集成到现有代理、网关或 Agent 运行时中，而无需接管 HTTP 栈。当你希望由该组件代为发起调用时，可将其与 switchyard-llm-client 搭配使用。
[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
有关设置和算法列表，请参阅入门指南，或 switchyard-libsy crate 文档。
| 策略 | 适用场景 | 路由 type |
|---|---|---|
| LLM 分类器 | 请求内容应决定某一轮需要使用弱模型层级还是强模型层级。 | llm_classifier |
| 阶段路由器 | 对话中已有的信号，例如工具结果和错误，应在无需额外模型调用的情况下为大多数轮次完成路由。 | stage_router |
| 升级路由器 | 每一轮先在弱模型层级运行，随后由评判模型读取答案，以决定是否将相同请求发送至强模型层级。 | llm_classifier，并设置 mode = "escalation" |
| 随机 | 你需要固定的流量分配比例，以进行 A/B 测试、基线测试或成本实验。 | random |
passthrough 路由会在一个模型 ID 下注册一个目标，不作任何路由决策。有关通用路由结构和自托管目标，请参阅路由概览。
flowchart LR
clients["Clients"]
switchyard["Switchyard<br/>routing · translation · fallback"]
backends["Model backends"]
clients -->|"OpenAI / Anthropic API"| switchyard
switchyard -->|"provider-native format"| backends
加载中
客户端保持其原生 OpenAI 或 Anthropic API 格式。Switchyard 选择已配置的后端，以该后端自身的格式转发请求，并将响应转换回客户端预期的结构。服务器接受 OpenAI Chat Completions、OpenAI Responses 和 Anthropic Messages。每个已配置的 LLM 客户端选择一种上游格式。
- 入门指南：完整的启动器和独立服务器操作指南
- 核心概念：LLM 客户端、目标、路由、模型 ID 和路由算法
- 路由概览：选择并配置路由算法
switchyard-server：服务器配置、路由算法和指标switchyard-libsy：在 Rust 应用中嵌入路由算法switchyard-protocol：提供商无关的请求、响应和流式传输类型switchyard-translation：请求、响应和流的转换
社区
Apache 2.0 许可证。版权所有 © NVIDIA Corporation。
English / 原文
Switchyard is a Rust proxy and library for LLM traffic. It routes requests across providers, translates between OpenAI and Anthropic APIs, records operational metrics, and provides typed, composable routing algorithms.
Why Switchyard? Point a coding agent such as Claude Code or Codex at an open-source model. Switchyard translates between the OpenAI Chat, Anthropic Messages, and OpenAI Responses formats, so the agent keeps speaking its native API while the request is served by vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint. The same proxy can spread traffic across several models for A/B benchmarking, apply signal-driven stage routing, or run a custom algorithm you write yourself.
- Protocol Translation : convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
- Multi-Backend Routing : random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
- Operational Metrics : Prometheus metrics cover requests, errors, latency, tokens, and routing overhead
Switchyard is pre-alpha software that is evolving rapidly. The API and algorithms are expected to change significantly before we reach v1.0.
Warning
Experimental software. Not for production use.
Choose the launcher path to run Claude Code, Codex CLI, or OpenClaw through Switchyard. Choose the server path to run Switchyard as a standalone proxy. Choose the library path to embed routing in your own Rust application.
Install uv if it is not already available, then install the published Switchyard tool:
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"
uv tool install --python 3.10 "nemo-switchyard[cli]"
The coding agent you launch must also be installed and on your PATH. This does not install the standalone switchyard-server binary; use the Server Path for that.
Set an OpenRouter key and launch against the packaged deployment:
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard launch claude --model switchyard
switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard
To use your own native TOML deployment, pass its route ID and configuration:
switchyard launch claude --model my-route --config routes.toml
Use this path to install and run the standalone Rust proxy. Install Rust with Cargo, then install the published binary:
cargo install --locked switchyard-server
switchyard-server --help
Cargo builds the release binary and installs it into ~/.cargo/bin by default.
Create routes.toml using the Getting Started guide, then validate it and start the server:
export OPENROUTER_API_KEY="your-openrouter-key" # pragma: allowlist secret
switchyard-server --config routes.toml --dry-run
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000
Verify the proxy in another terminal:
curl http://localhost:4000/health
For a complete configuration and a test request, follow Getting Started.
switchyard-libsy embeds the routing algorithms in your own Rust application. It never calls a model itself: an algorithm decides which target to use and hands every model call back to you, so it drops into an existing proxy, gateway, or agent runtime without owning an HTTP stack. Pair it with switchyard-llm-client when you want the calls made for you.
[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
See Getting Started for setup and the algorithm list, or the switchyard-libsy crate docs.
| Strategy | Use it when | Route type |
|---|---|---|
| LLM Classifier | Request content should decide whether a turn needs the weak or strong tier. | llm_classifier |
| Stage Router | Signals already in the conversation, such as tool results and errors, should route most turns without an extra model call. | stage_router |
| Escalation Router | Every turn runs on the weak tier first, and a judge reads that answer to decide whether to send the same request to the strong tier. | llm_classifier with mode = "escalation" |
| Random | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | random |
A passthrough route registers one target under one model ID with no routing decision. See the Routing Overview for the common route shape and self-hosted targets.
flowchart LR
clients["Clients"]
switchyard["Switchyard<br/>routing · translation · fallback"]
backends["Model backends"]
clients -->|"OpenAI / Anthropic API"| switchyard
switchyard -->|"provider-native format"| backends
Loading
Clients keep their native OpenAI or Anthropic API format. Switchyard picks a configured backend, forwards the request in that backend's own format, and translates the response back into the shape the client expects. The server accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Each configured LLM client selects one upstream format.
- Getting Started : complete launcher and standalone server walkthroughs
- Core Concepts : LLM clients, targets, routes, model IDs, and routing algorithms
- Routing Overview : choose and configure a routing algorithm
switchyard-server: server configuration, routing algorithms, and metricsswitchyard-libsy: embed routing algorithms in a Rust applicationswitchyard-protocol: provider-neutral request, response, and streaming typesswitchyard-translation: request, response, and stream translation
Community
Apache 2.0 License. Copyright NVIDIA Corporation.