通过会话感知型负载均衡扩展实时 AI Agent
英文标题：Scaling real-time AI agents with session-aware load balancing
实时 AI Agent 依赖长生命周期、双向且有状态的数据流，传统基于 QPS 或 CPU 利用率的负载均衡方式难以准确反映后端真实容量。文章提出，服务运行时应在应用层追踪活跃会话数，并将其与 CPU、内存等利用率信号结合，形成混合型路由模型。该方法可避免静默会话导致的容量误判、幽灵会话造成的错误路由，以及单个后端积累过多潜在工作负载。对于构建语音、视频及其他交互式 AI Agent 的开发者而言，基准测试和监控也应从短请求吞吐量转向长连接会话的分布、时延和可靠性。
11.7 KB
86.0 KB
62.5 KB
14.1 KB
62.3 KB
149.3 KB
81.8 KB
494.0 KB
496.8 KB
64.8 KB
351.9 KB
92.1 KB
中文
构建实时 AI Agent 与开发我们熟悉的标准 Web API 并不完全相同。对于典型 API，生命周期是可预测的：客户端发送请求，服务器处理请求、返回响应，然后继续处理下一项任务。这种短暂的模型非常理想，因为它易于追踪——你可以通过延迟、QPS 和 CPU 使用率等常见指标衡量性能。
实时 AI 系统改变了这一模型。后端不再处理彼此孤立的请求，而必须管理持续存在的实时双向数据流。音频分块、转录文本、模型输出和合成语音会同时持续地双向流动。
当用户打断系统时，事情会变得更加复杂。服务器必须立即停止当前的语音生成，转而更新上下文，可能触发新的工具调用，并开始拟定不同的回复；所有这些都必须在不中断连接的情况下完成。
这迫使我们重新思考基础设施。在实时 AI 的世界中，我们不再只是优化单个请求；我们管理的是一段持续进行的对话所带来的复杂性。
为什么 QPS 不够用
传统的负载均衡策略通常优先考虑请求吞吐量和当前 CPU 利用率。这些方法假定每个传入请求消耗的资源量可预测，并且任务完成后就会释放容量。然而，对于长生命周期、有状态的 AI 数据流，这些假设并不成立。
考虑两个后端任务：任务 A 处理 100 个短请求，每个请求在 50 毫秒内完成；任务 B 只接受 5 个请求，但每个请求都会变成持续 20 分钟的会话。如果只按请求到达速率判断，任务 B 的请求速率看起来较低；但实际上，它可能承担了显著更重、承诺更高的工作负载。
QPS 追踪的是到达流量，却无法反映服务器当前正在管理的实时对话数量。
同样，CPU 利用率也可能具有误导性。例如，一个语音运行时可能托管着 20 个静默会话；由于此时没有进行活跃的语音处理或模型推理，服务器看起来利用率很低。但只要这 20 名用户同时开始说话，CPU 使用率就可能突然飙升。
CPU 指标反映的是即时处理负载，而活跃会话数则揭示了后端已经承诺要处理的工作量。对于实时 AI，你需要平衡这两种信号。
数据流会产生应用层负载
实时 Agent 通常依赖 gRPC 或 WebSockets 等双向流式协议。虽然具体实现不同，但它们都会面临同一个基础设施难题：维护一条长期保持开放、数据持续双向流动的连接。
网络观察者看到的可能只是一条简单连接，但应用程序会将其视为复杂且有状态的会话。在单个实时 Agent 会话中，音频缓冲区、部分转录文本、活跃工具调用、模型上下文和用户专属指标等，都可能驻留在运行时内存中。
标准负载均衡器很难处理这种情况。它们能看到数据流，却无法区分真正活跃的用户对话、空闲监听者，或重试和健康检查等背景流量。由于基础设施无法看到这些内部状态，因此不能只依赖通用连接指标。相反，你需要应用层报告。后端服务本身是唯一拥有足够上下文的组件，能够准确追踪一个会话何时真正活跃，以及何时已经失败、完成或被取消。
在运行时内部追踪活跃会话
一种简单的模式是在流式会话生命周期开始和结束的位置追踪活跃会话。
suspend fun handleAudioSession(audioStream: Flow<AudioFrame>) {
activeSessions.incrementAndGet()
try {
withTimeout(20.minutes) {
audioStream.collect { frame ->
processAndRespond(frame)
}
}
} finally {
activeSessions.decrementAndGet()
}
}
Kotlin
finally 块的作用不止是基本清理。它确保活跃会话计数足够准确，从而可以用于路由决策。
如果计数器未能递减，即使会话早已结束，后端仍可能长时间看起来处于过载状态。反之，如果重复递减，就可能报告虚假的可用容量，吸引过多流量。生产环境的实现还需要谨慎处理边界情况：超时、取消或断开连接事件可能同时触发，并尝试清理同一个会话。
本质上，幽灵会话不仅仅是内存泄漏，而是会成为具有误导性的路由信号。
即使计数器足够精确，你仍必须考虑同步问题。负载均衡器通常按固定间隔拉取指标，而会话的开始和结束则是动态发生的，因此服务需要报告活跃会话的一致快照，避免负载均衡器基于过期数据作出决策。
将会话数量作为均衡信号
一旦后端能够报告活跃会话数，负载均衡模型就能更准确地反映工作负载。一个简单的容量模型可能只使用静态槽位：
remaining_capacity = max_sessions - active_sessions
Plain text
如果总容量为 100 个会话、当前持有 80 个会话，就还剩 20 个槽位。但这种方法很脆弱，因为它假设每个会话消耗相同的 CPU 资源；而在生成式 AI 中，这种假设很少成立。
因此，活跃会话数不应取代基于利用率的负载均衡；二者必须结合为混合模型。利用率（CPU/内存）反映当前资源压力，而会话数量反映已承诺的未来负载。
一种组合这些信号的方法，是通过反馈循环估算每个后端的有效容量。但首先，必须将两种信号归一化到相同的计量单位。由于负载均衡器天然以速率为思考单位，它们会将静态会话数转换为连续流量。例如，如果某个后端在 10 秒的报告窗口中持有 90 个活跃会话，一种实现可以将其视为 9 个“模拟 QPS”。通过将静态会话转换为标准速率，路由层便可将会话压力无缝叠加到 QPS 等传统信号上。
估算有效容量的一种简化方式是提出这样的问题：在达到目标利用率之前，该后端还可以接受多少额外工作？一种简化的容量估算公式可以如下所示：
让我们拆解一下：
- 目标利用率：希望安全运行的上限（例如 80% CPU）。
- 平均利用率：最近一个报告窗口内平滑后的 CPU 消耗（例如最近 10 秒）。
- 每会话成本：最近一个报告窗口内，动态计算出的每条活跃数据流的平均 CPU 成本。
- 安全缩放系数：一个抑制性乘数（例如 2.0）。由于 CPU 利用率通常会随并发数据流呈非线性增长，它充当惩罚因子，防止负载均衡器一次性向看似空闲的后端灌入过多新会话。
当负载均衡器计算出这个 Additional_Session_Rate 后，会将其乘以报告间隔，再加上当前活跃会话数，以得出真实的有效容量（Effective Capacity）。
这种混合方法有助于解决实时 AI 工作负载的核心路由问题。例如，一个拥有 10 个活跃会话但 CPU 使用率为 90% 的后端，会具有非常高的 Cost_Per_Session，使其 Additional_Session_Rate 降至零，因此不会接收新的流量。与此同时，一个拥有 80 个活跃会话但 CPU 使用率只有 40% 的后端，看起来似乎仍有余量；但 Safety_Scaler 会确保新会话仅被逐步分配给它，从而避免突然的负载峰值。
具体算法取决于你的代理和工作负载，但原则是一致的：实时 AI 负载均衡器必须同时理解当前状态的负担，以及已承诺会话的规模。
对会话感知型负载均衡进行基准测试
使用标准的发后即弃（fire-and-forget）负载测试来验证这些系统，往往会遗漏真正的故障模式。发送短请求突发流量的测试主要衡量请求吞吐量，无法复现长生命周期 AI 会话的行为。
有效的基准测试应当改变以下变量：
- 并发会话数量；
- 会话持续时间；
- 到达模式；
- 空闲状态与活跃说话状态的比例；
- 取消和断开连接的比例；
- 后端数量；
- 每个后端允许的最大会话数。
指标还必须捕捉流式行为。除了平均延迟和 QPS，还应追踪活跃会话在各后端之间的分布、过载分配率、p95 和 p99 启动延迟、首个数据流输出时间、丢失会话，以及强制断连后计数器的行为等指标。
目标是比较路由行为随时间的变化。
基于请求的负载均衡经常会造成不均匀流量：一些后端不断积累长生命周期会话，而另一些后端则处于空闲状态。会话感知型负载均衡能够更均匀地分配活跃对话，防止任何单一后端因潜在工作负载而不堪重负。
验证追踪器开销
即便是简单的会话追踪器也位于关键路径上。每次数据流开始和结束都会访问它们，因此，在管理大规模并发时，必须确保这类追踪机制不会成为服务瓶颈。
对于 JVM 服务，这通常意味着使用合适的微基准测试框架，以考量 JIT 优化、JVM 预热和死代码消除等因素；这些因素都很容易扭曲简单测试得出的结果。
通常，测试重点最好放在竞争而非仅仅是单线程延迟上。当多个线程或协程同时猛烈访问同一个计数器时会发生什么？以 Java 为例，AtomicInteger 对许多工作负载而言可能完全足够，但在高并发下，当多个线程不断尝试更新同一内存地址时，它可能遭遇严重的缓存行竞争（抖动）。在高吞吐量场景中，分片计数器或 LongAdder 风格的聚合设计可能更适合维持吞吐量。
会话追踪的成败取决于可靠性，而不只是实现复杂度。用于负载均衡的每个信号都必须准确、低开销，并经过真实流量模式的验证。
均衡会话，而不只是请求
实时 AI 将负载均衡的负担从网络问题转变为应用层问题。
QPS 告诉你到达量，CPU 揭示当前压力，而活跃会话数则标识了真实且已承诺的并发量。
对于语音、视频或世界模型等交互式 AI 工作负载，最有效的路由策略会综合这些信号。当后端能够传达其真实会话状态时，负载均衡器便可以作出更明智的决策，避免任何单个实例成为瓶颈。
随着 AI Agent 进入生产环境，基础设施也需要跟上步伐。扩展这些系统需要我们将关注点从均衡离散请求，转向均衡持续进行的实时对话。
English / 原文
Building real-time AI agents isn’t quite like working with the standard web APIs we’re used to. With a typical API, there’s a predictable lifecycle: the client sends a request, the server processes it, returns a response, and then moves on. This ephemeral model is great because it’s easy to track - you can measure performance through familiar metrics like latency, QPS, and CPU usage.
Real-time AI systems change that model. Instead of handling isolated requests, the backend has to manage a continuous, live bidirectional stream. You’re dealing with a constant stream of audio chunks, transcripts, model outputs, and synthesized speech flowing back and forth simultaneously.
Things get even more complex when a user interrupts. The server has to immediately halt its current speech generation, pivot to update the context, maybe trigger a new tool, and start drafting a different response; this must be done without dropping the connection.
This forces us to rethink our infrastructure. In the world of real-time AI, we aren’t just optimizing for the single request anymore; we’re managing the complexities of a living conversation.
Why QPS is not enough
Traditional load balancing strategies often prioritize request throughput and current CPU utilization. These approaches assume that each incoming request consumes a predictable amount of resources and that finishing a task clears capacity. However, for long-lived, stateful AI streams, these assumptions fall apart.
Consider two backend tasks: Task A handles 100 short requests, each finishing in 50 milliseconds. Task B accepts just 5 requests, but each turns into a 20-minute session. If you only judge these by request arrival rates, Task B appears less busy by request rate, yet it may actually be shouldering a significantly heavier, more committed workload.
QPS tracks arrival volume, but fails to capture the number of live conversations a server is already managing.
Similarly, CPU utilization can be deceptive. A voice runtime, for example, might host 20 silent sessions; because there’s no active speech processing or model inference happening, the server looks underutilized. But as soon as those 20 users start speaking simultaneously, CPU usage can spike suddenly.
While CPU metrics reflect the immediate processing load, active session counts reveal the work the backend has already promised to handle. For real-time AI, you need to balance both signals.
Streams create application-level load
Real-time agents typically rely on bidirectional streaming protocols like gRPC or WebSockets. While specific implementations differ, they all face the same infrastructure hurdle: maintaining an open, long-running connection where data flows constantly in both directions.
While a network observer sees a simple connection, the application treats it as a complex, stateful session. Inside a single real-time agent session, you might have audio buffers, partial transcripts, active tool calls, model context, and user-specific metrics all living within the runtime memory.
Standard load balancers struggle with this. They see the stream, but they can't distinguish between a genuinely active user conversation, an idle listener, or background noise like retries and health checks. Because the infrastructure lacks visibility into these internal states, you can't rely on generic connection metrics alone. Instead, you need application-level reporting. The backend service itself is the only component with enough context to accurately track when a session is truly active versus when it has failed, finished, or been canceled.
Tracking active sessions inside the runtime
A simple pattern is to track active sessions at the point where the streaming session lifecycle begins and ends.
suspend fun handleAudioSession(audioStream: Flow<AudioFrame>) {
activeSessions.incrementAndGet()
try {
withTimeout(20.minutes) {
audioStream.collect { frame ->
processAndRespond(frame)
}
}
} finally {
activeSessions.decrementAndGet()
}
}
Kotlin
The finally block goes beyond basic cleanup. It is what keeps the active-session count accurate enough for routing decisions.
If the counter fails to decrement, your backend might look overloaded long after the session finishes. Conversely, a double-decrement might report false capacity, drawing in excessive traffic. Production implementations also need to carefully manage edge cases where a timeout, cancellation, or disconnect event all trigger simultaneously to clean up the same session.
Essentially, ghost sessions act as misleading routing signals rather than simple memory leaks.
Even with a precise counter, you have to account for synchronization. Because load balancers generally pull metrics at intervals, while sessions start and stop fluidly, your service needs to report a consistent snapshot of active sessions to prevent the load balancer from making decisions based on outdated data.
Using session count as a balancing signal
Once the backend can report active sessions, the load-balancing model becomes more representative of the workload. A naive capacity model might just use static slots:
remaining_capacity = max_sessions - active_sessions
Plain text
If you have room for 100 sessions and are holding 80, you have 20 slots left. But this is brittle. It assumes every session costs the same amount of CPU, which is rarely true in generative AI.
This is why active session counts should not replace utilization-based balancing; they must be combined into a hybrid model. Utilization (CPU/Memory) captures current resource pressure, while the session count captures committed future load.
One way to combine these signals is to estimate each backend’s effective capacity using a feedback loop. But first, they must normalize both signals into a common unit of measurement. Because load balancers inherently think in rates, they translate the static session count into a continuous flow. For example, if a backend holds 90 active sessions over a 10-second reporting window, one implementation could treat this as 9 "pretend QPS." By converting static sessions into a standard rate, the routing layer can seamlessly add session pressure to traditional signals like QPS.
A simplified way to estimate effective capacity is to ask: how much additional work can this backend accept before it reaches the target utilization? A simplified capacity estimate formula could look like this:
Let's break that down:
- Target Utilization : The ceiling you want to safely run at (e.g., 80% CPU).
- Average Utilization : The smoothed CPU consumption over the last reporting window (e.g., the last 10 seconds).
- Cost Per Session : The dynamically calculated average CPU cost of an active stream during the last reporting window.
- Safety Scaler : A dampening multiplier (e.g., 2.0). Because CPU utilization often scales non-linearly with concurrent streams, this acts as a penalty factor to prevent the load balancer from dumping too many new sessions onto a seemingly idle backend at once.
Once the load balancer computes this Additional_Session_Rate, it multiplies it by the reporting interval and adds it to the currently active sessions to find the true Effective Capacity.
This kind of hybrid approach helps address the core routing problem for real-time AI workloads. For example, a backend with 10 active sessions and 90% CPU will have a very high Cost_Per_Session, driving its Additional_Session_Rate to zero, resulting in it receiving no new traffic. Meanwhile, a backend with 80 active sessions but only 40% CPU might seem like it has room, but the Safety_Scaler ensures it is only fed new sessions gradually, preventing sudden spikes.
The exact algorithm depends on your proxy and workload, but the principle is consistent: a real-time AI load balancer must understand both the weight of the current state and the volume of committed sessions.
Benchmarking session-aware balancing
Validating these systems with standard fire-and-forget load tests often misses real failure modes. Tests that send bursts of short requests primarily measure request throughput, which doesn't replicate the behavior of long-lived AI sessions.
Effective benchmarks should vary:
- concurrent session counts;
- session duration;
- arrival patterns;
- ratios of idle to active speaking;
- cancellation and disconnect rates;
- backend counts;
- maximum sessions per backend.
Metrics must also capture streaming behavior. Beyond average latency and QPS, track metrics like active-session distribution across backends, overloaded assignment rates, p95 and p99 startup latency, time-to-first-stream, dropped sessions, and counter behavior after forced disconnects.
The goal is to compare routing behavior over time.
Request-based balancing often creates lumpy traffic, where some backends accumulate long-lived sessions while others sit idle. Session-aware balancing distributes active conversations more evenly, preventing any single backend from becoming overwhelmed by latent work.
Validating tracker overhead
Even simple session trackers sit on a critical path. Every stream start and end hits them, so if you're managing massive concurrency, it’s important that it’s designed to ensure that this tracking doesn't bottleneck a service.
For JVM services, that typically means using a proper microbenchmarking framework to account for JIT optimization, JVM warmup, and dead-code elimination, all of which can easily skew naive results.
It’s generally best to focus testing on contention rather than just single-threaded latency. What happens when multiple threads or coroutines hammer the same counter at once? Using Java as an example, an AtomicInteger may be perfectly fine for many workloads, but at high concurrency, it can suffer from severe cache-line contention (bouncing) as multiple threads constantly attempt to update the same memory address. In high-throughput scenarios, designs such as sharded counters or LongAdder-style aggregation may become more appropriate to maintain throughput.
Session tracking succeeds or fails based on reliability, not just implementation complexity. Every signal used for load balancing must be accurate, low-overhead, and vetted against realistic traffic patterns.
Load balancing sessions, not just requests
Real-time AI shifts the load-balancing burden from a network concern to an application-level problem.
QPS tells you about arrivals, CPU reveals current pressure, and active session counts identify your true, committed concurrency.
For interactive AI workloads like voice, video, or world models, the most effective routing strategies synthesize these signals. When a backend communicates its true session state, a load balancer can make smarter decisions, preventing any single instance from becoming a bottleneck.
As AI agents move into production, infrastructure needs to catch up. Scaling these systems requires shifting our focus from balancing discrete requests to balancing continuous, live conversations.