OpenAI 如何为 9 亿用户提供低延迟语音 AI

OpenAI 每周为 9 亿用户运行语音 AI，为此他们使用了 WebRTC，因为其他方案意味着要重新发明互联网处理实时音频的方式。

难题在于 WebRTC 是为具有固定 IP 和端口的服务器设计的，而 Kubernetes 将这些地址视为可丢弃的。在这种规模下，传统的解决方案是 SFU，适合多人工作负载（如群组视频通话），但 OpenAI 的流量绝大多数是一个用户与一个模型对话。

为此，他们的架构将整个栈分成两部分：

将它们连接在一起的巧妙之处在于，使用 ICE ufrag——一个协议在连接建立阶段就已交换的字段——作为路由键，中继可以从新会话的第一个数据包中读出这个字段。此外，从全球中继到用户态 Go 实现、Redis 缓存以及精细的套接字级优化，一切都建立在这一核心思想之上。

在本文中，我们将详细审视整个历程以及 OpenAI 工程团队面临的挑战。

免责声明：本文基于 OpenAI 工程团队公开分享的细节。如有不准确之处，请在评论区指出。

语音 AI 要么感觉像对话，要么感觉像对讲机。这两种体验之间的界限以毫秒为单位。

当网络在听到用户和作出响应之间暂停时，这种错觉就会被打破。停顿变得尴尬，打断会被截断，用户被迫在 AI 话说到一半时打断，这也显得不太礼貌。换句话说，只有对话以说话的速度进行时，语音 AI 才会感觉自然。

底层的更严格的约束是连续流的特性。音频必须以稳定流的形式到达模型，而不是在用户说完后作为单次上传。正是这种流使得模型能够在用户还在说话时就开始转写、推理和调用工具。一旦流中断，体验就会崩塌成按键通话。

具体到 OpenAI，这些约束转化为三个具体需求：

• 系统必须能触达全球 9 亿周活用户。
• 连接建立必须足够快，让用户能在会话一开始就说话。
• 音频的往返时间必须保持低位和稳定，让话轮转换感觉干脆利落。

WebRTC 是业界为这类工作构建的协议。它是一组小型协议的捆绑（ICE 用于确定两个端点如何穿越防火墙相互到达，DTLS 用于加密通道，SRTP 用于音频包，RTCP 用于质量反馈）。WebRTC 的原始架构师之一 Justin Uberti 和维护 OpenAI 构建所基于的 Pion 库的 Sean DuBois，目前都在 OpenAI 工作。这种协议深度在他们交付的架构中体现了出来。

OpenAI WebRTC 基础设施的第一个版本是一个基于 Pion 构建的单体 Go 服务。它在一个地方处理了两项工作：

• 在信令侧，服务协商 SDP（客户端和服务器用来描述会话的格式）、选择编解码器、生成 ICE 凭据并建立会话。
• 在媒体侧，服务终结来自客户端的 WebRTC 连接，并维护与运行 AI 模型的后端服务（包括推理、转写、语音生成、工具使用和编排）的上行连接。

这个组合服务至今仍在支撑 ChatGPT 语音、实时 API 的 WebRTC 端点以及多个研究项目，并且一直处理得很好。OpenAI 遇到的问题是如何将它部署到 Kubernetes 上——这个容器编排系统运行着大多数现代云基础设施。

Kubernetes 假定计算资源是廉价且可移动的。Pod 会启动，被调度到任何有容量的地方，运行一段时间，然后被重新调度或替换。而标准的 WebRTC 部署模式则恰恰相反。这种不匹配体现在两个具体方面。

第一个是端口耗尽。

部署 WebRTC 的传统方式是为每个会话使用一个 UDP 端口。在 OpenAI 的规模下，这意味着每个服务需要数万个公共 UDP 端口。云负载均衡器是为处理少量众所周知的端口而建的，因此每增加一个端口范围都会增加负载均衡器配置、健康检查、防火墙策略和上线安全方面的运维复杂性。暴露的攻击面也加大了安全审计的难度。Kubernetes 的自动伸缩要求预留大而稳定的端口范围，这使得弹性变得脆弱。

第二个是状态粘性。

每个服务器运行一个 UDP 端口并在其后对会话进行多路复用，这解决了端口问题。然而，ICE 和 DTLS 都是有状态协议。启动了会话的进程必须持续接收其数据包，以验证连通性检查、完成 DTLS 握手、解密 SRTP 以及处理后续的会话变更（如 ICE 重启）。如果某个现有会话的数据包落在了其他进程上，连接建立就会失败，或者媒体传输中断。

这两个压力都指向同一个答案。部署架构必须改变，而客户端体验保持不变。

OpenAI 交付的架构将数据包路由与协议终结分离。

一个无状态的中继位于前端，向互联网暴露很小的公共足迹。一个有状态的收发器位于其后，承载所有繁重的 WebRTC 状态。信令仍然直接发送到收发器。媒体则先进入中继。

中继的职责被刻意缩小。它读取每个数据包的足够信息以选择目的地，然后将剩余部分作为不透明载荷转发。音频在途中保持加密，ICE 状态机留在收发器，编解码器协商在别处进行。从客户端角度看，WebRTC 会话在各方面看起来都正常。

收发器拥有 WebRTC 中必须记住信息的部分。ICE 连通性检查、DTLS 握手、SRTP 加密密钥以及会话生命周期全部驻留在此。收发器是完成握手并对实际媒体进行加密或解密的端点。

参见下图：

团队评估了一个显而易见的替代方案，并选择放弃。

SFU（选择性转发单元）是 WebRTC 大规模部署的标准媒体服务器架构。它为每个参与者终结一个 WebRTC 连接，并有选择地在参与者之间转发流。AI 作为另一个参与者加入。

这对群组通话、课堂和协作会议这类天然的多方产品很有效。OpenAI 的工作负载则不同。大多数会话是 1:1，一个用户对一个模型。对于这种流量，SFU 模型增加了开销，并迫使后端服务本身表现得像 WebRTC 对等体。收发器模型则让后端保持普通服务的形态。

参见下图展示了 SFU 方法：

TURN 也曾被考虑，但被搁置了。

TURN 是用于 NAT 穿透的标准协议终接中继。问题在于 TURN 分配在媒体流动之前会增加建立往返次数，而且跨服务器迁移或恢复 TURN 分配很困难。对于对延迟敏感的工作负载，这些额外的往返代价太大。

分离方案原则上解决了端口和状态问题。剩下的问题是让中继正确地路由第一个数据包。

任何新会话的第一个数据包都是难点。

后续数据包很简单，因为中继有一条映射，记录来自此源 IP 和端口的数据包去往该收发器。第一个数据包创建了这条映射，因此中继必须从数据包本身判断将其发送到哪里。

参见下图：

有两个天真的选项：

• 在热路径上进行数据库查询会增加延迟，并形成对另一个服务健康状态的硬依赖。
• 路由到随机收发器并在内部转发是可行的，但会使跳数加倍。

OpenAI 选择了第三种方案。

答案存在于 WebRTC 已经交换的一个字段里。每个 WebRTC 会话都携带一个 ICE 用户名片段，称为 ufrag，它在会话建立期间产生，并在 STUN 绑定请求中回显。STUN 绑定请求是 ICE 用来验证两个端点能否实际相互通信的数据包，通常也是客户端在媒体路径上发送的第一个数据包。

诀窍在于 OpenAI 在信令阶段生成服务端的 ufrag。他们可以在其中放入任何内容，因此将路由元数据编码进去。中继仅解析第一个 STUN 绑定请求中的足够内容以读取 ufrag，解码出路由提示，并将数据包转发给拥有该会话的收发器。第一个数据包之后的所有包都通过已建立的会话映射流动，完全跳过 ufrag 解析步骤。

参见下图详细展示了连接建立序列：

舰队中的每个收发器都监听一个共享的 UDP 套接字，即绑定到内部 IP 和端口的一个操作系统端点。该收发器的所有会话都在其后多路复用。

在信令期间，收发器在 SDP 应答中返回一个共享的中继 VIP 和 UDP 端口。VIP 是位于整个中继舰队前端的虚拟 IP 地址，因此客户端看到一个稳定的目的地，如 203.0.113.10:3478，尽管背后有许多中继实例。从客户端一侧看，只有一个发送数据包的地方，并且在整个会话生命周期内保持不变。

中继的状态被刻意保持得极小。它持有源地址到收发器目的地的内存映射，外加一些监控计数器和会话清理定时器。如果某个中继实例重启并丢失了映射，下一个 STUN 数据包将从 ufrag 重建它。为了加快恢复速度，一旦路由建立，Redis 缓存就保存源到目的地的映射。重启的中继可以立即从 Redis 查找映射。

此处原则具有普遍性。当我们在热路径上需要数据时，应查看协议已经在交换什么。载荷中的字段解析起来几乎是零成本。新增一次查找则会带来延迟、依赖以及一个可能出错的环节。

一旦公共 UDP 攻击面被缩小为一小组固定地址，同样的中继模式就可以在全球范围部署。

全球中继是 OpenAI 地理分布式中继入口点组成的集群。它们全都运行上述相同的数据包转发行为。唯一变化的是它们在地图上的位置。

地理分布缩短了客户端到 OpenAI 的第一跳。一个在用户附近（无论是地理上还是网络拓扑上）的中继点进入网络的数据包，比必须穿越公共互联网先到达遥远区域的数据包，要轻松得多。实际效果是更低的延迟、更稳定的时序，以及在流量到达 OpenAI 骨干网之前更干净的丢包特征。

OpenAI 在信令侧使用 Cloudflare 进行地理和就近导向。建立会话的初始 HTTP 或 WebSocket 请求被路由到附近的收发器集群。然后，请求上下文决定在 SDP 应答中通告给客户端的是哪个全球中继入口点。ufrag 携带了足够的信息，以便全球中继将媒体路由到正确的集群，再由集群的中继路由到正确的收发器。

结果是，信令和媒体路径均在靠近用户的点进入 OpenAI 网络，而会话本身在整个生命周期内始终锚定在一个特定的收发器上。建立往返和首次 ICE 连通性检查都得以缩短，直接减少了用户在能开始说话之前的等待时间。

中继是一个运行在用户态的 Go 服务，也就是说，一个从普通 UDP 套接字读取数据的常规进程。

Linux 内核从网络接口接收 UDP 数据包，将其投递到绑定到中继 IP 和端口的套接字，然后 Go 进程从该套接字读取数据，更新少量流状态，并将每个数据包转发到正确的收发器。

OpenAI 评估了内核旁路框架（允许用户态进程直接轮询网卡的队列），但选择不用。旁路虽然提高了数据包吞吐量，但代价是运维复杂性。团队的工作负载完全在精细的 Go 实现可以处理的范围内。

三个实现选择承担了大部分性能负载。

• SO_REUSEPORT 是一个 Linux 套接字选项，允许同一台机器上的多个工作进程绑定同一个 UDP 端口。然后内核将传入的数据包分发到这些工作进程，消除了单个读取循环的瓶颈。
• runtime.LockOSThread 将每个读取 UDP 的 goroutine（Go 中的轻量级线程）固定到一个特定的操作系统线程上。与 SO_REUSEPORT 结合，这往往能使同一流的数据包留在同一个 CPU 核心上，有助于缓存局部性并减少上下文切换。
• 在数据包解析期间使用预分配的缓冲区和最小化拷贝，保持低分配开销，避免对 Go 垃圾回收器施加压力。

结论是，常规优化已经足够。该中继承载着 OpenAI 的全球实时媒体流量，其处理能力

Matic is the first visually intelligent robot vacuum that sees your home like you do, so it can clean how you want it to.
Matic’s hero features:

• Runs entirely on cameras to deftly navigate obstacles

• Recognizes floor types to auto-switch between vacuuming and mopping

• Big wheels and a height-adjustable cleaning head handle thick rugs

• Quieter than conversations at 55 dB

• Handles pet hair without clogging or tangling

• A single bag collects dry and wet waste—diaper-salts gel dirty water, antimicrobial powder prevents mold

• A fresh HEPA filter in every bag for cleaner air, no washing or replacing

Experience autonomous cleaning with Matic with a 180-day money-back guarantee.

Get your Matic

---

OpenAI runs voice AI for 900 million users a week, and they use WebRTC for it because the alternative would mean reinventing how the internet handles live audio.

The catch is that WebRTC was designed for servers with stable IPs and ports, and Kubernetes treats those addresses as disposable. The conventional answer at this scale is an SFU, which suits multiparty workloads like group video calls, but OpenAI’s traffic is overwhelmingly one user talking to one model.

To deal with this, their architecture splits the stack into two pieces:

The trick that ties them together is using the ICE ufrag, a field the protocol already exchanges during setup, as a routing key that the relay can read off the first packet of a new session. Everything else, from Global Relay to the userspace Go implementation to the Redis cache and the careful socket-level optimizations, builds on top of that core idea.

In this article, we will look at the entire journey in detail and challenges the OpenAI engineering team faced.

Disclaimer: This post is based on publicly shared details from the Open AI Engineering Team. Please comment if you notice any inaccuracies.

Voice AI either feels like a conversation or it feels like a walkie-talkie. The line between those experiences is measured in milliseconds.

When the network pauses between hearing a user and responding, the illusion breaks. Pauses turn awkward, interruptions get clipped, and users are compelled to cut off the AI mid-sentence, which is also kind of rude. In other words, voice AI only feels natural if the conversation moves at the speed of speech.

The harder constraint underneath is the continuous-stream property. Audio has to arrive at the model as a steady flow, rather than as a single upload after the user finishes talking. That stream is what lets the model start transcribing, reasoning, and calling tools while the user is still speaking. The experience collapses into push-to-talk once it breaks.

For OpenAI specifically, those constraints translate into three concrete requirements:

• The system has to reach 900 million weekly active users wherever they are.

• Connection setup has to be completed quickly enough that users can start speaking as soon as a session begins.

• Round-trip time for audio has to stay low and stable so turn-taking feels crisp.

WebRTC is the protocol the industry built for this kind of work. It is a bundle of smaller protocols (ICE for figuring out how two endpoints reach each other across firewalls, DTLS for encrypting the channel, SRTP for the audio packets, and RTCP for quality feedback). Justin Uberti, one of WebRTC’s original architects, and Sean DuBois, who maintains the Pion library OpenAI builds on, both work at OpenAI today. That kind of protocol depth shows up in the architecture they shipped.

The first version of OpenAI’s WebRTC infrastructure was a single Go service built on Pion. It handled both jobs in one place:

• On the signaling side, the service negotiated SDP (the format clients and servers use to describe a session), selected codecs, generated ICE credentials, and set up sessions.

• On the media side, the service terminated WebRTC connections from clients and maintained upstream connections to the backend services that run the AI models, including inference, transcription, speech generation, tool use, and orchestration.

That combined service still powers ChatGPT voice, the Realtime API’s WebRTC endpoint, and several research projects, and it has handled that work well. The question OpenAI ran into was how to deploy it on Kubernetes, the container orchestration system that runs most modern cloud infrastructure.

Kubernetes assumes compute is cheap and movable. Pods come up, get scheduled wherever capacity exists, run for a while, then get rescheduled or replaced. Standard WebRTC deployment patterns assume the opposite. That mismatch shows up in two specific places.

The first is port exhaustion.

The conventional way to deploy WebRTC uses one UDP port per session. At OpenAI’s scale, that means tens of thousands of public UDP ports per service. Cloud load balancers were built for a handful of well-known ports, so each additional range adds operational complexity for load balancer config, health checks, firewall policy, and rollout safety. The exposed surface area also makes security audits harder. Kubernetes autoscaling clashes with the requirement to reserve large and stable port ranges, which makes elasticity brittle.

The second is state stickiness.

Running one UDP port per server and demultiplexing sessions behind it solves the port problem. ICE and DTLS, however, are stateful protocols. The process that started a session has to keep receiving its packets to validate connectivity checks, complete the DTLS handshake, decrypt SRTP, and process later session changes like ICE restarts. If a packet for an existing session lands on a different process, setup fails, or media breaks.

Both pressures point to the same answer. The deployment architecture has to change while the client experience stays identical.

The architecture OpenAI shipped splits packet routing from protocol termination.

A stateless relay sits at the front, presenting a small public footprint to the internet. A stateful transceiver sits behind it, owning all the heavy WebRTC state. Signaling still goes directly to the transceiver. Media enters through the relay first.

The relay’s scope is deliberately narrow. It reads enough of each packet to choose a destination, then forwards the rest as an opaque payload. Audio stays encrypted on the way through, ICE state machines stay with the transceiver, and codec negotiation happens elsewhere. From a client’s perspective, the WebRTC session looks normal in every way.

The transceiver owns the parts of WebRTC that have to remember things. ICE connectivity checks, the DTLS handshake, SRTP encryption keys, and the session lifecycle all live there. The transceiver is the endpoint that completes the handshakes and encrypts or decrypts the actual media.

See the diagram below:

There was an obvious alternative that the team evaluated and chose against.

An SFU, or Selective Forwarding Unit, is the standard media server architecture for WebRTC at scale. It terminates one WebRTC connection per participant and selectively forwards streams between them. The AI joins as another participant.

This works well for inherently multiparty products like group calls, classrooms, and collaborative meetings. OpenAI’s workload looks different. Most sessions are 1:1, with one user talking to one model. For that kind of traffic, the SFU model adds overhead and forces backend services to behave like WebRTC peers themselves. The transceiver model lets the backend stay an ordinary service.

See the diagram below that shows the SFU approach:

TURN was also considered and set aside.

TURN is the standard protocol-terminating relay used for NAT traversal. The trouble is that TURN allocations add setup round-trips before media can flow, and migrating or recovering them across servers is hard. For a latency-sensitive workload, those extra round-trip matters.

The split solves the port and state problems in principle. The remaining problem is making the relay route the first packet correctly.

The first packet of any new session is the difficult one.

Subsequent packets are easy because the relay has a mapping that says that packets from this source IP and port go to this transceiver. The first packet is what creates that mapping, so the relay has to figure out where to send it from the packet itself.

See the diagram below:

Two naive options were present:

• A database lookup on the hot path adds latency and a hard dependency on another service staying healthy.

• Routing to a random transceiver and forwarding internally works, but doubles the hop count.

OpenAI chose a third option.

The answer lives inside a field that WebRTC already exchanges. Every WebRTC session carries an ICE username fragment, called the ufrag, which is produced during session setup and echoed in STUN binding requests. STUN binding requests are the packets ICE uses to verify that two endpoints can actually reach each other, and they are usually the first thing a client sends on the media path.

The trick is that OpenAI generates the server-side ufrag during signaling. They can put whatever they want in it, so they encode routing metadata into it. The relay parses just enough of the first STUN binding request to read the ufrag, decode the routing hint, and forward the packet to the transceiver that owns the session. Every packet after the first one flows through the established session mapping, which skips the ufrag parsing step entirely.

See the diagram below that shows the connection establishment sequence in detail:

Each transceiver in the fleet listens on a shared UDP socket, which is one operating system endpoint bound to an internal IP and port. All sessions for that transceiver multiplex behind it.

During signaling, the transceiver returns a shared relay VIP and UDP port in the SDP answer. A VIP is a virtual IP address that fronts the entire relay fleet, so the client sees one stable destination like 203.0.113.10:3478, even though many relay instances sit behind that address. From the client’s side, there is one place to send packets, and it stays the same for the life of the session.

The relay’s state is purposefully tiny. It holds an in-memory map of source address to transceiver destination, plus some counters for monitoring and timers for session cleanup. If a relay instance restarts and loses the mapping, the next STUN packet rebuilds it from the ufrag. To make recovery faster, a Redis cache holds the source-to-destination mapping once a route is established. A restarted relay can look up the mapping from Redis immediately.

The principle here generalizes well. When we need data on the hot path, look at what the protocol is already exchanging. A field on the payload is essentially free to parse. A new lookup costs latency, a dependency, and one more thing that can break.

Once the public UDP surface was reduced to a small fixed set of addresses, the same relay pattern became deployable globally.

Global Relay is OpenAI’s fleet of geographically distributed relay ingress points. All of them run the identical packet-forwarding behavior described above. The only thing that changes is where on the map they sit.

Geographic distribution shortens the first client-to-OpenAI hop. A packet entering the network at a relay close to the user, in both geography and network topology, has a much easier time than a packet that has to traverse the public internet to reach a distant region first. The practical effect is lower latency, more stable timing, and a cleaner loss profile before traffic reaches the OpenAI backbone.

OpenAI uses Cloudflare for geographic and proximity steering on the signaling side. The initial HTTP or WebSocket request that sets up a session is routed to a nearby transceiver cluster. The request context then determines which Global Relay ingress point gets advertised back to the client in the SDP answer. The ufrag carries enough information for Global Relay to route media to the right cluster, and for the cluster’s relay to route to the right transceiver.

The result is that both the signaling and the media paths enter the OpenAI network at points close to the user, while the session itself stays anchored to one specific transceiver for its full lifetime. The setup round-trip and the first ICE connectivity check both shorten, which directly reduces how long a user waits before they can start speaking.

The relay is a Go service running in userspace, which is to say a regular process that reads from a regular UDP socket.

The Linux kernel receives UDP packets from the network interface, delivers them to a socket bound to the relay’s IP and port, and the Go process reads from that socket, updates a small amount of flow state, and forwards each packet to the right transceiver.

OpenAI evaluated kernel-bypass frameworks (which let a userspace process poll the network card’s queues directly) and chose to stay away. Bypass raises packet throughput at the cost of operational complexity. The team’s workload fit inside what a careful Go implementation could handle.

Three implementation choices carry most of the performance load.

• SO_REUSEPORT is a Linux socket option that lets multiple workers on the same machine bind the same UDP port. The kernel then distributes incoming packets across those workers, which removes the bottleneck of a single read loop.

• runtime.LockOSThread pins each UDP-reading goroutine (a lightweight thread in Go) to a specific OS thread. Combined with SO_REUSEPORT, this tends to keep packets from the same flow on the same CPU core, which helps cache locality and reduces context switching.

• Pre-allocated buffers and minimal copying during packet parsing keep allocation overhead low and avoid putting pressure on Go’s garbage collector.

The takeaway is that ordinary optimizations were enough. The relay handles OpenAI’s global real-time media traffic on a relatively small footprint.

Every architecture comes with tradeoffs, and this one carries several worth understanding.

• The whole design is built around 1:1 sessions. If OpenAI ever wants to add multiparty features (group voice calls, multiple participants in a single AI session, human handoff during a call), large parts of this architecture would probably need rework. Both the transceiver model and the choice to skip SFU rely on the assumption that most sessions have exactly two endpoints.

• OpenAI also took on a custom infrastructure burden. A standard SFU comes with documentation, a community, and battle-tested patterns. The relay, the transceiver, and the coordination between them are all internal code. Every engineer who works on this stack has to learn it from the inside out.

• The “stateless” relay turns out to be stateless mostly in spirit. It holds an in-memory flow table and uses a Redis cache to recover that table across restarts. The architecture works because the protocol can rebuild the table from the ufrag, but the soft state is still a state.

• Lastly, the ufrag trick depends on controlling both ends of signaling. OpenAI generates the server-side ufrag, so they can embed routing metadata in it freely. A team that uses an off-the-shelf signaling stack might find this technique harder to adapt directly.

The architecture OpenAI built for voice AI is a careful response to a specific pressure. WebRTC was designed for stable servers. Modern cloud infrastructure runs on the opposite assumption. OpenAI’s team had the protocol depth to determine whether a WebRTC session needs to live in one process at all.

Their answer separates the work into two pieces. A stateless relay forwards packets near the user. A stateful transceiver, anchored in one place, owns ICE, DTLS, SRTP, and the session lifecycle. The two pieces communicate through information that’s already in the WebRTC handshake, which keeps the routing decision on the packet path itself.

The implementation choices stayed deliberately simple. Userspace Go, SO_REUSEPORT, thread pinning, and careful memory management did the work that kernel bypass solves. The result handles 900 million weekly users on a relatively small relay footprint.

References: