LiveKit Agents：构建实时语音 AI Agent 的框架

正在寻找 JS/TS 库？请查看 AgentsJS

Agent Framework 专为构建运行在服务器上的实时、可编程参与者而设计。你可以用它创建能够看、听和理解的对话式多模态语音 Agent。

• 灵活集成：提供完善的生态系统，可自由组合适合用例的 STT、LLM、TTS 和 Realtime API。
• 集成式任务调度：通过内置的任务调度和分发能力，以及 dispatch APIs，将终端用户连接至 Agent。
• 丰富的 WebRTC 客户端：利用 LiveKit 的开源 SDK 生态构建客户端应用，支持所有主流平台。
• 电话集成：可与 LiveKit 的电话技术栈无缝协作，让 Agent 能够拨打或接听电话。
• 与客户端交换数据：使用 RPCs 和其他 Data APIs，与客户端无缝交换数据。
• 语义轮次检测：使用 Transformer 模型判断用户何时结束当前轮次，有助于减少打断。
• MCP 支持：原生支持 MCP。只需一行代码即可集成 MCP 服务器提供的工具。
• 内置测试框架：编写测试并使用裁判（judges），确保 Agent 按预期运行。
• 开源：完全开源，允许你在自己的服务器上运行整套技术栈，包括最广泛使用的 WebRTC 媒体服务器之一——LiveKit server。

要安装核心 Agents 库及热门模型提供商的插件：

pip install "livekit-agents[openai,deepgram,cartesia]"

有关该框架及其使用方法的文档，请参阅此处。

使用 AI 编程 Agent 进行开发

如果你使用 AI 编程助手基于 LiveKit Agents 进行开发，我们建议采用以下配置以获得最佳效果：

1. 安装 LiveKit Docs MCP server —— 为你的编程 Agent 提供最新 LiveKit 文档、跨 LiveKit 仓库的代码搜索以及可运行示例。

2. 安装 LiveKit Agent Skill —— 为你的编程 Agent 提供构建语音 AI 应用的架构指导和最佳实践，包括工作流设计、交接、任务和测试模式。

   npx skills add livekit/agent-skills --skill livekit-agents

Agent Skill 与 MCP server 配合使用效果最佳：Skill 教导你的 Agent 如何着手 使用 LiveKit 进行开发，而 MCP server 则提供正确实现所需的_当前 API 细节_。

• Agent：一个具有明确指令、基于 LLM 的应用。
• AgentSession：Agent 的容器，负责管理与终端用户的交互。
• entrypoint：交互式会话的起点，类似 Web 服务器中的请求处理程序。
• AgentServer：负责协调任务调度并为用户会话启动 Agent 的主进程。

---

from livekit.agents import ( Agent, AgentServer, AgentSession, JobContext, RunContext, cli, function_tool, inference, )

@function_tool async def lookup_weather( context: RunContext, location: str, ): """Used to look up weather information."""

return {"weather": "sunny", "temperature": 70}

server = AgentServer()

@server.rtc_session() async def entrypoint(ctx: JobContext): session = AgentSession( vad=inference.VAD(), # any combination of STT, LLM, TTS, or realtime API can be used # this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud # to use model provider keys directly, replace with the following: # from livekit.plugins import deepgram, openai, cartesia # stt=deepgram.STT(model="nova-3"), # llm=openai.LLM(model="gpt-4.1-mini"), # tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"), stt=inference.STT("deepgram/nova-3", language="multi"), llm=inference.LLM("google/gemma-4-31b-it"), # low-latency gemma, hosted on LiveKit tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"), )

agent = Agent(
    instructions="You are a friendly voice assistant built by LiveKit.",
    tools=[lookup_weather],
)

await session.start(agent=agent, room=ctx.room)
await session.generate_reply(instructions="greet the user and ask about their day")

if name == "main": cli.run_app(server)

运行此示例需要以下环境变量：

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

---

此代码片段经过缩写。完整示例请参阅 multi_agent.py。

... class IntroAgent(Agent): def init(self) -> None: super().init( instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging." "Ask the user for their name and where they are from" )

async def on_enter(self):
    self.session.generate_reply(instructions="greet the user and gather information")

@function_tool
async def information_gathered(
    self,
    context: RunContext,
    name: str,
    location: str,
):
    """Called when the user has provided the information needed to make the story personalized and engaging.

    Args:
        name: The name of the user
        location: The location of the user
    """

    context.userdata.name = name
    context.userdata.location = location

    story_agent = StoryAgent(name, location)
    return story_agent, "Let's start the story!"

class StoryAgent(Agent): def init(self, name: str, location: str) -> None: super().init( instructions=f"You are a storyteller. Use the user's information in order to make the story personalized." f"The user's name is {name}, from {location}", # override the default model, switching to Realtime API from standard LLMs llm=openai.realtime.RealtimeModel(voice="echo"), chat_ctx=chat_ctx, )

async def on_enter(self):
    self.session.generate_reply()

@server.rtc_session() async def entrypoint(ctx: JobContext): userdata = StoryData() session = AgentSession[StoryData]( vad=inference.VAD(), stt="deepgram/nova-3", llm="google/gemma-4-31b-it", # low-latency gemma, hosted on LiveKit tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc", userdata=userdata, )

await session.start(
    agent=IntroAgent(),
    room=ctx.room,
)

...

自动化测试对于构建可靠的 Agent 至关重要，尤其是在 LLM 具有非确定性行为的情况下。LiveKit Agents 包含原生测试集成，帮助你创建可信赖的 Agent。

@pytest.mark.asyncio async def test_no_availability() -> None: llm = google.LLM() async with AgentSession(llm=llm) as sess: await sess.start(MyAgent()) result = await sess.run( user_input="Hello, I need to place an order." ) result.expect.skip_next_event_if(type="message", role="assistant") result.expect.next_event().is_function_call(name="start_order") result.expect.next_event().is_function_call_output() await ( result.expect.next_event() .is_message(role="assistant") .judge(llm, intent="assistant should be asking the user what they would like") )

更多示例和详细设置说明，请参阅 examples 目录。如需更多示例，请参阅 python-agents-examples 仓库。

针对语音对话优化的入门 Agent。代码 |

🔄 多用户按键通话

通过按键通话响应房间内的多个用户。代码
---|---
用于提升真实感的后台环境音和思考音频。代码 | 动态创建函数工具。代码
可拨打外呼电话的 Agent 代码 | 使用 LLM 的结构化输出引导 TTS 语气。代码
使用 MCP 服务器中的工具 代码 | 完全跳过语音，并使用相同代码实现纯文本集成 代码
生成房间内所有用户的转录文本 代码 | 使用 Tavus、Bithuman、LemonSlice 等添加 AI Avatar 代码

🍽️ 餐厅点餐与预订

可处理餐厅来电的 Agent 完整示例。代码 | 可见的 Gemini Live Agent 完整示例（包括 iOS 应用）。代码

python myagent.py console

在终端模式下运行 Agent，支持本地音频输入和输出以进行测试。该模式不需要外部服务器或依赖，适合快速验证行为。

使用 LiveKit 客户端进行开发

启动 Agent server，并在文件变更时启用热重载。该模式可让每个进程高效承载多个并发 Agent。

Agent 会连接到 LiveKit Cloud 或你的自托管服务器。请设置以下环境变量：

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

你可以通过任意 LiveKit 客户端 SDK 或电话集成进行连接。要快速开始，请试用 Agents Playground。

以适用于生产环境的优化方式运行 Agent。

Agents framework 采用 Apache-2.0 许可证。LiveKit 轮次检测模型采用 LiveKit Model License 许可证。

Agents framework 所处领域快速演进，项目目前仍在积极开发中。我们欢迎并感谢各种形式的贡献，包括反馈、错误修复、功能、新插件和工具，或更好的文档。你可以在该仓库提交 issue、发起 PR，或在 LiveKit 社区与我们交流。

该项目使用 uv 管理包。要安装用于开发的依赖：

uv sync --all-extras --dev

该项目在 examples 目录中包含大量示例。要运行这些示例，请创建文件 examples/.env，其中包含 LiveKit Server 的凭据及

Looking for the JS/TS library? Check out AgentsJS

The Agent Framework is designed for building realtime, programmable participants that run on servers. Use it to create conversational, multi-modal voice agents that can see, hear, and understand.

• Flexible integrations : A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
• Integrated job scheduling : Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
• Extensive WebRTC clients : Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.
• Telephony integration : Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
• Exchange data with clients : Use RPCs and other Data APIs to seamlessly exchange data with clients.
• Semantic turn detection : Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.
• MCP support : Native support for MCP. Integrate tools provided by MCP servers with one line of code.
• Builtin test framework : Write tests and use judges to ensure your agent is performing as expected.
• Open-source : Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.

To install the core Agents library, along with plugins for popular model providers:

pip install "livekit-agents[openai,deepgram,cartesia]"

Documentation on the framework and how to use it can be found here

Building with AI coding agents

If you're using an AI coding assistant to build with LiveKit Agents, we recommend the following setup for the best results:

1. Install theLiveKit Docs MCP server — Gives your coding agent access to up-to-date LiveKit documentation, code search across LiveKit repositories, and working examples.

2. Install theLiveKit Agent Skill — Provides your coding agent with architectural guidance and best practices for building voice AI applications, including workflow design, handoffs, tasks, and testing patterns.

   npx skills add livekit/agent-skills --skill livekit-agents
   

The Agent Skill works best alongside the MCP server: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details to implement it correctly.

• Agent: An LLM-based application with defined instructions.
• AgentSession: A container for agents that manages interactions with end users.
• entrypoint: The starting point for an interactive session, similar to a request handler in a web server.
• AgentServer: The main process that coordinates job scheduling and launches agents for user sessions.

---

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)


@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""

    return {"weather": "sunny", "temperature": 70}


server = AgentServer()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        # any combination of STT, LLM, TTS, or realtime API can be used
        # this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
        # to use model provider keys directly, replace with the following:
        # from livekit.plugins import deepgram, openai, cartesia
        # stt=deepgram.STT(model="nova-3"),
        # llm=openai.LLM(model="gpt-4.1-mini"),
        # tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),  # low-latency gemma, hosted on LiveKit
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions="You are a friendly voice assistant built by LiveKit.",
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask about their day")


if __name__ == "__main__":
    cli.run_app(server)

You'll need the following environment variables for this example:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

---

This code snippet is abbreviated. For the full example, see multi_agent.py

...
class IntroAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging."
            "Ask the user for their name and where they are from"
        )

    async def on_enter(self):
        self.session.generate_reply(instructions="greet the user and gather information")

    @function_tool
    async def information_gathered(
        self,
        context: RunContext,
        name: str,
        location: str,
    ):
        """Called when the user has provided the information needed to make the story personalized and engaging.

        Args:
            name: The name of the user
            location: The location of the user
        """

        context.userdata.name = name
        context.userdata.location = location

        story_agent = StoryAgent(name, location)
        return story_agent, "Let's start the story!"


class StoryAgent(Agent):
    def __init__(self, name: str, location: str) -> None:
        super().__init__(
            instructions=f"You are a storyteller. Use the user's information in order to make the story personalized."
            f"The user's name is {name}, from {location}",
            # override the default model, switching to Realtime API from standard LLMs
            llm=openai.realtime.RealtimeModel(voice="echo"),
            chat_ctx=chat_ctx,
        )

    async def on_enter(self):
        self.session.generate_reply()


@server.rtc_session()
async def entrypoint(ctx: JobContext):
    userdata = StoryData()
    session = AgentSession[StoryData](
        vad=inference.VAD(),
        stt="deepgram/nova-3",
        llm="google/gemma-4-31b-it",  # low-latency gemma, hosted on LiveKit
        tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
        userdata=userdata,
    )

    await session.start(
        agent=IntroAgent(),
        room=ctx.room,
    )
...

Automated tests are essential for building reliable agents, especially with the non-deterministic behavior of LLMs. LiveKit Agents include native test integration to help you create dependable agents.

@pytest.mark.asyncio
async def test_no_availability() -> None:
    llm = google.LLM()
    async with AgentSession(llm=llm) as sess:
        await sess.start(MyAgent())
        result = await sess.run(
            user_input="Hello, I need to place an order."
        )
        result.expect.skip_next_event_if(type="message", role="assistant")
        result.expect.next_event().is_function_call(name="start_order")
        result.expect.next_event().is_function_call_output()
        await (
            result.expect.next_event()
            .is_message(role="assistant")
            .judge(llm, intent="assistant should be asking the user what they would like")
        )

For more examples and detailed setup instructions, see the examples directory. For even more examples, see the python-agents-examples repository.

A starter agent optimized for voice conversations. Code |

🔄 Multi-user push to talk

Responds to multiple users in the room via push-to-talk. Code
---|---
Background ambient and thinking audio to improve realism. Code | Creating function tools dynamically. Code
Agent that makes outbound phone calls Code | Using structured output from LLM to guide TTS tone. Code
Use tools from MCP servers Code | Skip voice altogether and use the same code for text-only integrations Code
Produce transcriptions from all users in the room Code | Add an AI avatar with Tavus, Bithuman, LemonSlice, and more Code

🍽️ Restaurant ordering and reservations

Full example of an agent that handles calls for a restaurant. Code | Full example (including iOS app) of Gemini Live agent that can see. Code

python myagent.py console

Runs your agent in terminal mode, enabling local audio input and output for testing. This mode doesn't require external servers or dependencies and is useful for quickly validating behavior.

Developing with LiveKit clients

Starts the agent server and enables hot reloading when files change. This mode allows each process to host multiple concurrent agents efficiently.

The agent connects to LiveKit Cloud or your self-hosted server. Set the following environment variables:

• LIVEKIT_URL
• LIVEKIT_API_KEY
• LIVEKIT_API_SECRET

You can connect using any LiveKit client SDK or telephony integration. To get started quickly, try the Agents Playground.

Runs the agent with production-ready optimizations.

The Agents framework is licensed under Apache-2.0. The LiveKit turn detection models are licensed under the LiveKit Model License.

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the LiveKit community.

This project uses uv for package management. To install dependencies for development:

uv sync --all-extras --dev

This project includes many examples in the examples directory. To run them, create the file examples/.env with credentials for LiveKit Server and any necessary model providers (see examples/.env.example), then run:

uv run examples/voice_agents/basic_agent.py dev

For more information, see the examples README.

Unit tests are in the tests directory and can be run with:

Integration tests for each plugin require various API credentials and run automatically in GitHub CI for PRs submitted by project maintainers. See the tests workflow for details.

This project uses ruff for formatting and linting:

uv run ruff format
uv run ruff check --fix

To generate docs locally with pdoc:

uv sync --all-extras --group docs
uv run --active pdoc --skip-errors --html --output-dir=docs livekit