AgentHands：为 XR 中具备空间感知的 Agent 对话生成交互式手势

随着 AI 助手从简单的文本界面演变为多模态伙伴，我们正看到交互方式朝着更主动、更具情境感知能力的辅助方向转变。诸如 Project Astra 和 Gemini 3.1 Flash Live 等近期创新成果，已经能够让用户实时讨论自己所处的物理环境，并常常利用视觉边界框叠加层来识别摄像头画面中的物体。尽管这些叠加层在 2D 屏幕上非常有效，但当交互迁移至 Android XR 等沉浸式平台时，一个独特的挑战随之出现：我们如何超越平面的 UI，打造真正具身化且具有空间感知能力的对话？

为弥合这一差距，我们推出了 AgentHands。这是一个发表于 CHI 2026 的研究原型，它将伴随语音的手势能力带入 3D 世界。在人类交流中，我们的双手不只是用于指向；它们还会描述形状、模仿动作、强调重点，并与声音保持同步。借助 扩展现实（XR）的空间理解能力，AgentHands 复现了这种自然的协同方式。作为我们此前在 Human I/O 和 Sensible Agent 研究基础上的延续，AgentHands 进一步为 AI Agent 配备了富有表现力、与语音同步的手势，将抽象的口头指令转化为直观的物理演示，使围绕用户周边环境展开的对话更加自然、更具吸引力。

As AI assistants evolve from simple text interfaces to multimodal companions, we are seeing a shift toward more proactive, situated assistance. Recent innovations like Project Astra and Gemini 3.1 Flash Live already allow users to discuss their physical surroundings in real time, often utilizing visual bounding box overlays to identify objects in a camera feed. While these overlays are highly effective for 2D screens, the transition to immersive platforms like Android XR presents a unique challenge: how do we move beyond flat UI to create a truly embodied, spatially aware dialogue?

To bridge this gap, we introduce AgentHands, published at CHI 2026, a research prototype that brings the power of co-speech gestures to the 3D world. In human communication, our hands do more than just point; they describe shapes, mimic actions, and emphasize points, all synchronized with our voice. By leveraging the spatial understanding capabilities of Extended Reality (XR), AgentHands replicates this natural synergy. Following up our prior research in Human I/O and Sensible Agent, AgentHands further equips AI agents with expressive, synchronized hand gestures that transform abstract verbal instructions into intuitive, physical demonstrations, making conversations about your surroundings more natural and engaging.