Show HN：为你的 AI Agent 提供屏幕点击指引

嘿，HN。我是 Christian，Frigade（YC W23）的联合创始人之一。我注意到，许多应用内 AI Agent 很难真正理解它们所服务的产品。

例如，假设用户询问某个 Agent，如何在某个 SaaS 产品中完成一项操作。理想情况下，Agent 可能会回复说它拥有完成该任务的工具，并直接为用户全自动执行。这是很好的结果。

但很多时候并非如此。也许没有针对该确切任务的工具调用，或者用户的问题最适合通过特定的 UI 工作流或界面来解决。在这些情况下，许多 Agent 往往会退回到基于帮助中心的基础 RAG，有时甚至会通过搜索互联网来了解它们自己的产品。这可能是一个非常缓慢的过程，而且随着如今产品迭代速度快于帮助中心文章更新速度，帮助中心文章大多数时候都已过时。更糟糕的是，没有人喜欢阅读一长串要点，再将其对应回 UI。

我的工具（Assist API）通过如下定义的一次工具调用来解决这一缺口：

const frigade_guide_tool = { description: 'Call this tool to answer product questions or guide the user through a task.', parameters: { query: { type: 'string', description: 'What the user is asking or wants to do', }, }, run: ({ query }) => frigade.assist({ query }), }

以下是我录制的一段演示，介绍如何通过 Vercel AI SDK 进行设置：

视频/嵌入内容：使用 Vercel AI SDK 设置 Assist API 的演示

调用后，该工具将执行以下操作：

1. 收集用户当前在屏幕上看到的内容、其权限、功能开关等上下文信息。随后执行以下操作之一：

2a) 如果可以解决：生成一份屏幕内指引，说明如何解决给定问题。

2b) 如果属于概念性问题：向父 Agent 返回描述如何解决该问题的文本。

2c) 拒绝（即无法提供帮助）。

工具如何知道该做什么？

该工具使用基于浏览器的 Agent 学习特定应用的 UI。你需要为自己的软件提供一个测试账户（即 staging 或 preview 环境），浏览器 Agent 会登录并逐步浏览整个产品。随后，它会建立一份关于应用如何运作的地图，可以据此查询任何与产品相关的问题，或者了解如何在 UI 中从 A 到 B。它还会基于这份地图自行编写文档。该 Agent 会按计划重新运行，也可以通过 CI/CD 触发。

文档及更多详情：https://frigade.com/assist-api

Hey HN. I'm Christian, one of the founders of Frigade (YC W23). I've noticed that a lot of in-app AI agents struggle to actually understand the products they exist in.

For instance, let's say a user asks an agent how to do something in a given SaaS product. In an ideal case, maybe that agent replies saying it has a tool to do the task and just automates that work entirely for the user. That's a great outcome.

But often that's not the case. Maybe there is no tool call for that exact task, or maybe the user's question is best solved by a specific UI workflow or interface. In these cases, many agents tend to fall back on basic RAG on their help center, or sometimes even searching the internet for an understanding of their own product. This can be a very slow process and most of the time help center articles are outdated as products evolve faster than them today. Even worse, no one likes reading a long list of bullets and mapping that back to a UI.

My tool (Assist API) solves this gap with a single tool call defined like this:

  const frigade_guide_tool = {
    description: 'Call this tool to answer product questions or guide the user through a task.',
    parameters: {
      query: {
        type: 'string',
        description: 'What the user is asking or wants to do',
      },
    },
    run: ({ query }) => frigade.assist({ query }),
  }

Here's a demo I recorded on how to set it up with the Vercel AI SDK:

https://www.youtube.com/watch?v=9WQ0UbLjC6I

When called, the tool will do the following:

1. Gather context on what the user is seeing on screen, their permissions, feature flags, and more. Then one of the following:

2a) If solvable: Generate an on screen guide for how to fix a given problem

2b) If conceptual: Return text describing to the parent agent how to solve the problem

2c) Reject (i.e. unable to help)

How does the tool know what to do?

The tool learns a given application UI by using a browser-based agent. You provide a test account to your software (i.e. staging og preview), and a browser agent logs in and works its way through the entire product. It then builds its own map of how the application works which can the be queried about any product-related question or how to get from A to B in the UI. It also writes its own documentation from this map. The agent re-runs on a schedule or can be triggered through CI/CD.

Docs and more details: https://frigade.com/assist-api