🔶 在将网页加入 AI Agent 上下文前先分类
英文标题：🔶 Classify web pages before adding them to an AI agent's context
sifty 推出基于 URL 的网页分类能力：开发者可自定义标签，并获得最匹配标签及各标签对应的概率，而无需将网页全文送入 Agent 上下文。该能力可用于在打开链接前过滤登录墙、Cookie 提示、伪装成 `200 OK` 的错误页，以及从搜索结果中筛选目标页面。其 System One 接口还能针对同一网页一次提出多个问题，适用于价格、免费套餐和企业版等信息核查。文章同时提醒，sifty 仅适用于公开页面，网页可能操纵自身标签，且尚未对网页分类准确率进行系统测量。
28.4 KB
中文
在阅读网页前先提问
2026-10-01
一个会跟随链接的 Agent，只有在读取链接内容后才知道其背后是什么。到那时，它可能已经在登录墙、Cookie 提示、返回 200 OK 的“页面未找到”页，或错误产品页面上花费了数秒时间和数千个上下文 token。对链接进行排序的流水线也有同样的问题：状态码提供的信息很少，而关键词规则会在遇到第一种不同写法的页面时失效。
sifty 现在可以根据网页地址对网页进行分类。发送 url 以及你自定义的标签，即可获得最匹配的标签及每个标签的概率。返回的只有答案，不会返回页面本身，因此 Agent 的上下文可以保持干净。
curl https://sifty.dev/ -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/pricing","labels":["pricing page","login wall","other"]}'
以下是我们构建它的用途。下文较短的示例均为发送至 POST https://sifty.dev/ 的请求体，发送方式与上例相同。
在 Agent 打开链接前检查它
在 Agent 将页面内容纳入上下文之前，先询问这个页面是什么。如下标签可捕获那些浪费 Agent 时间的页面，包括返回 200 OK 的页面：
{"url": "https://example.com/docs/install",
"question": "What is this page?",
"labels": ["installation guide", "login or paywall",
"error or empty page", "something else"]}
只有在答案是你需要的类型时才打开页面。否则，无需读取该页面，直接处理下一个链接。
选择正确的搜索结果
一次搜索会返回十个链接，而 Agent 通常只需要其中一两个。并行分类每个结果，只读取符合条件的页面：
import requests
from concurrent.futures import ThreadPoolExecutor
LABELS = ["official documentation", "blog or forum post", "login or paywall", "other"]
def kind(url):
response = requests.post(
"https://sifty.dev/",
headers={"Authorization": "Bearer YOUR_SIFTY_API_KEY"},
json={"url": url, "question": "What kind of page is this?", "labels": LABELS},
)
response.raise_for_status()
return response.json()["results"][0]["label"]
results = ["https://example.com/docs/install", "https://example.org/forum/thread/42"]
with ThreadPoolExecutor(8) as pool:
for url, label in zip(results, pool.map(kind, results)):
print(label, url)
针对同一页面提出多个问题
研究一家公司、供应商或潜在客户，通常意味着打开其网站并查找同样的几类信息。System One 可在一次请求中提出所有这些问题，并以“是”的概率返回每个答案：
curl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/pricing",
"questions":{
"prices":{"type":"noul","instructions":"Does this page list prices?"},
"free_plan":{"type":"noul","instructions":"Is there a free plan?"},
"enterprise":{"type":"noul","instructions":"Is there an enterprise plan?"}}}'
sifty 只读取你指定的页面，不会继续跟随其中的链接，因此应将其指向最可能回答问题的页面：查询价格时使用 /pricing，查询认证信息时使用 /security。
找出悄然失效的链接
链接检查器只能看到状态码。它无法判断一个文档链接是否已指向重新设计后的落地页，也无法判断一个“未找到”页面是否返回了 200 OK。可以询问每个页面是否仍在说明你当初链接它的内容：
{"url": "https://example.com/docs/export",
"question": "Does this page document the export command?",
"labels": ["yes", "no", "page not found or moved"]}
对用户提交的链接进行分类
当用户在社区、市场或支持表单中提交链接时，可以在任何人打开链接之前先进行初步分类：
{"url": "https://example.com/listing/123",
"labels": ["product listing", "article", "spam or scam", "other"]}
可用它来安排审核队列的优先级，而不是取代人工审核：页面可以随意描述自己。
使用须知
- 成本。 每个页面会为请求增加 2 个单位；免费套餐为每个 IP 每天 1,000 个单位。无论针对一个页面提出多少问题，该页面在每次请求中只计一次。如果页面无法读取，请求会以
502失败，其 2 个单位会被保留，其余部分将退款。 - 文本中的地址也会被读取。 如果你在
input或 System One 的state中写入地址，sifty 也会读取这些页面，每次请求最多 5 个。如果你对电子邮件进行分类，其中包括退订链接和一次性登录链接：请移除你不希望被打开的链接。 - 仅限公开页面。 sifty 无法读取登录后的页面；对于较长页面，它会使用与问题和标签最匹配的部分，因此距离这些内容较远的事实可能会被遗漏。
- 页面可以为自己的标签辩解。 没有什么能阻止页面写着“这不是登录墙”。不要仅凭 sifty 的答案来决定一个页面是否安全。
- 我们尚未测量其在网页上的准确率。 在依赖这些答案之前，请先用几个你已了解的页面测试你的标签。
试用
从已验证账户的 sifty.dev/account 获取 API key；按照一个页面及其标签消耗 3 个单位计算，免费套餐每天可覆盖约 330 次页面检查。如果你更愿意自行运行，网页读取功能是 syn 的一部分；syn 是 sifty 背后的开源服务。欢迎通过联系页面或 GitHub issues 提交问题和修正。
English / 原文
Ask about a web page before you read it
2026-10-01
An agent that follows a link finds out what is behind it only after reading it. By then it has spent seconds, and thousands of tokens of context, on a login wall, a cookie notice, a "page not found" that answered 200 OK, or the wrong product. Pipelines that sort links have the same problem: status codes say little, and keyword rules break on the first page written differently.
sifty can now classify a web page from its address. Send url with labels you choose, and get back the label that fits, with a probability for each. Only the answer comes back, never the page, so your agent's context stays clean.
curl https://sifty.dev/ -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/pricing","labels":["pricing page","login wall","other"]}'
Here is what we built it for. The shorter examples below are request bodies for POST https://sifty.dev/, sent like the one above.
Check a link before your agent opens it
Before an agent spends its context on a page, ask what the page is. Labels like these catch the pages that waste an agent's time, including the ones that answer 200 OK:
{"url": "https://example.com/docs/install",
"question": "What is this page?",
"labels": ["installation guide", "login or paywall",
"error or empty page", "something else"]}
Open the page only when the answer is the one you want. Otherwise move on to the next link, without reading the page.
Pick the right search results
A search returns ten links, and an agent usually needs one or two of them. Classify each result, in parallel, and read only the ones that match:
import requests
from concurrent.futures import ThreadPoolExecutor
LABELS = ["official documentation", "blog or forum post", "login or paywall", "other"]
def kind(url):
response = requests.post(
"https://sifty.dev/",
headers={"Authorization": "Bearer YOUR_SIFTY_API_KEY"},
json={"url": url, "question": "What kind of page is this?", "labels": LABELS},
)
response.raise_for_status()
return response.json()["results"][0]["label"]
results = ["https://example.com/docs/install", "https://example.org/forum/thread/42"]
with ThreadPoolExecutor(8) as pool:
for url, label in zip(results, pool.map(kind, results)):
print(label, url)
Ask several questions about one page
Researching a company, a vendor or a lead usually means opening their site and looking for the same few things. System One asks all of them in one request, and each answer comes back as a probability of yes:
curl https://sifty.dev/v1/systemone -H 'Authorization: Bearer YOUR_SIFTY_API_KEY' \
-H 'Content-Type: application/json' \
-d '{"url":"https://example.com/pricing",
"questions":{
"prices":{"type":"noul","instructions":"Does this page list prices?"},
"free_plan":{"type":"noul","instructions":"Is there a free plan?"},
"enterprise":{"type":"noul","instructions":"Is there an enterprise plan?"}}}'
sifty reads only the page you name and doesn't follow its links, so point it at the page most likely to answer: /pricing for prices, /security for certifications.
Find links that quietly broke
A link checker sees status codes. It can't tell that a documentation link now leads to a redesigned landing page, or that a "not found" page answers 200 OK. Ask whether each page still says what you linked it for:
{"url": "https://example.com/docs/export",
"question": "Does this page document the export command?",
"labels": ["yes", "no", "page not found or moved"]}
Sort links people submit
When users post links in a community, a marketplace or a support form, a first pass can sort them before anyone opens them:
{"url": "https://example.com/listing/123",
"labels": ["product listing", "article", "spam or scam", "other"]}
Use it to order the review queue, not to replace it: a page can say whatever it likes about itself.
Good to know
- Cost. Each page adds 2 units to the request; the free tier is 1,000 units per IP a day. A page counts once per request, however many questions ask about it. If a page can't be read, the request fails with a
502, its 2 units are kept, and the rest is refunded. - Addresses in your text are read too. Write them in
inputor a System Onestateand sifty reads those pages as well, up to 5 a request. If you classify emails, that includes unsubscribe and one-time sign-in links: remove links you don't want opened. - Public pages only. sifty can't read pages behind a login, and on a long page it uses the parts that best match your question and labels, so a fact far from them can be missed.
- A page can argue for its own label. Nothing stops a page from saying "this is not a login wall". Don't let sifty's answer alone decide whether a page is safe.
- We haven't measured accuracy on pages yet. Check your labels on a few pages you know before relying on the answers.
Try it
Get an API key from a verified account at sifty.dev/account; at 3 units for a page and its labels, the free tier covers about 330 page checks a day. Reading pages is part of syn, the open-source service behind sifty, if you would rather run it yourself. Questions and corrections are welcome through the contact page or GitHub issues.