AI 巨头联手支持全新退出机制：允许网站统一拒绝数据抓取与训练

正如《搜索引擎圆桌》（Search Engine Roundtable）此前报道的那样，OpenAI、Google、微软和 Anthropic 共同表示，它们将采纳一项全新标准，使内容发布商能够自主选择不被 AI 爬取数据。通过这一控制机制，发布商可以一次性将自己的 AI 训练偏好同步应用于多家 AI 公司，而无需针对每家机构分别进行单独设置。

这一共享“策略标识符”（policy token）信号由 Anthropic 标准技术负责人 Kevin Kelley 作为互联网工程任务组（IETF）的一部分提出。它将被写入网站的 robots.txt 文件中。长期以来，该文件一直用于标识哪些网络爬虫可以或不可抓取该站点——无论目的是搜索引擎索引、存档还是 AI 训练。而在目前，各站点必须针对每一家 AI 公司分别表明其 AI 训练偏好。

该声明发布之际，内容发布商正疲于应对不断涌入、用于喂养 AI 模型的爬虫。一些公司和组织此前已尝试抗击未经授权的抓取行为，例如通过极简许可（RSL）标准让发布商有能力对访问其站点的爬虫进行收费，以及 Cloudflare 默认屏蔽 AI 爬虫。

OpenAI、Google、微软和 Anthropic 均表示，当遇到该共享标识时，它们将限制其 AI 爬虫访问相应站点。Meta 通信主管 Andy Stone 也表达了对新设置的支持，但尚未明确该公司是否会遵循该标识。Stone 表示：“我们认为，创建一个能够同时向多个服务商传递此类偏好的通用信号是重要的一步。我们期待与其他 AI 公司合作，推动行业标准的建立与实施。”

微软表示，它将提供细粒度的控制选项，允许发布商选择退出 AI 训练或 AI 数据锚定（Grounding）——即通过引导 AI 模型调用经过验证的信息源来减少幻觉的过程。微软同时指出，无论发布商的偏好如何设置，网站依然会照常出现在 Bing 的搜索结果中。Google 此前也推出了类似控制功能，允许网站自主控制其内容是否会出现在 AI 搜索功能中或用于 AI 训练，且同样不会影响其在传统搜索中的可见度。

目前 Anthropic 已经采纳了该信号，Google 和微软则计划于明年开始支持。OpenAI 表示将在“未来几个月内”予以落实。

更新于 10 月 30 日： 补充了来自 Meta 的评论。

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.

The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.

The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default.

OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Meta communications head Andy Stone also signaled support for the new setting, but doesn’t say whether the company will honor it. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone says. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”

Microsoft says it will offer granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences. Google previously implemented similar controls that allow sites to control whether their content appears in AI search features or is used for AI training, without impacting visibility in search.

While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”

Update, October 30th: Added a comment from Meta.