未删节文件披露：微软高管称 AI 抓取是“人类历史上规模最大的劳动窃取”

《纽约时报》三年前对 OpenAI 和微软提起的版权诉讼中，最新公开的未删节信息显示，AI 抓取被承认为近似于窃取，同时 AI 产品也会对出版物构成重大威胁。

根据诉讼文件，微软一名高管曾私下将两家公司的 AI 训练做法描述为“窃取”；OpenAI 自身的领导层则称，其 AI 模型对那些作品被用于训练模型的出版商和记者构成“生存性威胁”。

解封材料还详细说明，两家公司被指如何通过不被察觉地绕过付费墙、大规模抓取构建训练数据集，以及蓄意从训练数据中删除版权声明，来获取和使用这些内容。

值得注意的是，大量新增信息来自《纽约时报》自身提交的诉状，而非仍处于密封状态的底层证据附件。以下引文未附带其原始上下文。

这份未删节诉状是这场已持续三年的诉讼中的最新升级。《纽约时报》最初指控两家公司以其内容训练生成式 AI 模型，违反了版权法。

AI 公司能否合法使用受版权保护的材料训练 AI，目前尚无明确答案；但法官整体上较为倾向于接受 AI 公司提出的观点，即训练构成“合理使用”。这项法律规则允许人们在某些情形下未经许可使用受版权保护的作品，例如戏仿、新闻报道或批评。本月早些时候，特朗普政府提交了一份意见书，为 OpenAI 未经授权使用受版权保护材料训练其 LLM 的做法辩护。

然而，其中数项最新披露的承认与 OpenAI 的合理使用抗辩相冲突，尤其是合理使用规则中关于该使用不应替代原作品或损害原作品市场的要求。

例如，微软自身数据显示，与传统 Bing 搜索相比，其 Copilot“答案引擎”使《纽约时报》域名的点击率最多下降了 93%。微软应用科学总监 Brent Hecht 于 2024 年 1 月撰写的一份内部演示文稿，将这种下滑描述为一个“厄运循环”（“doom loop”），它将“同时损害我们模型和整个网络的表现”。

正如诉状所引述的微软文件所写：“最终产品威胁其关键供应商的经济基础，这种情况极不寻常；但就我们 LLM 业务的‘内容供应链’而言，这正是我们所造成的局面。”

微软 CEO 萨提亚·纳德拉（Satya Nadella）今年早些时候还在证词中表示，“任何设有付费墙的内容，都应由任何希望将其用于……检索增强（grounding）或训练的人获得授权”；他还明确表示，如果他“得知 OpenAI 曾抓取并训练使用付费墙之后的信息”，他会“行使[微软的权利]，要求 OpenAI 重新训练其模型”。

其他承认则冲击了合理使用测试的不同支柱：OpenAI 的 ChatGPT 负责人 Nick Turley 在内部沟通中写道，出版商面临来自聊天机器人等产品的“生存性威胁”，因为这些产品“在很大程度上具有替代性”，并且“随着其能力提升，将变得越来越具有替代性”。

OpenAI 总裁 Greg Brockman 将这些模型描述为“非常擅长新闻”。纳德拉今年早些时候在宣誓作证时也认同，与聊天机器人对话“已经替代了……你在 AI 平台网站上直接获得信息，而不是必须前往底层来源”。

这类表述涉及该技术如何可能直接与原作品竞争，而非对其进行转换性使用。

微软的一份文件称，生成式 AI 存在“真实风险”，即它可能“显著扰乱那些生成了基础模型训练数据的人群的就业”。

复制的规模之大尤为引人注目。文件首次披露，仅 OpenAI 的中期训练数据集就包含超过 91,692 份由《纽约时报》、《每日新闻》和调查报道中心发布的作品副本。一个源自 Common Crawl 的数据集仅从 nytimes.com 就收录了超过 200 万份文档。

在一份 2023 年 1 月的内部备忘录中，Hecht 将其称为“规模惊人、前所未有的窃取”，以及“人类历史上规模最大的劳动窃取”。

诉状进一步详细描述了 OpenAI 和微软被指如何获取原告的内容，其中包括从 Bing 索引中抓取内容。

“OpenAI 向微软交付了完整的 GPT-3 训练数据集，微软利用该数据集评估如何在自身商业产品中实施 OpenAI 的模型，”诉状写道。“微软同样通过名为 Project Taxi 和 Project Mango 的项目向 OpenAI 提供训练数据。”

两家公司被指将 Project Mango 数据汇编为一个训练数据集，其中包含新闻出版商至少 160,903 篇独特作品的副本。

为尽可能提高抓取收益，OpenAI 员工被指制定了在不被察觉的情况下规避付费墙的方案。文件显示，当 OpenAI 研究员 Nick Ryder 告知 Brockman 有一个“绕过 nytimes 付费墙的 hack”时，Brockman 回复道：“ah nice。”

OpenAI 员工还被指构建了 WebText 和 WebText2 等训练数据集，这些数据集不成比例地依赖抓取的新闻内容。他们还被指从 Common Crawl 获取数百万篇文章；Common Crawl 是一个免费、开放的网页爬取数据存储库。相关发现还描述了，在训练数据进入模型前，员工曾蓄意移除其中的版权声明，因为研究人员“不希望模型向用户输出”“版权声明”。

“首次在这里披露的证据表明，OpenAI 和微软知道他们的所作所为是错误的，”《纽约每日新闻》法律顾问 Steven Lieberman 在向 TechCrunch 分享的一份声明中表示。

OpenAI 和微软未回应置评请求。

New unredacted information in the copyright lawsuit The New York Times brought against OpenAI and Microsoft three years ago reveals an admission that AI scraping was tantamount to theft, and that AI products pose a major threat to publications.

Per the lawsuit, a top Microsoft executive privately described the companies’ AI training practices as “theft,” and OpenAI’s own leadership said its AI models posed an “existential threat” to the publishers and journalists whose work trained them.

The unsealed material also details how the companies allegedly obtained and used that content by bypassing paywalls undetected, building training datasets via mass scraping, and deliberately stripping copyright notices from training data.

It’s worth noting that much of the new information comes from The Times’ own brief, not the underlying exhibits, which remain sealed. The quotes below are presented without their original context.

The unredacted filing is the latest escalation in the three-year-old lawsuit, in which The New York Times initially alleged the firms violated copyright law by training generative AI models on its content.

The question of whether AI firms can legally use copyrighted material to train AI has no clear answer, but judges have been largely favorable to AI companies’ arguments that training constitutes “fair use.” This legal rule lets people use copyrighted work without permission in certain cases, like parody, news reporting, or criticism. Earlier this month, the Trump administration contributed a brief in defense of OpenAI’s unlicensed use of copyrighted material to train its LLMs.

Several of the new admissions, however, run counter to OpenAI’s fair use defense, particularly the rule’s requirement that use doesn’t substitute or harm the market for the original work.

For example, Microsoft’s own data shows its Copilot “answer engine” caused click-through rates for The New York Times’ domain to drop as much as 93% compared to traditional Bing search. An internal Microsoft presentation written by Microsoft’s director of Applied Science, Brent Hecht, in January 2024 describes the decline as a “doom loop” that would “hurt the performance of our models and the entire web at the same time.”

“It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its ‘content supply chain,’” reads the Microsoft document, as quoted in the filing.

Microsoft CEO Satya Nadella also testified in a deposition earlier this year that “anything that is paywalled should be licensed by anyone who wants to use it…for grounding or training,” and made clear that, if he “had been made aware that OpenAI had scraped and trained on information that was behind a paywall,” he would have “invoked [Microsoft’s right to] require OpenAI to retrain its models.”

Other admissions cut against different pillars of the fair-use test: OpenAI’s head of ChatGPT, Nick Turley, wrote in internal communication that publishers face an “existential threat” from products like the chatbot, which are “largely substitutive” and “will get more and more substitutive as they get better.”

OpenAI President Greg Brockman described the models as “excellent at news.” Nadella agreed under oath earlier this year that conversing with chatbots “has substituted … giving you the information right there on the website on the AI platform versus needing to go to the underlying source.”

That kind of language speaks to how the technology could directly compete with, rather than transform, the original work.

A Microsoft document states that there is a “real risk” that generative AI could “significantly disrupt the employment of the very people who generated the data on which the foundation model was trained.”

The sheer scale of the copying is striking. The documents reveal for the first time that OpenAI’s mid-training datasets alone contain more than 91,692 copies of works published by the NYT, Daily News, and Center for Investigative Reporting. A Common Crawl-derived dataset included more than 2 million documents from nytimes.com alone.

In a January 2023 internal memo, Hecht called it “an astonishing theft of unprecedented proportions” and “the largest theft of labor in human history.”

The filing lays out in new detail how OpenAI and Microsoft went about acquiring the plaintiffs’ content, including scraping it from the Bing Index.

“OpenAI delivered the entire GPT-3 training dataset to Microsoft, which Microsoft used to evaluate how to implement OpenAI’s models within its own commercial products,” the filing reads. “Microsoft similarly provided training data to OpenAI through initiatives called Project Taxi and Project Mango.”

The companies allegedly assembled the Project Mango data into a training dataset that contains copies of at least 160,903 unique works from the news publishers.

In order to get the most out of their scraping, OpenAI employees allegedly came up with a plan to circumvent paywalls without detection. The filings show that when OpenAI researcher Nick Ryder told Brockman about a “hack to get around nytimes paywall,” Brockman replied: “ah nice.”

OpenAI employees also allegedly built training datasets like WebText and WebText2 that disproportionately relied on scraped news content. They also allegedly pulled millions of articles from Common Crawl, a free, open repository of web crawl data. The findings also describe deliberate efforts to strip copyright notices from training data before it reached the model, since researchers “wouldn’t want model outputting” “copyright notices” to users.

“The evidence revealed here for the first time shows that OpenAI and Microsoft knew that what they were doing was wrong,” Steven Lieberman, counsel for the New York Daily News, said in a statement shared with TechCrunch.

OpenAI and Microsoft did not return requests for comment.