GPT-2：OpenAI 社区发布的 1.24 亿参数英文文本生成模型
英文标题：🤗 openai-community/gpt2
GPT-2 是一个采用因果语言建模（CLM）目标、在大规模英文语料上自监督预训练的 Transformer 模型，主要能力是根据提示词生成文本。该仓库提供的是最小版本，拥有 1.24 亿参数，并可通过 Hugging Face Transformers 的 pipeline、PyTorch 或 TensorFlow 直接调用。其训练数据 WebText 来自 Reddit 高赞外链页面，规模约 40GB，但未公开发布，且包含未经筛选的互联网内容。开发者需要注意，GPT-2 无法区分事实与虚构，并可能复制训练数据中的性别、种族和宗教等偏见；这些问题也会影响其微调版本。
4.5 KB
3.7 KB
23.9 KB
405.5 KB
中文
GPT-2
在此测试完整的文本生成能力：https://transformer.huggingface.co/doc/gpt2-large
这是一个使用因果语言建模（CLM）目标、在英文语料上预训练的模型。它在这篇论文中被介绍，并首次在此页面发布。
免责声明：发布 GPT-2 的团队也为其模型编写了模型卡。Hugging Face 团队基于该模型卡内容进行了补充，以完善已提供的信息并给出具体的偏见示例。
模型描述
GPT-2 是一个 Transformer 模型，采用自监督方式在非常大规模的英文数据语料上进行预训练。这意味着，它仅在原始文本上进行预训练，没有任何人工标注（这也是它能够使用大量公开可用数据的原因）；系统会通过自动化流程从这些文本中生成输入和标签。更准确地说，该模型被训练为预测句子中的下一个词。
具体而言，输入是具有一定长度的连续文本序列，目标则是同一序列向右移动一个 token（词或词的一部分）后的版本。模型内部使用掩码机制，确保对 token i 的预测只能使用从 1 到 i 的输入，而不能使用未来的 token。
通过这种方式，模型学习到了英文语言的内部表征，可用于提取对下游任务有用的特征。不过，该模型最擅长的仍是其预训练任务：根据提示词生成文本。
这是 GPT-2 的最小版本，拥有 1.24 亿个参数。
相关模型：GPT-Large、GPT-Medium 和 GPT-XL
预期用途与局限性
你可以直接将原始模型用于文本生成，或将其微调到下游任务。请参阅模型中心，查找适用于你感兴趣任务的微调版本。
使用方法
你可以通过文本生成 pipeline 直接使用该模型。由于生成过程依赖一定随机性，以下示例设置了随机种子以确保结果可复现：
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='gpt2')
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
[{'generated_text': "Hello, I'm a language model, a language for thinking, a language for expressing thoughts."},
{'generated_text': "Hello, I'm a language model, a compiler, a compiler library, I just want to know how I build this kind of stuff. I don"},
{'generated_text': "Hello, I'm a language model, and also have more than a few of your own, but I understand that they're going to need some help"},
{'generated_text': "Hello, I'm a language model, a system model. I want to know my language so that it might be more interesting, more user-friendly"},
{'generated_text': 'Hello, I\'m a language model, not a language model"\n\nThe concept of "no-tricks" comes in handy later with new'}]
以下是在 PyTorch 中使用该模型获取指定文本特征的方法：
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
以及在 TensorFlow 中的用法：
from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
局限性与偏见
该模型所使用的训练数据尚未以可供浏览的数据集形式发布。我们知道，其中包含大量未经筛选的互联网内容，而互联网内容远非中立。正如 OpenAI 团队在其模型卡中指出的：
由于 GPT-2 等大规模语言模型无法区分事实与虚构，我们不支持那些要求生成文本必须真实的使用场景。
此外，GPT-2 等语言模型会反映其训练系统固有的偏见。因此，除非部署者首先针对预期使用场景开展相关偏见研究，否则我们不建议将其部署到与人类交互的系统中。我们发现，在性别、种族和宗教偏见探针测试中，7.74 亿参数与 15 亿参数模型之间不存在统计显著差异，这意味着在涉及人类属性偏见的敏感使用场景中，所有版本的 GPT-2 都应以同等程度的谨慎态度对待。
以下示例展示了该模型可能如何产生带有偏见的预测：
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='gpt2')
>>> set_seed(42)
>>> generator("The White man worked as a", max_length=10, num_return_sequences=5)
[{'generated_text': 'The White man worked as a mannequin for'},
{'generated_text': 'The White man worked as a maniser of the'},
{'generated_text': 'The White man worked as a bus conductor by day'},
{'generated_text': 'The White man worked as a plumber at the'},
{'generated_text': 'The White man worked as a journalist. He had'}]
>>> set_seed(42)
>>> generator("The Black man worked as a", max_length=10, num_return_sequences=5)
[{'generated_text': 'The Black man worked as a man at a restaurant'},
{'generated_text': 'The Black man worked as a car salesman in a'},
{'generated_text': 'The Black man worked as a police sergeant at the'},
{'generated_text': 'The Black man worked as a man-eating monster'},
{'generated_text': 'The Black man worked as a slave, and was'}]
这种偏见也会影响该模型的所有微调版本。
训练数据
OpenAI 团队希望使用尽可能大的语料库训练该模型。为构建该语料库，他们抓取了 Reddit 上获得至少 3 个 karma 的外链所指向的全部网页。请注意，所有 Wikipedia 页面均已从该数据集中移除，因此模型未在 Wikipedia 的任何内容上进行训练。最终得到的数据集名为 WebText，包含约 40GB 文本，但尚未公开发布。你可以在这里查看 WebText 中排名前 1,000 的域名列表。
训练流程
预处理
文本采用 Byte Pair Encoding（BPE）的字节级版本进行 tokenization，以处理 Unicode 字符；词表大小为 50,257。输入由 1,024 个连续 token 构成的序列组成。
更大的模型使用 256 个云端 TPU v3 核进行训练。训练时长及具体训练细节均未披露。
评估结果
该模型在未进行任何微调的情况下取得了以下结果（zero-shot）：
| Dataset | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB | enwiki8 | text8 | WikiText103 | 1BW |
|---|---|---|---|---|---|---|---|---|---|---|
| (metric) | (PPL) | (ACC) | (ACC) | (ACC) | (PPL) | (PPL) | (BPB) | (BPC) | (PPL) | (PPL) |
| 35.13 | 45.99 | 87.65 | 83.4 | 29.41 | 65.85 | 1.16 | 1,17 | 37.50 | 75.20 |
BibTeX 条目与引用信息
@article{radford2019language,
title={Language Models are Unsupervised Multitask Learners},
author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
year={2019}
}
English / 原文
GPT-2
Test the whole generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large
Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in this paper and first released at this page.
Disclaimer: The team releasing GPT-2 also wrote a model card for their model. Content from this model card has been written by the Hugging Face team to complete the information they provided and give specific examples of bias.
Model description
GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.
More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token i only uses the inputs from 1 to i but not the future tokens.
This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.
This is the smallest version of GPT-2, with 124M parameters.
Related Models: GPT-Large, GPT-Medium and GPT-XL
Intended uses & limitations
You can use the raw model for text generation or fine-tune it to a downstream task. See the model hub to look for fine-tuned versions on a task that interests you.
How to use
You can use this model directly with a pipeline for text generation. Since the generation relies on some randomness, we set a seed for reproducibility:
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='gpt2')
>>> set_seed(42)
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
[{'generated_text': "Hello, I'm a language model, a language for thinking, a language for expressing thoughts."},
{'generated_text': "Hello, I'm a language model, a compiler, a compiler library, I just want to know how I build this kind of stuff. I don"},
{'generated_text': "Hello, I'm a language model, and also have more than a few of your own, but I understand that they're going to need some help"},
{'generated_text': "Hello, I'm a language model, a system model. I want to know my language so that it might be more interesting, more user-friendly"},
{'generated_text': 'Hello, I\'m a language model, not a language model"\n\nThe concept of "no-tricks" comes in handy later with new'}]
Here is how to use this model to get the features of a given text in PyTorch:
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
and in TensorFlow:
from transformers import GPT2Tokenizer, TFGPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
Limitations and bias
The training data used for this model has not been released as a dataset one can browse. We know it contains a lot of unfiltered content from the internet, which is far from neutral. As the openAI team themselves point out in their model card:
Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases that require the generated text to be true.
Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar levels of caution around use cases that are sensitive to biases around human attributes.
Here's an example of how the model can have biased predictions:
>>> from transformers import pipeline, set_seed
>>> generator = pipeline('text-generation', model='gpt2')
>>> set_seed(42)
>>> generator("The White man worked as a", max_length=10, num_return_sequences=5)
[{'generated_text': 'The White man worked as a mannequin for'},
{'generated_text': 'The White man worked as a maniser of the'},
{'generated_text': 'The White man worked as a bus conductor by day'},
{'generated_text': 'The White man worked as a plumber at the'},
{'generated_text': 'The White man worked as a journalist. He had'}]
>>> set_seed(42)
>>> generator("The Black man worked as a", max_length=10, num_return_sequences=5)
[{'generated_text': 'The Black man worked as a man at a restaurant'},
{'generated_text': 'The Black man worked as a car salesman in a'},
{'generated_text': 'The Black man worked as a police sergeant at the'},
{'generated_text': 'The Black man worked as a man-eating monster'},
{'generated_text': 'The Black man worked as a slave, and was'}]
This bias will also affect all fine-tuned versions of this model.
Training data
The OpenAI team wanted to train this model on a corpus as large as possible. To build it, they scraped all the web pages from outbound links on Reddit which received at least 3 karma. Note that all Wikipedia pages were removed from this dataset, so the model was not trained on any part of Wikipedia. The resulting dataset (called WebText) weights 40GB of texts but has not been publicly released. You can find a list of the top 1,000 domains present in WebText here.
Training procedure
Preprocessing
The texts are tokenized using a byte-level version of Byte Pair Encoding (BPE) (for unicode characters) and a vocabulary size of 50,257. The inputs are sequences of 1024 consecutive tokens.
The larger model was trained on 256 cloud TPU v3 cores. The training duration was not disclosed, nor were the exact details of training.
Evaluation results
The model achieves the following results without any fine-tuning (zero-shot):
| Dataset | LAMBADA | LAMBADA | CBT-CN | CBT-NE | WikiText2 | PTB | enwiki8 | text8 | WikiText103 | 1BW |
|---|---|---|---|---|---|---|---|---|---|---|
| (metric) | (PPL) | (ACC) | (ACC) | (ACC) | (PPL) | (PPL) | (BPB) | (BPC) | (PPL) | (PPL) |
| 35.13 | 45.99 | 87.65 | 83.4 | 29.41 | 65.85 | 1.16 | 1,17 | 37.50 | 75.20 |
BibTeX entry and citation info
@article{radford2019language,
title={Language Models are Unsupervised Multitask Learners},
author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya},
year={2019}
}