> "power" [comes from] the entire Internet Are you sure? See "the Pile": https://pile.eleuther.ai/ https://pile.eleuther.ai/ The paper cited there by contra

> "power" [comes from] the entire Internet Are you sure? See "the Pile": https://pile.eleuther.ai/ https://pile.eleuther.ai/ The paper cited there by contrast, argues for select training sets: Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present \textit{the Pile}: an 825 GiB English text corpus targeted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets -- both existing and newly constructed -- many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve significantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027