3 ms·
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- tetete 3y agoMight be useful to those who have not been in the space but this is old news. 2020.
- tosh 3y agoI tried to find out how many "tokens" (I know: depends on the tokenizer) "The Pile" has but couldn't find it. As far as I understand RedPajama has 1.2T (https://github.com/togethercomputer/RedPajama-Data https://github.com/togethercomputer/RedPajama-Data) and has a table in the readme listing the main parts and how many tokens each part has.
- perone 3y agoI wrote an article about recent data pipelines as well if anyone is interested: https://blog.christianperone.com/2023/06/appreciating-llms-data-pipelines/ https://blog.christianperone.com/2023/06/appreciating-llms-d...
- startupsfail 3y agoArguably “The Pile” approach is the bane of open source and small player LLMs. Because the models are shaped by data. And there is little beauty in “The Pile”. — anonymous shaper
- physicsgraph 3y agoPrevious post: https://news.ycombinator.com/item?id=25607809 https://news.ycombinator.com/item?id=25607809