34 ms·
Text is very compressible. It can probably fit in a single consumer HDD (8TB or so) compressed with zstd.
by alwayslikethis 4y ago
Text is very compressible. It can probably fit in a single consumer HDD (8TB or so) compressed with zstd.
- alchemist1e9 4y agoIt would be neat of there could be defined curated chunks of standardize text data sets for LLMs training distributed over bittorrent. Maybe there already is and I just haven’t looked hard enough.
- kangalioo 4y ago> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. https://pile.eleuther.ai/ https://pile.eleuther.ai/