4 ms·
Am I the only one who is surprised how small the storage size is? I understand it’s a lot of text but to realize it easily fits on a few cheap HDDs raided toget
by alchemist1e9 4y ago
Am I the only one who is surprised how small the storage size is? I understand it’s a lot of text but to realize it easily fits on a few cheap HDDs raided together is still incredible to me.
- alwayslikethis 4y agoText is very compressible. It can probably fit in a single consumer HDD (8TB or so) compressed with zstd.
- alchemist1e9 4y agoIt would be neat of there could be defined curated chunks of standardize text data sets for LLMs training distributed over bittorrent. Maybe there already is and I just haven’t looked hard enough.
- kangalioo 4y ago> The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. https://pile.eleuther.ai/ https://pile.eleuther.ai/