2 ms·
Wrote this up as a blog post: https://simonwillison.net/2023/Apr/17/redpajama-data/ https://simonwillison.net/2023/Apr/17/redpajama-data/
by simonw 4y ago
Wrote this up as a blog post: https://simonwillison.net/2023/Apr/17/redpajama-data/ https://simonwillison.net/2023/Apr/17/redpajama-data/
- csris 4y agoHi! I'm the VP of Engineering at Together. Thanks for writing up these instructions! FYI, you can also download all the files with one wget command: wget -i https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt This is also mentioned on the dataset card for redpajama-data-1T on Huggingface [1]. [1]: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T https://huggingface.co/datasets/togethercomputer/RedPajama-D...
- simonw 4y agoI made sure to include that in my blog post - along with a note that you need 2.67TB of disk space first!
- bhaney 4y ago> you need 2.67TB of disk space The data looks like it should compress pretty well. If you use something like btrfs's transparent compression, I wouldn't be surprised if it all fit in less than 0.75TB of disk space while still being usable to any tool that expects uncompressed data. Edit: It looks like some of this data is already compressed, so maybe not.
- csris 4y agoNote that you also need about 5TB of disk for the full decompressed dataset. However, only common crawl are compressed in jsonl.zst, everything else is uncompressed jsonl.
- deleted 4y ago[deleted]