3 ms·
Hi! I'm the VP of Engineering at Together. Thanks for writing up these instructions! FYI, you can also download all the files with one wget command: wget -i
by csris 3y ago
Hi! I'm the VP of Engineering at Together. Thanks for writing up these instructions! FYI, you can also download all the files with one wget command:
wget -i https://data.together.xyz/redpajama-data-1T/v1.0.0/urls.txt
This is also mentioned on the dataset card for redpajama-data-1T on Huggingface [1].
[1]: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T https://huggingface.co/datasets/togethercomputer/RedPajama-D...
- simonw 3y agoI made sure to include that in my blog post - along with a note that you need 2.67TB of disk space first!
- bhaney 3y ago> you need 2.67TB of disk space The data looks like it should compress pretty well. If you use something like btrfs's transparent compression, I wouldn't be surprised if it all fit in less than 0.75TB of disk space while still being usable to any tool that expects uncompressed data. Edit: It looks like some of this data is already compressed, so maybe not.
- csris 3y agoNote that you also need about 5TB of disk for the full decompressed dataset. However, only common crawl are compressed in jsonl.zst, everything else is uncompressed jsonl.
- deleted 3y ago[deleted]