2 ms·
I made sure to include that in my blog post - along with a note that you need 2.67TB of disk space first!
by simonw 4y ago
I made sure to include that in my blog post - along with a note that you need 2.67TB of disk space first!
- bhaney 4y ago> you need 2.67TB of disk space The data looks like it should compress pretty well. If you use something like btrfs's transparent compression, I wouldn't be surprised if it all fit in less than 0.75TB of disk space while still being usable to any tool that expects uncompressed data. Edit: It looks like some of this data is already compressed, so maybe not.
- csris 4y agoNote that you also need about 5TB of disk for the full decompressed dataset. However, only common crawl are compressed in jsonl.zst, everything else is uncompressed jsonl.
- deleted 4y ago[deleted]