100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of th

100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet. the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain. it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.