3 ms·

Yes, commoncrawl indexes

• lostmsu 3y ago
  100TB. Dang! I will need to expand my storage significantly to mirror it.
  • paulmd 3y ago
    100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet. the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain. it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.