3 ms·
Let's assume that happens. How do I hedge against it? Is there a convenient way to mirror the bits of the web that are open now? Perhaps a mirror of archive/Way
by lostmsu 3y ago
Let's assume that happens. How do I hedge against it? Is there a convenient way to mirror the bits of the web that are open now? Perhaps a mirror of archive/WayBack machine that could be viewed locally similar to Wikipedia dumps?
- paulmd 3y agoYes, commoncrawl indexes
- lostmsu 3y ago100TB. Dang! I will need to expand my storage significantly to mirror it.
- paulmd 3y ago100TB so far ;) And that's probably just 1 year. There's data that's vanished over time. The point of all this is that they're point-in-time references of the internet. the more practical thing is that (the royal) they have set up a rclone like utility that allows you to retrieve a slice of the whole snapshot, if you know your domain. it would be extremely funni if commoncrawl was a de-facto ipfs, allowing queries along domains (or arbitrary dimensions/clustering) of interest.