3 ms·

I hope this doesn't bankrupt the Internet Archive (either the legal fees or the case - I don't recall what they're asking for). It would be bad if the Wayback M

I hope this doesn't bankrupt the Internet Archive (either the legal fees or the case - I don't recall what they're asking for). It would be bad if the Wayback Machine, the biggest internet archivist around, went under, and also all the books, software, et cetera that the Archive hosts. I wonder if there's any way to archive all of the Wayback Machine (82.3 petabytes), or, better yet, all of the Internet Archive (which is, by my count, around 120 petabytes?) Who would have the capability/interest in doing so, ideally without charge?

pcaharrier 2y ago
Shutting down IA altogether seems unlikely at this point (even if legal fees are substantial). They've written before about what the decision means (this appellate court affirmed what the district court did last year) and it doesn't touch things like the Wayback Machine: https://blog.archive.org/2023/08/17/what-the-hachette-v-internet-archive-decision-means-for-our-library/ https://blog.archive.org/2023/08/17/what-the-hachette-v-inte...
- leotravis10 2y ago
  Keep a close eye on the inevitable lawsuits that they're going to be up against. Obviously the music publishers are currently trying to kill the Great 78 Project and I expect other industries to pounce. (Especially the gaming and media industries). I can't see the Internet Archive exist in 2-5 years and we need to prepare for the worst case scenario. https://old.reddit.com/r/DataHoarder/comments/1bswhdj/if_there_is_a_book_on_internet_archive_your/ https://old.reddit.com/r/DataHoarder/comments/1bswhdj/if_the... Discussion: https://news.ycombinator.com/item?id=39908676 https://news.ycombinator.com/item?id=39908676 Projects such as Anna's Archive are going to be much more critical than ever.
  - mezzie2 2y ago
    > Projects such as Anna's Archive are going to be much more critical than ever. I look forward to all the new AA mirrors + contributions that are going to come from disgruntled archivists. And everybody in the libraries/archives space going on downloading sprees. Most of us have personal illegal libraries. We just don't share them.
stepupmakeup 2y ago
For wayback machine, are those compressed, deduplicated numbers? A semi-popular domain can have millions of results on their CDX api, but with https/https duplicated and about 90% of results are error pages or pages with deliberate garbage / LFI attempts in them.
- smallerize 2y ago
  Deduplication is not trivial. Each scrape is stored in a WARC archive, so you would have to unpack several large files, dedupe, and then pack them back up again. I believe they are at least compressed within each snapshot though.
  - account42 2y ago
    Yes, that seems to be a silly way to go about it if your goal is to store the whole web and not just a single scrape. Of course anything that deduplicates data is more vulnerable to data corruption (or at least corruption can have wider consequences) so it's not a trivial problem but you'd think deduplicating identical resources would be something added the first time they came close to their storage limits.
beretguy 2y ago
We need to archive the archive.

pcaharrier 2y ago

Shutting down IA altogether seems unlikely at this point (even if legal fees are substantial). They've written before about what the decision means (this appellate court affirmed what the district court did last year) and it doesn't touch things like the Wayback Machine: https://blog.archive.org/2023/08/17/what-the-hachette-v-internet-archive-decision-means-for-our-library/ https://blog.archive.org/2023/08/17/what-the-hachette-v-inte...

leotravis10 2y ago
Keep a close eye on the inevitable lawsuits that they're going to be up against. Obviously the music publishers are currently trying to kill the Great 78 Project and I expect other industries to pounce. (Especially the gaming and media industries). I can't see the Internet Archive exist in 2-5 years and we need to prepare for the worst case scenario. https://old.reddit.com/r/DataHoarder/comments/1bswhdj/if_there_is_a_book_on_internet_archive_your/ https://old.reddit.com/r/DataHoarder/comments/1bswhdj/if_the... Discussion: https://news.ycombinator.com/item?id=39908676 https://news.ycombinator.com/item?id=39908676 Projects such as Anna's Archive are going to be much more critical than ever.
- mezzie2 2y ago
  > Projects such as Anna's Archive are going to be much more critical than ever. I look forward to all the new AA mirrors + contributions that are going to come from disgruntled archivists. And everybody in the libraries/archives space going on downloading sprees. Most of us have personal illegal libraries. We just don't share them.

stepupmakeup 2y ago

For wayback machine, are those compressed, deduplicated numbers? A semi-popular domain can have millions of results on their CDX api, but with https/https duplicated and about 90% of results are error pages or pages with deliberate garbage / LFI attempts in them.

smallerize 2y ago
Deduplication is not trivial. Each scrape is stored in a WARC archive, so you would have to unpack several large files, dedupe, and then pack them back up again. I believe they are at least compressed within each snapshot though.
- account42 2y ago
  Yes, that seems to be a silly way to go about it if your goal is to store the whole web and not just a single scrape. Of course anything that deduplicates data is more vulnerable to data corruption (or at least corruption can have wider consequences) so it's not a trivial problem but you'd think deduplicating identical resources would be something added the first time they came close to their storage limits.

beretguy 2y ago

We need to archive the archive.