14 ms·

The Backing Up of the Internet Archive Continues

175 points by bane 11y ago

sneak 11y ago
This isn't a lot of data in the general scheme of things. Why doesn't AWS or Google offer to pop it in glacier/S3 or GCS for free for the PR? They both have a huge multiple of that much unused disk - it would cost them effectively nothing beyond inbound bandwidth.
- tracker1 11y ago
  It kind of is a lot of data... trying to archive internet websites whole and keeping multiple snapshots is pretty close to what google and other search engines do, only search engines need to generate distributed indexes as well as some information regarding versioning. Not to mention, that historically speaking, you cannot trust google to keep this information preserved, or public. Look at what happened to any number of other tools google once offered. TBH, I would like to see funding via a grant from the library of congress towards archive.org. I'm curious as to how many copies a piece of data is needed to be "safe" in such a flexible unknown as end user/volunteer storage. It's one thing for compute items that can be re-queued for work in a day or two if abandoned... it's another where every copy of a record happens to walk away. Let alone the communications protocol.. this goes way beyond most bigtable implementations.
  - manigandham 11y ago
    On a purely numerical comparison, the article mentions 27TB which really isn't much in terms of size, especially compared to what some of the companies using AWS produce daily. EDIT: according to comments below, looks like its about 20+ petabytes which is actually a fairly large amount.
    - vidarh 11y ago
      The article mentions 27TB so far at a point where they appear to still be focused on making their tools better. 27TB is a tiny proportion of IA.
      - philh 11y ago
        Cite: https://en.wikipedia.org/wiki/Internet_Archive https://en.wikipedia.org/wiki/Internet_Archive > As of October 2012, its collection topped 10 petabytes. I'd be curious what it's grown to since, but google didn't immediately tell me.
        Mithrandir 11y ago
        About 21-22 petabytes: http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
    - sneak 11y ago
      Not for Google or S3.
      - res0nat0r 11y ago
        Seeing as how the out of the door price for 12PB of data hosted on RRD S3 storage @ 99.999999999% durability is roughly $300k/month, I'd hardly call it trivial.
        sneak 11y ago
        A few things: 1) They have it sitting unused presently. The nominal cost of providing it would be zero. 2) RRS is fine. So would Glacier be. This reduces costs further. 3) There is significant PR benefit to such a move.
        tracker1 11y ago
        Google, and Amazon could very well run within the archive.org network once it is up... and could very well offload a significant amount of data. The nominal cost is anything but near zero... just on the wear of hard drives alone, it will be costly. My point was you can't rely on them to keep said data available. Not that they couldn't participate.. but I wouldn't trust anything less than a "Lifetime" (of the company) or a 25 year minimum commitment as anything but transient.
        res0nat0r 11y ago
        I'm not sure how you've figured they have 12PB of disk (Or possibly more like 36PB due to replication for 11 9s of durability) just lying around. The whole meme that ec2 came about due to it just being extra capacity is incorrect. Same goes for all of the other services. Running a business whose MO is to keep lowering costs and make a profit on razor thin margins doesn't lend itself to lots of unused infrastructure. They aren't going to sink costs into infrastructure without realizing a return as soon as possible. Server amortization costs, future cost calculation planning, depreciation costs, power consumption, etc are all closely calculated and factored into budgets. Just thinking they can support that much data for free and at no cost, or minimal cost because it feels good is naive.
  - deleted 11y ago
    [deleted]
    - deleted 11y ago
      [deleted]
      - deleted 11y ago
        [deleted]
  - yohui 11y ago
    > Not to mention, that historically speaking, you cannot trust google to keep this information preserved, or public. Look at what happened to any number of other tools google once offered. Google is hardly going to abandon cloud storage anytime soon. Google's willingness to release and kill small projects has no rational impact on the reliability of their core services. Historically speaking, Google helped preserve Usenet archives via Google Groups and digitized library archives via Google Books. Both projects have seen their share of ups and downs, but they're still publicly available.
    - leni536 11y ago
      > Google is hardly going to abandon cloud storage anytime soon. I wonder what "soon" means in this context. If I wanted to archive anything I would think in the timescale of decades, and that's only for my personal photos and videos.
    - DanBC 11y ago
      It's odd that you'd use the deja news archive. They way Google handled that aas pretty terrible. Arguably moving away from the Google Groups search page and into the general search page but using the site:groups.google.com term is better but now it's really hard to find stuff and it's really hard to search by different parameters.
      - yohui 11y ago
        Yes, and Google Books was hampered by backlash from authors. That's why I noted both projects had a history of ups and downs. However, both are still going. The point is that, if we're talking about Google's history, they have shown an interest in preserving data. And in regards to the Internet Archive, they'd only be serving as a backup storage provider, not in a frontend role. This is not to say that Google (or Amazon or Microsoft) should be the only backup provider for the Internet Archive, for example, but it would hardly be a bad thing, as tracker1 suggested, if they cared to donate the resources.
- 4ad 11y ago
  There's a huge amount of copyrighted material on the Internet Archive. They can get away with it, being a library and all, but I don't think anyone like Google or Amazon would want this liability. Plus it works the other way around too. I don't think the Internet Archive would want the Amazon liability. Amazon might delete to delete all or some of the data someday for whatever reason.
  - aw3c2 11y ago
    They could just act as dumb host behind the scenes. The hosting provider is not liable for its customers.
  - sneak 11y ago
    Yeah, I'm sure Google holding copies of vast archives of other peoples' published copyrighted data is something they want to avoid. Oh, wait.
- vidarh 11y ago
  That would seem to lose at least some of the benefits of creating tools and getting people involved to create properly distributed backup not just in terms of physically distributing the copies but making it independent of any single entity.
  - yohui 11y ago
    It wouldn't be bad to have another option, though. The volunteer effort currently covers only a fraction of the total Internet Archive. Better a single backup than no backup at all.
- gojomo 11y ago
  It'd be great if they would, as an extra copy! However, reliance on charity from any single commercial entity would be somewhat against the spirit of the project. Companies (and their PR budgets) come and go. Archive.org has about 22 petabytes of archived material – and growing. Even at AWS Glacier's cold-storage rate of $0.01/GiB per month, that's $220,000/month, $2.64MM a year.
  - sevenadrian 11y ago
    yeah, definitely not a trivial amount.
- userbinator 11y ago
  I wonder if anyone else also thought of Backblaze's "unlimited" storage for $5/month...
andyjohnson0 11y ago
I'm curious about how archive.org feel about this. Presumably they are paying for the outgoing bandwidth.
- zcore 11y ago
  This is being done by Jason Scott and the rest of the team that runs archive.org. Presumably they're ok with using their own bandwidth!
  - textfiles 11y ago
    It's being overseen by Jason Scott (me) but I am not utilizing other employees of Internet Archive - they're quite busy enough as it is.
- edward 11y ago
  The link into 300 Funston Ave is 10Gbit. There is no shortage of bandwidth.
  - textfiles 11y ago
    Your intel is old, edward ;) they're past 40Gbit now.
    - userbinator 11y ago
      Even saturating a 40Gbit link at 5GB/s would require nearly 50 days to transfer 20PB. Realistically, I'd estimate mirroring the IA to take years... providing you can keep up with its own growth.
- joepie91_ 11y ago
  Don't forget that the purpose of the Internet Archive is primarily to be an open archive. In that sense, I'd imagine that bandwidth use is not really a factor, and anything that contributes to the longevity of the archive is welcome. "It's costing us bandwidth" is only a relevant factor if your goal is to turn a profit, and the bandwidth use isn't contributing to that. An archive doesn't have the same considerations.
unicornporn 11y ago
Perhaps approaching this with LOCKSS (Lots Of Copies Keep Stuff Safe)[1] or a LOCKSS-like system would be a good idea. [1] http://www.lockss.org http://www.lockss.org
rtpg 11y ago
what exactly is the objective here? just making sure there's multiple copies of the internet archive? Nifty stuff in any case
- textfiles 11y ago
  There's multiple bits of objectives here - education, research, practice, and awareness. As time goes on, it's obvious that distributed sharing of data is the only real vaccine against the inherent problems of the modern Internet, and learning more about different ways this might be done benefit all. Plus it's nice to have some of this data scattered about the world.
  - mdaniel 11y ago
    In the sphere of research & practice, are there any long-term plans for distributed processing applied to the data, akin to the Common Crawl in AWS allowing one to run map-reduce jobs against it? There is a long history of sandboxing the JVM, which means in theory it should be safe to bring the code to the data without running the risk of having your local machine p0wned.
textfiles 11y ago
Jason Scott here. Just wanted to address the questions that always come up when this project gets some attention. (Also: Come volunteer to be a client! The more the merrier.) * We are only backing up public facing data. (Roughly 12pb) * We are only backing up curated sets of data. (So less than that.) * We are stepping carefully to learn more about the whole process as we go, documenting, etc. * The hope is this will produce some real-world lessons and code that other sites can use. * This project uses non Internet Archive infrastructure, and is not an Internet Archive project. It's going well, and the more people who join up, the better. Oh, and support the Internet Archive with a donation - it's a meaningful non-profit making a real difference in the world. http://archive.org/donate http://archive.org/donate
- meesterdude 11y ago
  Are you guys concerned with decay at all? How do you know the file stored 2 years ago is still the same file? what happens if its become corrupted?
  - textfiles 11y ago
    Currently, the system requires you to check in (using the git-annex facility) on a regular basis. If you don't check in (it's a script run by cron) within 2 weeks, you're considered decayed, and after 30 days, your contributions fall back into the pool and are taken in elsewhere.
    - meesterdude 11y ago
      I'm talking more about the checksums of files over time, and error correction should they deviate. I recently read about facebooks cold storage and it got me thinking about it (https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...) though for them they just want one copy total.
      - textfiles 11y ago
IanCal 11y ago
I had a crack at running this, and it was quite interesting. I'm certainly up for having some of my spare space for such a worthy cause. I stopped, however, mostly due to the speed. I have an 80Mb connection but couldn't pull more than ~0.5-1Mb down. At that rate, even filling 500GB would take about two months. Perhaps this was due to the size of the files (downloading music). For anyone looking to try it out, please do, and see what you get. One thing to be aware of is it asked for the amount of space to leave free not how much to use, and got the size of my disk wrong by about a factor of 10 so I had to be a bit cautious about what I selected.
- Filligree 11y ago
  > One thing to be aware of is it asked for the amount of space to leave free not how much to use, and got the size of my disk wrong by about a factor of 10 so I had to be a bit cautious about what I selected. Well, that's a deal-breaker for me. I can easily dedicate, say, a terabyte or two to the project, but I can't do "Free space minus this number". I need predictability.
  - sp332 11y ago
    Isn't it more predictable to know that you'll have at least (say) 500MB free, instead of gradually filling up your drive until one day you don't have any more space? Edit: to be clear, it will actually remove backup data from the drive over time to maintain the amount of free space you set. It updates the central server to tell it which files you still have.
    - Filligree 11y ago
      I have a 24TB disk array, configured as RAIDZ2; 12TB of usable space. When I run out of space, I add more disks. (Or, more usually, replace old ones.) I use Prometheus to keep track of its state, and have an email alert set up when disk space drops below a certain point. The problem with a program that tries to maintain a certain amount of free space is that I won't know how much space there really is. I'd need to subtract the amount the archive uses, then.. add back in the amount I actually want it to use.. and it all ends up as a bit of a mess. It'd be much easier for me if I could just set it to use 2TB, and be done with it. I could probably rely on it actually using the space.
userbinator 11y ago
One of the comments there brought up an interesting point: Is this pure archival, meaning a onetime download and no uploads? Imagine a distributed, peer-to-peer style Internet Archive. That would be awesome. It reminds me of a decade ago, when the rapid rise of P2P file sharing (mostly torrents) made it possible to find literally anything that existed and someone felt neighbourly enough to share. Multimedia, software, books, anything that was present in digital form. It was probably the closest we've ever been to a "global library", but too bad antipiracy groups/commercial interests/security focus mostly killed it off and replaced it with locked-down content silos...
- icebraining 11y ago
  Since it uses git-annex, it actually can fetch files in a P2P fashion, using Bittorrent: http://git-annex.branchable.com/special_remotes/bittorrent/ http://git-annex.branchable.com/special_remotes/bittorrent/
- bane 11y ago
  One thing to note, IA distributes many of their items via Torrent (as a download option). I've kind of found it interesting TPB doesn't mirror all of them.
jerf 11y ago
"We’ve intentionally and unintentionally punched clients in the gut" Oh come now, you can't drop that in a blog post without elaboration. :) Share your pain!
s0me0ne 11y ago
The Internet Archive will still archive your site even if you use their recommended robots.txt suggestions to have them not archive your site. I had several domains where I setup the robots.txt immediately as one of the first things on my sites. While I owned the sites and ran them with the robots.txt Archive.org would not show any archive of them. However once the sites went down/changed domains, etc, the archive of them is now on archive.org. So all they do is block you from seeing what is on the site as of now, not forever. So you'd have to own the domain infinitely to have it not archived by them.

175 points by bane 11y ago

sneak 11y ago

This isn't a lot of data in the general scheme of things. Why doesn't AWS or Google offer to pop it in glacier/S3 or GCS for free for the PR? They both have a huge multiple of that much unused disk - it would cost them effectively nothing beyond inbound bandwidth.

tracker1 11y ago
It kind of is a lot of data... trying to archive internet websites whole and keeping multiple snapshots is pretty close to what google and other search engines do, only search engines need to generate distributed indexes as well as some information regarding versioning. Not to mention, that historically speaking, you cannot trust google to keep this information preserved, or public. Look at what happened to any number of other tools google once offered. TBH, I would like to see funding via a grant from the library of congress towards archive.org. I'm curious as to how many copies a piece of data is needed to be "safe" in such a flexible unknown as end user/volunteer storage. It's one thing for compute items that can be re-queued for work in a day or two if abandoned... it's another where every copy of a record happens to walk away. Let alone the communications protocol.. this goes way beyond most bigtable implementations.
- manigandham 11y ago
  On a purely numerical comparison, the article mentions 27TB which really isn't much in terms of size, especially compared to what some of the companies using AWS produce daily. EDIT: according to comments below, looks like its about 20+ petabytes which is actually a fairly large amount.
  - vidarh 11y ago
    The article mentions 27TB so far at a point where they appear to still be focused on making their tools better. 27TB is a tiny proportion of IA.
    - philh 11y ago
      Cite: https://en.wikipedia.org/wiki/Internet_Archive https://en.wikipedia.org/wiki/Internet_Archive > As of October 2012, its collection topped 10 petabytes. I'd be curious what it's grown to since, but google didn't immediately tell me.
      - Mithrandir 11y ago
        About 21-22 petabytes: http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK
  - sneak 11y ago
    Not for Google or S3.
    - res0nat0r 11y ago
      Seeing as how the out of the door price for 12PB of data hosted on RRD S3 storage @ 99.999999999% durability is roughly $300k/month, I'd hardly call it trivial.
      - sneak 11y ago
        A few things: 1) They have it sitting unused presently. The nominal cost of providing it would be zero. 2) RRS is fine. So would Glacier be. This reduces costs further. 3) There is significant PR benefit to such a move.
        tracker1 11y ago
        Google, and Amazon could very well run within the archive.org network once it is up... and could very well offload a significant amount of data. The nominal cost is anything but near zero... just on the wear of hard drives alone, it will be costly. My point was you can't rely on them to keep said data available. Not that they couldn't participate.. but I wouldn't trust anything less than a "Lifetime" (of the company) or a 25 year minimum commitment as anything but transient.
        res0nat0r 11y ago
        I'm not sure how you've figured they have 12PB of disk (Or possibly more like 36PB due to replication for 11 9s of durability) just lying around. The whole meme that ec2 came about due to it just being extra capacity is incorrect. Same goes for all of the other services. Running a business whose MO is to keep lowering costs and make a profit on razor thin margins doesn't lend itself to lots of unused infrastructure. They aren't going to sink costs into infrastructure without realizing a return as soon as possible. Server amortization costs, future cost calculation planning, depreciation costs, power consumption, etc are all closely calculated and factored into budgets. Just thinking they can support that much data for free and at no cost, or minimal cost because it feels good is naive.
- deleted 11y ago
  [deleted]
  - deleted 11y ago
    [deleted]
    - deleted 11y ago
      [deleted]
- yohui 11y ago
  > Not to mention, that historically speaking, you cannot trust google to keep this information preserved, or public. Look at what happened to any number of other tools google once offered. Google is hardly going to abandon cloud storage anytime soon. Google's willingness to release and kill small projects has no rational impact on the reliability of their core services. Historically speaking, Google helped preserve Usenet archives via Google Groups and digitized library archives via Google Books. Both projects have seen their share of ups and downs, but they're still publicly available.
  - leni536 11y ago
    > Google is hardly going to abandon cloud storage anytime soon. I wonder what "soon" means in this context. If I wanted to archive anything I would think in the timescale of decades, and that's only for my personal photos and videos.
  - DanBC 11y ago
    It's odd that you'd use the deja news archive. They way Google handled that aas pretty terrible. Arguably moving away from the Google Groups search page and into the general search page but using the site:groups.google.com term is better but now it's really hard to find stuff and it's really hard to search by different parameters.
    - yohui 11y ago
      Yes, and Google Books was hampered by backlash from authors. That's why I noted both projects had a history of ups and downs. However, both are still going. The point is that, if we're talking about Google's history, they have shown an interest in preserving data. And in regards to the Internet Archive, they'd only be serving as a backup storage provider, not in a frontend role. This is not to say that Google (or Amazon or Microsoft) should be the only backup provider for the Internet Archive, for example, but it would hardly be a bad thing, as tracker1 suggested, if they cared to donate the resources.
4ad 11y ago
There's a huge amount of copyrighted material on the Internet Archive. They can get away with it, being a library and all, but I don't think anyone like Google or Amazon would want this liability. Plus it works the other way around too. I don't think the Internet Archive would want the Amazon liability. Amazon might delete to delete all or some of the data someday for whatever reason.
- aw3c2 11y ago
  They could just act as dumb host behind the scenes. The hosting provider is not liable for its customers.
- sneak 11y ago
  Yeah, I'm sure Google holding copies of vast archives of other peoples' published copyrighted data is something they want to avoid. Oh, wait.
vidarh 11y ago
That would seem to lose at least some of the benefits of creating tools and getting people involved to create properly distributed backup not just in terms of physically distributing the copies but making it independent of any single entity.
- yohui 11y ago
  It wouldn't be bad to have another option, though. The volunteer effort currently covers only a fraction of the total Internet Archive. Better a single backup than no backup at all.
gojomo 11y ago
It'd be great if they would, as an extra copy! However, reliance on charity from any single commercial entity would be somewhat against the spirit of the project. Companies (and their PR budgets) come and go. Archive.org has about 22 petabytes of archived material – and growing. Even at AWS Glacier's cold-storage rate of $0.01/GiB per month, that's $220,000/month, $2.64MM a year.
- sevenadrian 11y ago
  yeah, definitely not a trivial amount.
userbinator 11y ago
I wonder if anyone else also thought of Backblaze's "unlimited" storage for $5/month...

andyjohnson0 11y ago

I'm curious about how archive.org feel about this. Presumably they are paying for the outgoing bandwidth.

zcore 11y ago
This is being done by Jason Scott and the rest of the team that runs archive.org. Presumably they're ok with using their own bandwidth!
- textfiles 11y ago
  It's being overseen by Jason Scott (me) but I am not utilizing other employees of Internet Archive - they're quite busy enough as it is.
edward 11y ago
The link into 300 Funston Ave is 10Gbit. There is no shortage of bandwidth.
- textfiles 11y ago
  Your intel is old, edward ;) they're past 40Gbit now.
  - userbinator 11y ago
    Even saturating a 40Gbit link at 5GB/s would require nearly 50 days to transfer 20PB. Realistically, I'd estimate mirroring the IA to take years... providing you can keep up with its own growth.
joepie91_ 11y ago
Don't forget that the purpose of the Internet Archive is primarily to be an open archive. In that sense, I'd imagine that bandwidth use is not really a factor, and anything that contributes to the longevity of the archive is welcome. "It's costing us bandwidth" is only a relevant factor if your goal is to turn a profit, and the bandwidth use isn't contributing to that. An archive doesn't have the same considerations.

unicornporn 11y ago

Perhaps approaching this with LOCKSS (Lots Of Copies Keep Stuff Safe)[1] or a LOCKSS-like system would be a good idea. [1] http://www.lockss.org http://www.lockss.org

rtpg 11y ago

what exactly is the objective here? just making sure there's multiple copies of the internet archive? Nifty stuff in any case

textfiles 11y ago
There's multiple bits of objectives here - education, research, practice, and awareness. As time goes on, it's obvious that distributed sharing of data is the only real vaccine against the inherent problems of the modern Internet, and learning more about different ways this might be done benefit all. Plus it's nice to have some of this data scattered about the world.
- mdaniel 11y ago
  In the sphere of research & practice, are there any long-term plans for distributed processing applied to the data, akin to the Common Crawl in AWS allowing one to run map-reduce jobs against it? There is a long history of sandboxing the JVM, which means in theory it should be safe to bring the code to the data without running the risk of having your local machine p0wned.

textfiles 11y ago

Jason Scott here. Just wanted to address the questions that always come up when this project gets some attention. (Also: Come volunteer to be a client! The more the merrier.) * We are only backing up public facing data. (Roughly 12pb) * We are only backing up curated sets of data. (So less than that.) * We are stepping carefully to learn more about the whole process as we go, documenting, etc. * The hope is this will produce some real-world lessons and code that other sites can use. * This project uses non Internet Archive infrastructure, and is not an Internet Archive project. It's going well, and the more people who join up, the better. Oh, and support the Internet Archive with a donation - it's a meaningful non-profit making a real difference in the world. http://archive.org/donate http://archive.org/donate

meesterdude 11y ago
Are you guys concerned with decay at all? How do you know the file stored 2 years ago is still the same file? what happens if its become corrupted?
- textfiles 11y ago
  Currently, the system requires you to check in (using the git-annex facility) on a regular basis. If you don't check in (it's a script run by cron) within 2 weeks, you're considered decayed, and after 30 days, your contributions fall back into the pool and are taken in elsewhere.
  - meesterdude 11y ago
    I'm talking more about the checksums of files over time, and error correction should they deviate. I recently read about facebooks cold storage and it got me thinking about it (https://code.facebook.com/posts/1433093613662262/-under-the-hood-facebook-s-cold-storage-system-/ https://code.facebook.com/posts/1433093613662262/-under-the-...) though for them they just want one copy total.
    - textfiles 11y ago

IanCal 11y ago

I had a crack at running this, and it was quite interesting. I'm certainly up for having some of my spare space for such a worthy cause. I stopped, however, mostly due to the speed. I have an 80Mb connection but couldn't pull more than ~0.5-1Mb down. At that rate, even filling 500GB would take about two months. Perhaps this was due to the size of the files (downloading music). For anyone looking to try it out, please do, and see what you get. One thing to be aware of is it asked for the amount of space to leave free not how much to use, and got the size of my disk wrong by about a factor of 10 so I had to be a bit cautious about what I selected.

Filligree 11y ago
> One thing to be aware of is it asked for the amount of space to leave free not how much to use, and got the size of my disk wrong by about a factor of 10 so I had to be a bit cautious about what I selected. Well, that's a deal-breaker for me. I can easily dedicate, say, a terabyte or two to the project, but I can't do "Free space minus this number". I need predictability.
- sp332 11y ago
  Isn't it more predictable to know that you'll have at least (say) 500MB free, instead of gradually filling up your drive until one day you don't have any more space? Edit: to be clear, it will actually remove backup data from the drive over time to maintain the amount of free space you set. It updates the central server to tell it which files you still have.
  - Filligree 11y ago
    I have a 24TB disk array, configured as RAIDZ2; 12TB of usable space. When I run out of space, I add more disks. (Or, more usually, replace old ones.) I use Prometheus to keep track of its state, and have an email alert set up when disk space drops below a certain point. The problem with a program that tries to maintain a certain amount of free space is that I won't know how much space there really is. I'd need to subtract the amount the archive uses, then.. add back in the amount I actually want it to use.. and it all ends up as a bit of a mess. It'd be much easier for me if I could just set it to use 2TB, and be done with it. I could probably rely on it actually using the space.

userbinator 11y ago

One of the comments there brought up an interesting point: Is this pure archival, meaning a onetime download and no uploads? Imagine a distributed, peer-to-peer style Internet Archive. That would be awesome. It reminds me of a decade ago, when the rapid rise of P2P file sharing (mostly torrents) made it possible to find literally anything that existed and someone felt neighbourly enough to share. Multimedia, software, books, anything that was present in digital form. It was probably the closest we've ever been to a "global library", but too bad antipiracy groups/commercial interests/security focus mostly killed it off and replaced it with locked-down content silos...

icebraining 11y ago
Since it uses git-annex, it actually can fetch files in a P2P fashion, using Bittorrent: http://git-annex.branchable.com/special_remotes/bittorrent/ http://git-annex.branchable.com/special_remotes/bittorrent/
bane 11y ago
One thing to note, IA distributes many of their items via Torrent (as a download option). I've kind of found it interesting TPB doesn't mirror all of them.

jerf 11y ago

"We’ve intentionally and unintentionally punched clients in the gut" Oh come now, you can't drop that in a blog post without elaboration. :) Share your pain!

s0me0ne 11y ago

The Internet Archive will still archive your site even if you use their recommended robots.txt suggestions to have them not archive your site. I had several domains where I setup the robots.txt immediately as one of the first things on my sites. While I owned the sites and ran them with the robots.txt Archive.org would not show any archive of them. However once the sites went down/changed domains, etc, the archive of them is now on archive.org. So all they do is block you from seeing what is on the site as of now, not forever. So you'd have to own the domain infinitely to have it not archived by them.