3 ms·
I keep thinking that the Bittorrent protocol would be an ideal fit for this; specifically, it would need a way to have all peers agree on what chunks of a file
by Cthulhu_ 6y ago
I keep thinking that the Bittorrent protocol would be an ideal fit for this; specifically, it would need a way to have all peers agree on what chunks of a file they themselves host and share to ensure the most complete copies available. And it would need a system for keeping it updated.
I wouldn't mind hosting a NAS in my closet and some bandwidth + storage to donate to the IA.
I also think cloud providers - all of them - have the capacity to host backups of the IA; I'd prefer if companies like Amazon, Google would donate their storage and bandwidth capacity to the IA instead of just money.
- rbanffy 6y agoBittorrent would require the totality of the archive to be preserved in multiple copies. I can't have a copy of that ledger here at home. We'd need a distributed highly redundant filesystem for that.
- toomuchtodo 6y ago44 million items in the Internet Archive are available as torrents. It's pretty straightforward to enumerate them, download them, and serve them up in a swarm.
- rbanffy 6y agoThat doesn't make it sure the torrented data is always available. The index is important, but the data and its metadata are much more important.
- toomuchtodo 6y agoRight, which is why you want to start getting the data served by seeders outside of the Internet Archive proper. The Pirate Bay is all magnet links. A model to be considered.
- rbanffy 6y agoThat still doesn't ensure all the data is available all the time. A solution should be one where whoever hosts the data can't pick which pieces they want to host. Otherwise content that's not popular will disappear.
- nix23 6y agogit-annex is much better for that situation. https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... and http://iabak.archiveteam.org/ http://iabak.archiveteam.org/