5 ms·

I've got a synology NAS with at least 10TB spare capacity I could donate... If only there were a way to use it, and thousands more, to back up important project

by sxates 6y ago

I've got a synology NAS with at least 10TB spare capacity I could donate... If only there were a way to use it, and thousands more, to back up important projects like this in a decentralized way.

Cthulhu_ 6y ago
I keep thinking that the Bittorrent protocol would be an ideal fit for this; specifically, it would need a way to have all peers agree on what chunks of a file they themselves host and share to ensure the most complete copies available. And it would need a system for keeping it updated. I wouldn't mind hosting a NAS in my closet and some bandwidth + storage to donate to the IA. I also think cloud providers - all of them - have the capacity to host backups of the IA; I'd prefer if companies like Amazon, Google would donate their storage and bandwidth capacity to the IA instead of just money.
- rbanffy 6y ago
  Bittorrent would require the totality of the archive to be preserved in multiple copies. I can't have a copy of that ledger here at home. We'd need a distributed highly redundant filesystem for that.
  - toomuchtodo 6y ago
    44 million items in the Internet Archive are available as torrents. It's pretty straightforward to enumerate them, download them, and serve them up in a swarm.
    - rbanffy 6y ago
      That doesn't make it sure the torrented data is always available. The index is important, but the data and its metadata are much more important.
      - toomuchtodo 6y ago
        Right, which is why you want to start getting the data served by seeders outside of the Internet Archive proper. The Pirate Bay is all magnet links. A model to be considered.
        rbanffy 6y ago
        That still doesn't ensure all the data is available all the time. A solution should be one where whoever hosts the data can't pick which pieces they want to host. Otherwise content that's not popular will disappear.
- nix23 6y ago
  git-annex is much better for that situation. https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... and http://iabak.archiveteam.org/ http://iabak.archiveteam.org/
boomlinde 6y ago
IPFS seems perfect for the purpose. Files are content-addressed and a request for a part of a file can resolve to any accessible peer that has it. A peer has it if they have pinned it or if they have accessed it themselves recently so that it is cached on their node. This way, users share the load for popular files, donating both bandwidth and storage. Torrents also seem like a perfect way to donate bandwidth and storage, and the IA already uses those. The only problem with these approaches or generally any distributed file store is that someone will have to share a copy, so you'll mostly be donating bandwidth by storing a redundant copy of data that they likely have to store themselves as well to guarantee accessibility.
jefurii 6y ago
I'd go out and buy a NAS and fill it up with backups from IA if it would help the collection survive. Point me at the project.

by sxates 6y ago

I've got a synology NAS with at least 10TB spare capacity I could donate... If only there were a way to use it, and thousands more, to back up important projects like this in a decentralized way.

Cthulhu_ 6y ago

I keep thinking that the Bittorrent protocol would be an ideal fit for this; specifically, it would need a way to have all peers agree on what chunks of a file they themselves host and share to ensure the most complete copies available. And it would need a system for keeping it updated. I wouldn't mind hosting a NAS in my closet and some bandwidth + storage to donate to the IA. I also think cloud providers - all of them - have the capacity to host backups of the IA; I'd prefer if companies like Amazon, Google would donate their storage and bandwidth capacity to the IA instead of just money.

rbanffy 6y ago
Bittorrent would require the totality of the archive to be preserved in multiple copies. I can't have a copy of that ledger here at home. We'd need a distributed highly redundant filesystem for that.
- toomuchtodo 6y ago
  44 million items in the Internet Archive are available as torrents. It's pretty straightforward to enumerate them, download them, and serve them up in a swarm.
  - rbanffy 6y ago
    That doesn't make it sure the torrented data is always available. The index is important, but the data and its metadata are much more important.
    - toomuchtodo 6y ago
      Right, which is why you want to start getting the data served by seeders outside of the Internet Archive proper. The Pirate Bay is all magnet links. A model to be considered.
      - rbanffy 6y ago
        That still doesn't ensure all the data is available all the time. A solution should be one where whoever hosts the data can't pick which pieces they want to host. Otherwise content that's not popular will disappear.
nix23 6y ago
git-annex is much better for that situation. https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... and http://iabak.archiveteam.org/ http://iabak.archiveteam.org/

boomlinde 6y ago

IPFS seems perfect for the purpose. Files are content-addressed and a request for a part of a file can resolve to any accessible peer that has it. A peer has it if they have pinned it or if they have accessed it themselves recently so that it is cached on their node. This way, users share the load for popular files, donating both bandwidth and storage. Torrents also seem like a perfect way to donate bandwidth and storage, and the IA already uses those. The only problem with these approaches or generally any distributed file store is that someone will have to share a copy, so you'll mostly be donating bandwidth by storing a redundant copy of data that they likely have to store themselves as well to guarantee accessibility.

jefurii 6y ago

I'd go out and buy a NAS and fill it up with backups from IA if it would help the collection survive. Point me at the project.