3 ms·

Are there any technologies today that would enable a distributed backup of IA? Something like BitTorrent, but where I can specify I want to host XXX GB of data,

Are there any technologies today that would enable a distributed backup of IA? Something like BitTorrent, but where I can specify I want to host XXX GB of data, and all the nodes in the swarm would provide a redundant, distributed backup in aggregate? I realize things like IPFS exist, but as far as I'm aware those require manual file pinning. I want to just donate storage and bandwidth to back the files up, not have to specify which ones to back up.

jmeyer2k 6y ago
IPFS works based on files visited. I wonder if you could write a script that crawls randomly until it hits a certain threshold stored...
- andrewmunsell 6y ago
  Just a cursory look at the CLI, it looks like you have the ability to get stats on each block (including size, https://docs.ipfs.io/reference/cli/#ipfs-block-stat https://docs.ipfs.io/reference/cli/#ipfs-block-stat), and pin them (https://docs.ipfs.io/how-to/pin-files/#three-kinds-of-pins https://docs.ipfs.io/how-to/pin-files/#three-kinds-of-pins). I'm betting someone could create an open source project to traverse the Merkel tree, choose random blocks, and pin them... It could be made smarter by also checking the number of peers per block and prioritize the least pinned blocks. Edit: And a related discussion I found: https://www.reddit.com/r/ipfs/comments/b4he2m/idea_partialselective_pinning/ https://www.reddit.com/r/ipfs/comments/b4he2m/idea_partialse...
- ashleysmithgpu 6y ago
  On a related note (Yes crypto, sorry) Filecoin is encouraging people to help backup certain large data sets: https://filecoin.io/blog/intro-filecoin-discover/ https://filecoin.io/blog/intro-filecoin-discover/
gpm 6y ago
If you're donating storage and bandwidth to backup "files" up without specifying which ones, you're going to be backing up random peoples personal encrypted backups which are useless to anyone who doesn't have the key. In the end you need a decision layer to decide what to back up that decides what based on some standard of what's good for society. But maybe you don't need to centralize the storage, I could imagine a distributed IA which mostly piggybacks on other people's storage.
- andrewmunsell 6y ago
  Specifically, I'm referring to files that are supposed to be public to begin with like the Internet Archive. The goal there is to replicate the data and make it publicly accessible. There'd probably have to be some index (can IPNS be used for this?) to specify the root, and then you could specify you want to store XXX GB of random data from this root node.
nope96 6y ago
"Let's Say You Wanted to Back Up The Internet Archive" post by Jason Scott https://old.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ https://old.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...

jmeyer2k 6y ago

IPFS works based on files visited. I wonder if you could write a script that crawls randomly until it hits a certain threshold stored...

andrewmunsell 6y ago
Just a cursory look at the CLI, it looks like you have the ability to get stats on each block (including size, https://docs.ipfs.io/reference/cli/#ipfs-block-stat https://docs.ipfs.io/reference/cli/#ipfs-block-stat), and pin them (https://docs.ipfs.io/how-to/pin-files/#three-kinds-of-pins https://docs.ipfs.io/how-to/pin-files/#three-kinds-of-pins). I'm betting someone could create an open source project to traverse the Merkel tree, choose random blocks, and pin them... It could be made smarter by also checking the number of peers per block and prioritize the least pinned blocks. Edit: And a related discussion I found: https://www.reddit.com/r/ipfs/comments/b4he2m/idea_partialselective_pinning/ https://www.reddit.com/r/ipfs/comments/b4he2m/idea_partialse...
ashleysmithgpu 6y ago
On a related note (Yes crypto, sorry) Filecoin is encouraging people to help backup certain large data sets: https://filecoin.io/blog/intro-filecoin-discover/ https://filecoin.io/blog/intro-filecoin-discover/

gpm 6y ago

If you're donating storage and bandwidth to backup "files" up without specifying which ones, you're going to be backing up random peoples personal encrypted backups which are useless to anyone who doesn't have the key. In the end you need a decision layer to decide what to back up that decides what based on some standard of what's good for society. But maybe you don't need to centralize the storage, I could imagine a distributed IA which mostly piggybacks on other people's storage.

andrewmunsell 6y ago
Specifically, I'm referring to files that are supposed to be public to begin with like the Internet Archive. The goal there is to replicate the data and make it publicly accessible. There'd probably have to be some index (can IPNS be used for this?) to specify the root, and then you could specify you want to store XXX GB of random data from this root node.

nope96 6y ago

"Let's Say You Wanted to Back Up The Internet Archive" post by Jason Scott https://old.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/ https://old.reddit.com/r/DataHoarder/comments/h02jl4/lets_sa...