4 ms·
I wanted to write a piece of FOSS that would allow dissemination of a large dataset. It would basically work kind of like a torrent swarm, you'd have a "tracker
by stavros 3y ago
I wanted to write a piece of FOSS that would allow dissemination of a large dataset. It would basically work kind of like a torrent swarm, you'd have a "tracker" (the Internet Archive) deciding what it wants each person to store (usually the rarest content would get stored preferentially), and the user could say "I want to donate 2 TB of space to the Internet Archive" and would download whatever files the IA thought were most at risk of being lost.
This would have the added benefit that, if the IA went down, the public could reconstruct (some of) the dataset from this swarm.
I spoke to a few archiving organizations about whether they'd find it useful, but there wasn't much interest. Too bad, I think a lot of people would like to donate some disk space right about now.
- amanda99 3y agoYou could also do some kind of distributed proof that they still have it: ask every host that claims to have a file compute the hash(prefix+content) of the file, where prefixes differ: distributed k prefixes to each client, and then compare them to get a proof that at least some number of copies exist.
- SahAssar 3y agoMaybe I'm just being slow, but how is that different from a normal bittorrent (with DHT)? Are you asking for coordination of which parts are most rare? I though bittorrent already prioritized the least common parts in a swarm.
- stavros 3y agoIt has quite a few differences, namely it's not one swarm, each file (or collection) is its own torrent. This allows for easy updates (clients can get new files without having to re-download everything), it has protection against some attacks, etc.
- SahAssar 3y agoThat is supported in the bittorrent standard: https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0039.html https://www.bittorrent.org/beps/bep_0039.html Not sure what the client support looks like.
- stavros 3y agoThis is interesting, thank you!
- neets 3y agoWhatever you are describing I want to run a node
- stavros 3y agoYeah, but the Internet Archive would have to run the server for it, and I don't know if they want to :(
- remram 3y agoIPFS is meant for this, you could also use torrents with RSS (a lot of clients support downloading torrents from a feed, but there's no signaling for replacement/removal).
- stavros 3y agoIPFS doesn't do most of what I described, though.
- remram 3y agoIt doesn't track how much each file is being stored or pick what's most at risk. That seems particularly difficult, even impossible in the presence of bad actors. Unless you make a system where the clients don't know what they are storing nor who else does too. That would be novel I think.
- anacrolix 3y agoIt's an improvement over regular BitTorrent because it sounds like there's some coodination about how to spread the data across contributing nodes. BitTorrent doesn't do that out of the box.