4 ms·
It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resi
by mabbo 8y ago
It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency.
I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem.
If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth.
Hmm. This sounds like a fun project to try...
- DataWraith 8y agoThat sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive. [1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Edit: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ has more details.
- zrm 8y agoIf you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding. If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting. If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.