6 ms·

What would the Millennium Clock[0] version of a data storage device look like? [0]http://longnow.org/essays/millennium-clock/ http://longnow.org/essays/millenn

What would the Millennium Clock[0] version of a data storage device look like? [0]http://longnow.org/essays/millennium-clock/ http://longnow.org/essays/millennium-clock/

mabbo 8y ago
It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency. I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem. If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth. Hmm. This sounds like a fun project to try...
- DataWraith 8y ago
  That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive. [1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Edit: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ has more details.
- zrm 8y ago
  If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding. If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting. If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.
  - mabbo 8y ago
    And here I go on a wiki spiral. Thanks!
  - voltagex_ 8y ago
    Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.
roywiggins 8y ago
From the same people: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ It doesn't really scale, but then neither does the Clock.
dekhn 8y ago
It would be a collection of lithographically etched plates. These are stable across a wide collection of conditions. The coding scheme would be reed solomon or a variant of it, or possible even just massive duplication with checksums. Multiple duplicate backups of the entire data would be stored in geographically disparate locations. This reduces the "asteroid hits <city>" problem. A wide range of different technologies would be used, randomly, across the copies. This reduces the likelihood of systematic catastrophic failure due to a specific technology not working as expected. This is not that different from how current millenia-scale data storage has worked for... well, millenia. The primary challenge is in finding a universal key (if somebody comes along a million years later, will they share enough context to decode the archive?)

What would the Millennium Clock[0] version of a data storage device look like? [0]http://longnow.org/essays/millennium-clock/ http://longnow.org/essays/millennium-clock/

mabbo 8y ago

It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency. I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem. If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth. Hmm. This sounds like a fun project to try...

DataWraith 8y ago
That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive. [1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Edit: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ has more details.
zrm 8y ago
If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding. If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting. If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.
- mabbo 8y ago
  And here I go on a wiki spiral. Thanks!
- voltagex_ 8y ago
  Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.

roywiggins 8y ago

From the same people: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ It doesn't really scale, but then neither does the Clock.

dekhn 8y ago

It would be a collection of lithographically etched plates. These are stable across a wide collection of conditions. The coding scheme would be reed solomon or a variant of it, or possible even just massive duplication with checksums. Multiple duplicate backups of the entire data would be stored in geographically disparate locations. This reduces the "asteroid hits <city>" problem. A wide range of different technologies would be used, randomly, across the copies. This reduces the likelihood of systematic catastrophic failure due to a specific technology not working as expected. This is not that different from how current millenia-scale data storage has worked for... well, millenia. The primary challenge is in finding a universal key (if somebody comes along a million years later, will they share enough context to decode the archive?)