4 ms·

> Distribute your public data globally for free to ensure it is available forever! What steps are in place to ensure this over reasonable timescales (20-50 yea

by IanCal 8y ago

> Distribute your public data globally for free to ensure it is available forever! What steps are in place to ensure this over reasonable timescales (20-50 years)?

stephengillie 8y ago
What would the Millennium Clock[0] version of a data storage device look like? [0]http://longnow.org/essays/millennium-clock/ http://longnow.org/essays/millennium-clock/
- mabbo 8y ago
  It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency. I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem. If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth. Hmm. This sounds like a fun project to try...
  - DataWraith 8y ago
    That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive. [1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Edit: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ has more details.
  - zrm 8y ago
    If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding. If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting. If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.
    - mabbo 8y ago
      And here I go on a wiki spiral. Thanks!
    - voltagex_ 8y ago
      Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.
- roywiggins 8y ago
  From the same people: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ It doesn't really scale, but then neither does the Clock.
- dekhn 8y ago
  It would be a collection of lithographically etched plates. These are stable across a wide collection of conditions. The coding scheme would be reed solomon or a variant of it, or possible even just massive duplication with checksums. Multiple duplicate backups of the entire data would be stored in geographically disparate locations. This reduces the "asteroid hits <city>" problem. A wide range of different technologies would be used, randomly, across the copies. This reduces the likelihood of systematic catastrophic failure due to a specific technology not working as expected. This is not that different from how current millenia-scale data storage has worked for... well, millenia. The primary challenge is in finding a universal key (if somebody comes along a million years later, will they share enough context to decode the archive?)
ieee8023 8y ago
The project is run by the U.S. 501(c)3 Non-profit called Institute for Reproducible Research (http://reproducibilityinstitute.org http://reproducibilityinstitute.org) and this site has an overhead cost of ~$500/year. We plan to fund this project for at least the next 30 years. The community hosts the data and we also coordinate donations of hosting from our sponsors (listed on the home page). We also run the project ShortScience.org! Check it out!
- IanCal 8y ago
  > The community hosts the data This is the key thing for me, so there are no guarantees of the data being available? Or is all the data backed by an owner hosted box (and backups)?
  - ieee8023 8y ago
    The main goal is to facilitate distribution. We work with research groups and provide "best effort" hosting with the resources from our sponsors. We expect the uploader has a backup of the data in all cases.

by IanCal 8y ago

> Distribute your public data globally for free to ensure it is available forever! What steps are in place to ensure this over reasonable timescales (20-50 years)?

stephengillie 8y ago

What would the Millennium Clock[0] version of a data storage device look like? [0]http://longnow.org/essays/millennium-clock/ http://longnow.org/essays/millennium-clock/

mabbo 8y ago
It wouldn't be a single device. That's far too subject to attacks. Instead, let's picture a decentralized system like BitTorrent, but with the goal of data resiliency. I picture a system where a person adds themselves as a node to a certain group of files- like BitTorrent- but instead of downloading everything, they choose how much space and bandwidth they're offering and the system grabs the "best" pieces. Best in this case is about spreading the data out to improve resiliency, like any dates storage service would. It's not a human choice but an optimization problem. If 250 people each offered 100MB of space, a 5GB file could be maintained with quadruple redundancy. Nodes come and go, and the system would minimize data transfers while aiming to maximize redundancy. Try to spread the files far and wide geographically, to put popular pieces of the data onto nodes offering more bandwidth. Hmm. This sounds like a fun project to try...
- DataWraith 8y ago
  That sounds a lot like IA.BAK [1], a git-annex backed attempt at making a backup copy of the Internet Archive. [1] https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.BAK/git-annex_implementation https://www.archiveteam.org/index.php?title=INTERNETARCHIVE.... Edit: https://git-annex.branchable.com/design/iabackup/ https://git-annex.branchable.com/design/iabackup/ has more details.
- zrm 8y ago
  If you're going to build something like this, especially when the data sets are typically large, this is a very strong case for erasure coding. If you split a 5GB file into (20) 250MB chunks and then mirror each of them four times, there is a nontrivial probability that one of the chunks loses all four mirrors. Especially when you're using notoriously unreliable volunteer hosting. If you split the same 5GB file into (20) 250MB data chunks and (60) 250MB erasure chunks, you consume the same amount of total storage but have to lose 61 chunks (>75% of all the hosts) before you lose any data.
  - mabbo 8y ago
    And here I go on a wiki spiral. Thanks!
  - voltagex_ 8y ago
    Offtopic: Have you looked at PAR2? I seem to recall it being really good but having some critical bug.
roywiggins 8y ago
From the same people: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ It doesn't really scale, but then neither does the Clock.
dekhn 8y ago
It would be a collection of lithographically etched plates. These are stable across a wide collection of conditions. The coding scheme would be reed solomon or a variant of it, or possible even just massive duplication with checksums. Multiple duplicate backups of the entire data would be stored in geographically disparate locations. This reduces the "asteroid hits <city>" problem. A wide range of different technologies would be used, randomly, across the copies. This reduces the likelihood of systematic catastrophic failure due to a specific technology not working as expected. This is not that different from how current millenia-scale data storage has worked for... well, millenia. The primary challenge is in finding a universal key (if somebody comes along a million years later, will they share enough context to decode the archive?)

ieee8023 8y ago

The project is run by the U.S. 501(c)3 Non-profit called Institute for Reproducible Research (http://reproducibilityinstitute.org http://reproducibilityinstitute.org) and this site has an overhead cost of ~$500/year. We plan to fund this project for at least the next 30 years. The community hosts the data and we also coordinate donations of hosting from our sponsors (listed on the home page). We also run the project ShortScience.org! Check it out!

IanCal 8y ago
> The community hosts the data This is the key thing for me, so there are no guarantees of the data being available? Or is all the data backed by an owner hosted box (and backups)?
- ieee8023 8y ago
  The main goal is to facilitate distribution. We work with research groups and provide "best effort" hosting with the resources from our sponsors. We expect the uploader has a backup of the data in all cases.