11 ms·
How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less t
by allovernow 7y ago
How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less than 30 years before physical degradation- NAND, tape, disc, whatever. That makes for a brittle civilization when the vast majority of our knowledge is stored on media and would be unrecoverable just 3 decades after a global calamity.
- tialaramex 7y agoIf you want data to survive you've always needed to copy it. Digital storage just makes that easier to do in bulk. Copying Bibles was a full time task for huge teams of monks but you can (and should) make backups routinely on a daily or weekly basis with barely a thought.
- crmrc114 7y agoThis is a huge problem for any data- https://www.theatlantic.com/technology/archive/2015/02/how-to-send-a-message-1000-years-to-the-future/385720/ https://www.theatlantic.com/technology/archive/2015/02/how-t... Even communication of a simple warning message for yucca mountain and WIPP proved how hard it was going to be to communicate danger over 10000 years from now. https://en.wikipedia.org/wiki/Long-time_nuclear_waste_warning_messages https://en.wikipedia.org/wiki/Long-time_nuclear_waste_warnin...
- CaptArmchair 7y agoThe LOCKSS project is an approach used in university networks around the world to preserve research data for the long term. https://www.lockss.org/ https://www.lockss.org/ The big issue isn't the technology, it's the vast amounts of data that are being created at this point. Storage is cheap, but the labor that goes into managing the longevity of datasets isn't: it's essentially continually keeping your infrastructure up-to-date whilst also ensuring the integrity and readability of the datasets as was intended when they were first created. It implies regular checks of bit integrity, readability of your data, checking that you can restore your data, ensuring that you can access the data, making sure that you can find the data and everything is catalogued, ensuring that you have the rights and license to use the data,... When it comes to physical archives of the past, you have to be aware of your own survivorship bias. We only have an idea of what is preserved to the extent that documents are archived, recorded and thus discoverable. What we do not know is how much knowledge and information was lost to the past. When you look at documents, you're always limited to what's there. And when you hit the boundaries of what's there, then you may have indications that there was far more in the past, but you have to conclude: sadly that's lost. Either because it is physically lost, or because it might be somewhere in the archive but it's not registered yet in a catalogue and therefor not accessible. That's why I think that making backups with "barely a thought" is only as effective as to the extent to which you have organized your data, used accessible / readable data formats and filesystems. For instance, most people these days generate endless streams of photos with their digital devices, which then get automagically uploaded to cloud services. And that's great. The downside of that is that your ability to find a specific picture from 5 years ago is entirely restricted to the extent that you were able to organize and add specific metadata to that picture. Let alone, if you did take the opportunity to do so. That's why I advise people to sit down, and take time to go through their digital albums to pick the nicest or most important pictures they have, print them out on quality photo paper in several copies and store them with labels in albums at different physical locations. When it comes to longevity, your physical albums will still be accessible to your descendants some 70 or 100 years down the line. Something that isn't remotely guaranteed by cloud solutions. And that's just photos. Consider e-mail or the countless of closed messaging apps you have been using these past years. And then scale the problem beyond the personal but to entirety of large organizations, many of which are required by law to keep an archive of their documents, correspondence and so on, not just for decades but sometimes also for perpetuity.
- codetrotter 7y ago> The downside of that is that your ability to find a specific picture from 5 years ago is entirely restricted to the extent that you were able to organize and add specific metadata to that picture. I disagree with the premise that we should spend time manually organizing and tagging our pictures all that much. The metadata that the phone adds to pictures – time stamp and GPS coordinates – is already sufficient in a lot of cases for finding pictures that I look for. And where that metadata is insufficient, improved search powered by machine learning will come to the rescue. And not just tomorrow but even today. Just the other day, a few weeks back, I was standing in the kitchen that I share with two other people and I wondered to myself whether the kitchen knife in the dishwasher was mine (I’d bought a new one a few days prior but couldn’t remember what it looked like). I take a lot of picture of random stuff and mundane things, most of which I never bother to organize or tag or anything. I pull up my phone, search my photo library for “knife” and lo and behold, I did take a picture of it when I bought it and my phone has recognized the object in the photo to be a knife so it was able to find it for me. Important files and photos I do organize. Specifically for three reasons: 1. Ease of access. 2. Grouping related data together. 3. Tying photos and other data to abstract concepts like ideas for possible games or products. So I am not advocating no organization or tagging at all. But I think a lot of people are unaware or at least haven’t really incorporated the distinction between information that is already present in the data, and information that must be manually added. So they spend a lot of time manually creating folder structures that encode information which could already be automatically derived from the data itself. As for messages in closed apps, I just screenshot them. And I am relying on OCR technology to be or become good enough to refind those messages in the future. That way, if the platform itself is gone by then or the messages are not on the platform itself or hard to find on the platform itself for whatever reason. So far I haven’t even needed to use OCR. Because if I look for a message I often have other memories of where I was, when it was or something else that happened around that time. So I just jump back in time in my photo stream and either find the screenshot right away or I find pictures near-by in time and spend a tiny amount of time looking forwards and/or backwards in time and I find the screenshot. I do wish though, that iOS would automatically tag screenshots with the name of the app that the screenshot was taken in. And I think it would be cool if the screenshots were stored as SVG with pure text and vector shapes plus embedded bitmaps, so that the whole potentially needing robust OCR in the future thing could be side-stepped.
- aglavine 7y agoI have 30 year old CDs that play as good as the first time, as well as older vinyl discs. I'm sure I own 20 year old DVDs in same condition.
- ChrisArchitect 7y agosame, but I've also seen some cheaper makes of CDRs break down after 20 years and the storage layer crack etc.
- sigstoat 7y agoquite. the writable discs don't last nearly as long. i had a pile of writable DVDs that were unreadable 8 years later when i went back to fetch the data from them.
- iamphilrae 7y agoAudio CDs can survive a bit of degradation to the odd data bits here or there; the music will just skip the millisecond of missing data and your ears won’t notice. Likewise with vinyl. Data discs on the other hand have the issue that if a single data bit is lost, a whole file could be corrupted, especially if it’s a zipped file. In addition, printed audio CDs are of a different build than CD-Rs which have been found to not be as resistant to moisture and light.
- jandrese 7y agoIt takes more than a single bit--there are Huffman codes built into the spec, but it is certainly possible for the bits to degrade enough to render a sector or even the whole disc unreadable.
- pmiller2 7y agoIf you're serious about using optical media as archival storage, you can mitigate this by incorporating your own error correcting codes into the data storage format.
- 7y ago
- quag 7y agoAfter going down this rabbit hole, I concluded that M-Disc is the right trade off at the moment. They’re not too expensive, the writers are available, they’ll last, and they could be reverse engineered if discovered in the future. The second choice is using hard drives (easily available) and every so often power them up and copy data to new drives. If you have a small quantity of data, then encode and laser print onto paper, with a font designed for optical scanning or QR code’s. https://en.wikipedia.org/wiki/M-DISC https://en.wikipedia.org/wiki/M-DISC
- eitland 7y agoSeems hard to get hold of M-DISCs now. Are they still produced?
- Seenso 7y ago> They’re not too expensive, the writers are available, they’ll last, and they could be reverse engineered if discovered in the future. IIRC, the writers for M-DISCs are special, but the reader can be any DVD or Blu-ray drive. Honestly, I think the biggest consideration for digital archival media isn't so much the longevity of the media, but the future availability of equipment to read it. That's one of the biggest benefits of paper, IMHO. Besides being very well-understood material, nearly everyone is born with the necessary reading equipment and the decoding software is very common.
- reificator 7y ago> the decoding software is very common. [Citation needed]
- Spooky23 7y agoDo what archivists do - copy. The challenge is keeping things accessible. Only copying does that for electronic media. That includes getting things like photos printed.
- allovernow 7y agoHow many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I think this is an excellent place for neural networks. They can preserve vast amounts of data compactly for many data types because they statistically compress high level abstract data which can then be used to fill in regions with high error rate, although if you did that at a large scale you'd probably end up with some constant error rate fluctuating around the average true value. All indications point to the fact that we seem to be working against the unstoppable Force of entropy - indefinite error free preservation of data is ultimately impossible.
- bobthepanda 7y agoConsidering we can read parchments thousands of years old, and stone tablets going back even further, perhaps we should focus efforts on copying what we can to the time-tested preserved data formats.
- FillardMillmore 7y agoIm going to assume this is tongue-in-cheek, but if you happen to find a way to copy Mario 64 to a stone tablet, I'd love to hear about it.
- EwanG 7y agoNot THAT difficult - assuming you have enough stone tablets. I mean the program code exists. The problem I think you'd have (and true of most of these digital artifacts) is all the other hardware and software you would have to similarly encode to be able to recreate the machine and OS capable of running it...
- allovernow 7y agoHonestly, paper lasts thousands of years. Maybe the solution is microprint and OCR? How long does microfiche last?
- ThomasBHickey 7y agohttps://www.nedcc.org/free-resources/preservation-leaflets/6.-reformatting/6.1-microfilm-and-microfiche https://www.nedcc.org/free-resources/preservation-leaflets/6... "500+ years under proper storage conditions"
- orbital-decay 7y agoDigital storage depends on a long and complicated chain of formats, standards, technologies, businesses, software, services etc which come and go every decade or two, or even more often. Any of it is lost, and your archive isn't an archive anymore. So to store something long-term, you have to eliminate single points of failure, such as encodings, formats, and even human languages. It can be completely non-obvious, and the physical media isn't the most important one. Naturally, "civilizational scale archival" is only feasible for a proper archival organization such as a museum, a library or an archive. As a person, you can't have this. You can use the archival-grade media like M-Disc, but don't expect to put something on it and recover it 50 years later easily. You have to design the process to validate and migrate the data every once in a while. Digital storage can't offer something comparable to a simple printed photo. > when the vast majority of our knowledge ... would be unrecoverable just 3 decades after a global calamity. The vast majority of our knowledge is encoded in the societal and economic context. There's simply no way to translate it to any media, and any disruption would be the end of it.
- WalterBright 7y agoDo what I do - copy it forward every couple years onto new media. My oldest files are from 1977 - proof: https://github.com/DigitalMars/Empire-for-PDP-10 https://github.com/DigitalMars/Empire-for-PDP-10 My files have gone from magtape to 8" floppy to various 5.25" floppy to 3.5 floppy to zip drives to cdroms to dvdroms, then to hard disks of ever-increasing size. (My old hard drives are completely unreadable now.) I'm sorry I never kept my punch card decks. I'm sure there was nothing but crap on them, but it would be fun to see what kind of crap it was.
- dmix 7y agoFortran always seems like such an odd looking language to a modern developer like myself, who's learned a fair share of languages. I want to sit down and figure it out one day given how prominent it was (and still is for many critical systems).
- WalterBright 7y agoThe Fortran I used for Empire was FORTRAN-10, a DEC variant with many extensions to the then-Standard FORTRAN IV. There was an earlier Empire I wrote in BASIC, but alas it was lost. Some of its vestiges remain in the Fortran version, such as a variable named Z6. (Variables in that BASIC were one letter and one digit.)
- eaurouge 7y agoNot particularly user friendly, yet, but Arweave [0] was built specifically for this use case. 0. https://www.arweave.org/archiving.html https://www.arweave.org/archiving.html
- EwanG 7y agoSounds a lot like Archive.org? Am I missing something?
- penguinpalace 7y agoYeah, it's similar but Archive.org relies on servers to store information, whereas Arweave is a decentralised peer-to-peer network that offers permanent data storage. Arweave are in talks with them and are working with the IA to timestamp all of their collections.
- jointpdf 7y agoIt may not be “accessible to the layman” (yet?), but this seems rather intriguing: “The memory crystal is capable of storing up to 360 terabytes of data for billions of years.” https://en.m.wikipedia.org/wiki/5D_optical_data_storage https://en.m.wikipedia.org/wiki/5D_optical_data_storage
- rogerdpack 7y agomaybe forever.com ...
- slim 7y agoI've always wondered what makes micro SD cards not suitable for long term storage. What makes them decay
- ta999999171 7y agoThey store things the same way SSDs, do, with electrical charge. It weakens. That's a shit explanation, I'd definitely look up "how does flash storage degrade".
- rasz 7y agoIts flash memory, physically pushing electrons into a jar and hoping to count them later before all leak out. Endurance tests of SSD drives showed unplugging a worn drive for a couple of days was enough to lose all data. Not to mention SD cards usually get the lowest grade of flash memory.
- nicolaslem 7y agoI never understood why SD cards are so unreliable (or a least have this reputation) as they are usually used in cameras to store pictures. Something that people don't really joke with.