9 ms·

Backing up 18 years in 8 hours

186 points by chaosmachine 10y ago

derefr 10y ago
Is there a good reason to back up raw disk images with block-level deduplication, rather than running file recovery to get filesets, and then doing file-level deduplication? On the one hand, I can imagine "cryonically" preserving a disk image for a later, better filesystem recovery program to come around. (This "cryonic" approach would give even better results by preserving bit-level analogue flux recordings of the disk platters, rather than relying on the output of digital reads from the disk heads.) On the other hand, the longer you leave these disk images as dead blobs of data, the more layers of legacy container+encoding formats you'll have to try to get your system emulating when you finally do want to pull the files off. One day your OS won't have drivers for reading e.g. FAT16, or zfs2, or ReiserFS2.11, nor will it be able to parse out the meaning of an MBR-partitioned disk. Reaching back through Linux kernel archives for something old enough to understand those things, will result in a kernel that won't boot your PC. You'll end up having to do something convoluted with qemu just to get your disk read. Personally, I'd much rather throw out all the intermediate containers I can, as soon as I can: not just extracting files from the disk's filesystem, but further extracting files from any proprietary archive formats on the disk (using the extractor tools probably installed on the same disk), and even canonicalizing containers like AVI by remuxing them into modern extensible formats like MKV. The goal being to give a file-level deduplication process the best possible inputs to work with, most likely to match: not just for space-saving, but because reducing "junk duplicates" helps greatly in actually finding anything in all that mess, let alone organizing it.
- toomuchtodo 10y ago
  Disk is cheap in online storage services, so its "cheaper" in time to dump the image and pull out whatever you need in the future. > One day your OS won't have drivers for reading e.g. FAT16, or zfs2, or ReiserFS2.11, nor will it be able to parse out the meaning of an MBR-partitioned disk. Maybe in 10-20 years.
  - rangibaby 10y ago
    Couldn't you just load them up in a VM anyway?
    - deleted 10y ago
      [deleted]
    - chromakode 10y ago
      Yes, for reading these disk images, I've typically resorted to using VMs anyway, so I can experiment with old code in a sandbox. Actually, one project I've had in the back of my mind is attempting to boot some of those old Linux installs in a VM. Can you imagine re-opening a carbon copy of your old workspace from over a decade ago, right where you left it? I think the exercise would be thought-provoking.
      - marak830 10y ago
        Damn that would be interesting. I should make a backup of my current one and see how it changes year after year. Good idea!
      - evincarofautumn 10y ago
        Since I’m young, a decade ago I probably hadn’t discovered proper version control. I expect I’d find folders of neatly organised, timestamped ZIP files of my code. Perhaps some things are best left in the past.
    - michaelt 10y ago
      A few weeks ago I tried to copy a Windows 7 VM from VirtualBox to VMWare Player. I expected it to be easy, but it was actually difficult enough that I gave up on it after a few days. If your backup plan involves mounting a disk image on a VM, I strongly advise you to test it before you need it :)
      - chromakode 10y ago
        Emphatically agreed. Migrating VMs (IME especially Windows VMs) is a frustrating proposition. Creating new Linux VMs expressly for the purpose of reading particular file formats is a much safer bet. It is unlikely that the ability to conveniently emulate a basic i386 system with block storage will go away in the next few decades. My assumption is that any formats I can read on Linux today will be readable in the future, as long as I have a copy of the source / binaries. This is why I included a copy of lrzip's git tree with my backups -- everything else is in a standard Ubuntu install.
- chromakode 10y ago
  Interesting point! To be honest, I didn't consider that it might be more difficult to interpret the underlying filesystems and encodings in the future. My goal was to get a lossless (well, as much as possible) archive of the disks since my time and physical access was limited. It is hard to predict what I will want to access in these images over the scale of decades, so my thinking was to leave them as untouched as possible, since I don't know what information will be important. That being said, a good guess would be that the most interesting data will be media files (especially old photos) and documents. For that data, your advice of collecting and re-encoding the files is wise. For the purpose of discovering the media files in these backups, I found my favored approach to be a brute force recursive search for file types. Exploring the original structure of the filesystems was interesting, but my intuition for where the valuable data was usually proved wrong.
- hvidgaard 10y ago
  You want to convert the data to an open format and keep that around in modern containers, where you can easily transfer it to a different container should that be necessary. BUT for many applications you also want to preserve the environment. For instance, if you write your thesis in LaTeX, whatever installation you have is unlikely to be replicable in 10 years, so keeping a minimal VM that can compile it is preferable. You would need to keep this VM runable and upgrade it to newer formats as things progress.
  - todd8 10y ago
    LaTeX is plain text markup so old LaTeX documents can usually just be read without processing. (The major exception is that graphics are hard to visualize as a sequence of draw commands!). Further, ordinary LaTeX from the last few decades can still be processed without difficulty. The older versions of LaTeX are still available online. Of course, things are never as simple as they should be. LaTeX is a markup language that through macro expansion ends up expanding into TeX. Since TeX 3.0 in 1989, Knuth has attempted to keep the TeX system stable. Since then TeX documents should produce the same output, pixel for pixel, as they do now running on the current version 3.14159265 (yes, the version number is converging to pi, there won't ever be a TeX 4.0). Few people, however, produce documents in plain TeX--the LaTeX markup is so much more convenient than the lower level TeX. LaTeX has been slowly evolving and there are some backward compatibility issues, but they are minor. The first release of LaTeX seeing general use was described by Leslie Lamport (the creator of LaTeX) in his 1985 book[1]. That version of LaTeX, 2.09, can still be processed by today's LaTeX 2e in compatibility mode. LaTeX 3 is supposed to supersede LaTeX 2e someday, but it's not clear how many more years that will be. Since LaTeX is open source the distributions from TUG (the TeX User's Group) are easily obtained and they have all the historical versions of LaTeX and TeX available. This all makes LaTeX/TeX seem like one of the best ways to maintain a document's source for the future. A few tips: - Fonts can be a problem because fonts evolve. Either use something like TeX's extensive collection of "built in" Computer Modern fonts or save the font files along with the source of anything that you might want to work on in twenty years. - LaTeX has a wide number of very sophisticated third party extensions. Along with the document source it would be a good idea to keep the contemporaneous versions the extensions used by the document. (These extensions are just files of additional macros.) - If one is only interested in the typeset results of a LaTeX document, use a LaTeX extension like pdfx (or Adobe Acrobat) to generate a PDF/A version of the document (LaTeX programs normally generate pdfs). PDF/A is a pdf specification from Adobe that is intended for archival use and rendering far in the future (fonts are embedded in the output, etc.). - Pandoc can convert LaTeX to a wide number of alternative formats (HTLM etc.) with some success depending on the target language and the document complexity. [1] http://www.amazon.com/Latex-Document-Preparation-System-Users/dp/020115790X/ref=sr_1_2?ie=UTF8&qid=1460728027&sr=8-2&keywords=latex+lamport http://www.amazon.com/Latex-Document-Preparation-System-User...
- hendryau 10y ago
  Do you know of any good resources on storing/maintaining video files? I recently ripped over a TB of uncompressed video from old 8mm tapes and I'd like to preserve these memories digitally for as long as possible.
  - voltagex_ 10y ago
    Lossless x264 (with FLAC audio) in MKV with a recent version of FFMPEG.
    - chinathrow 10y ago
      Better store the sourcecode of the exact version used for transcoding/playback and back it up propperly too. You might need to re-recreate the binaries in dozens of years from now.
- cableshaft 10y ago
  Just curious, why would you choose MKV over MP4? MP4 has native HTML5 support, so I figured that would have more long-term viability. And as far as I know, they're both open formats (maybe MP4 isn't and is just free to use, I'm not sure).
  - Intermernet 10y ago
    > maybe MP4 isn't and is just free to use, I'm not sure Correct. MP4 is not an open format. From [1]: > MPEG-4 contains patented technologies, the use of which requires licensing in countries that acknowledge software algorithm patents. Over two dozen companies claim to have patents covering MPEG-4. MPEG LA licenses patents required for MPEG-4 Part 2 Visual from a wide range of companies (audio is licensed separately) and lists all of its licensors and licensees on the site. New licenses for MPEG-4 System patents are under development and no new licenses are being offered while holders of its old MPEG-4 Systems license are still covered under the terms of that license for the patents listed (MPEG LA – Patent List). > AT&T is trying to sue companies such as Apple Inc. over alleged MPEG-4 patent infringement. The terms of Apple's QuickTime 7 license for users describes in paragraph 14 the terms under Apple's existing MPEG-4 System Patent Portfolio license from MPEG LA. [1]: https://en.wikipedia.org/wiki/MPEG-4#Licensing https://en.wikipedia.org/wiki/MPEG-4#Licensing
peterburkimsher 10y ago
I still have a disk image of the Mac Plus I used until I was 10, when I upgraded to a new Blueberry iBook. The Mac Plus experience can be recreated with Mini vMac, an emulator. Files since 1999 are all still in a folder on my local drive. I've made an effort not to lose the photos and chat logs. My iTunes database dates back to 2002, and I'm disappointed that I lost my old SoundJam MP database that I used for 2 years before that. Unfortunately, my archives continuously come under threat from forced upgrades. Upgrading iPhoto to the Photos app will lose a lot of metadata. The latest versions of iTunes removed USB sync for contacts & calendars, which really bothers me when I want to maintain my files without the cloud. Many old documents are now unreadable (e.g. Clarisworks). I've realised that simplicity is essential for keeping long-term records. Keeping a copy as plain text is important for preservation. Just because "there's an app for that" today doesn't mean that 10 years from now, the app will still work. This is (especially) true for companies that should know better, such as budget tracking tools. I recently went through my old records to compile a list of every rock concert I've been to, along with the location and price when possible. With heavy use of archive.org, my iCal, and my iPhoto library, I figured out all the dates, but it was a major effort. Most of them were in the last 10 years. My dad still has his PhD thesis on a magnetic tape. Anybody who knows how to read that onto a PDP-8 should get in touch!
- chromakode 10y ago
  I feel your pain. Especially with tricky formats like Clarisworks (or in my case, old Cakewalk and Hash Animation Studio) reviving the data might require emulating the entire OS and environment they existed in. At least for full disk images, the backups contain copies of the software used to create the files. It's really disappointing when software uses unnecessarily complex file formats. Chat logs are an important example: I was dismayed to discover that my Yahoo Messenger chat logs on Windows are stored XOR-obfuscated! [1] There's a lot of compromises to be made between the convenience of making backups and their long term viability. My hope is that by greedily saving all of the bits, there will be enough context to make sense of the data if I really needed to, even if it is an intensive process. A good alternate strategy would be to scoop up the most likely interesting files when processing the backups and re-encode them in more future-ready formats, as derefr suggested. [1] https://cryptome.org/isp-spy/yahoo-chat-spy.pdf https://cryptome.org/isp-spy/yahoo-chat-spy.pdf
khc 10y ago
No comment from him trying to do a restore?
- chromakode 10y ago
  Thanks for mentioning this. I did test the restore flow (download from Nearline, decrypt, unlrzip, and verify), but didn't note it in the blog. It's critical to actually test the backups! Edit: updated the post to mention this as well.
imrehg 10y ago
I've lost so much data over the years (at least 10-15 years worth of digital photos, collected documents, everything), and multiple times. And still... I only have plans to have organized and scheduled backups, and never really executed on it. It's a bit like a big part of my life never existed (just in my mind, and when lucky in lower resolution on Facebook).
- hvidgaard 10y ago
  You should get CrashPlan or similar that is more or less a "fire and forget" kind of backup where you can recover most of what you lose.
  - zerd 10y ago
    Oh, if only CrashPlan existed back then. I've also lost numerous files due to stupid mistakes and failing hard drives. Can't find any photos from before 2006. Now I use CrashPlan, Dropbox w/packrat, Arq and local backups to ZFS w/snapshots.
  - imrehg 10y ago
    Oh, this looks interesting, thanks a lot!
- deleted 10y ago
  [deleted]
agumonkey 10y ago
I recently put back an old P3 box (laptops don't have Floppy Disk ports) to access an HP Colorado Backup Tape. 400MB only but some interesting stuff like Word HTML page, some old games, the much nicer than I remember QBASIC.EXE[1] and most impressively TURBO.EXE (Turbo Pascal 7 IDE) which clocks at a whopping 600KB. [1] that thing had pseudo modules, a visual tracer, live indent and lint (limited but still).
- agumonkey 10y ago
  Couldn't resist: http://imgur.com/yKarXLK http://imgur.com/yKarXLK http://imgur.com/oT3u4fR http://imgur.com/oT3u4fR
evincarofautumn 10y ago
“Treat any object that is elevated from the floor like it will fall.” I find this an excellent metaphor for programming. In particular: low-level debugging of high-level code is inevitable, so choose your abstractions carefully.
- erelde 10y ago
  I want to apply that to politics/History as well. That said, this is the one sentence my 1st grade teacher always kept repeating and stuck with me ever since. (it was, at the time, about pens falling from desks)
k_sze 10y ago
Mandatory XKCD reference: https://xkcd.com/1360/ https://xkcd.com/1360/ My point is. The technical aspect of moving data around is actually not that challenging compared to the "business logic" of organizing your old files.
illumen 10y ago
cd Desktop/old/old/old/old/old/old/old/old/old/
- chromakode 10y ago
  Some of the images I recovered actually contained directory structures like that, which included some of the oldest files I was able to recover. This approach can work out if the filesystem eventually gets preserved!
simula67 10y ago
Did you consider using OneDrive or Google Drive ? Something like http://www.duplicati.com/ http://www.duplicati.com/
- cm2187 10y ago
  I'd say the only thing more volatile than a forgotten hard drive is a cloud service. They will come and go, go bust or get shut down, get depreciated and replaced. And if you forget to pay, don't count on finding your old forgotten data in 15 years.
  - drdaeman 10y ago
    Glacier, Nearline or B2 are "cloud" services as well, and while they're more stable than ordinary-consumer-grade services (like Google Drive or OneDrive), the it's not like their future is set in stone (affordability, service policies, etc) I believe the way to get the best reliability is to keep multiple replicas of the archive (git-annex's great here) over storage options from as many different vendors as possible, and hope the chance of everyone going down at nearly the same time is too low. And probably schedule some periodic fully-automated checks to minimize the chance some service had degraded and had silently lost the data.
    - cm2187 10y ago
      In that respect my recommendation is to buy some of these massive Seagate archive disks. I don't know how durable they are but given you can have 8TB for less than $200, I would have one or two in independent places. But I would encrypt the data though. But then where do you store long term the key (brain is a bad choice!)? And as technologies evolve, in 5y there will probably be cheap 30TB HDD, so it will make sense to consolidated, etc
      - Intermernet 10y ago
        > where do you store long term the key I know this is clunky, but I'd print out the key (or possible laser-cut the key!) onto durable media and store it in a safe / safety deposit box etc. It's going to be annoying to manually type it in, but it's a reliable method of last-resort.
jbverschoor 10y ago
Oh yeah, ddrescue rescues my ass once. My macbook disk crashed, and my timemachine wasn't working for some time.. It was 200GB of data. What I did was connect the disk to a linux machine, and use ddrescue for a month 24/7 to try and read the bad disk. I lost 0 bytes.
Esau 10y ago
For me, the most important take away from this article is the importance of keeping files in formats that are open. On my Macbook Air, all of my personal files are in text, jpeg/png, pdf, or mp3 formats. Of course, even open formats can fall out of favor, but that doesn't seem to happen as often. I also like that you encrypted your backups, although in doing so you probably made Senator Dianne Feinstein cry.
21 10y ago
Will he in another ten years remember the password? Happend to me. Encrypted an archive, and the funny thing is I did save the password into a password manager, but I didn't label it descriptively enough, so when I tried decrypting the archive I couldn't locate the right password. I could have just tried them all, but I had a copy of the files from the archive in another place so I just deleted the encrypted archive.
- cpach 10y ago
  I’m a happy Arq user since a few weeks back, but this scenario scares me. I have even considered using a weak password such as ”abc123” in order to prevent this issue.
  - aidos 10y ago
    Do you think you'll remember that weak password in 10 years time? Or are you thinking you'll brute force it when the time comes? In which case there's not much reason to encrypt in the first place.
    - cpach 10y ago
      I don’t know, but I definitely think it’s easier to remember that string than a 30 characters long random string. I could even use my given name to make it more memorable. To clarify: I didn’t primarily choose my backup application because it encrypts the backups but because it seemed like a reliable application with a solid user interface. (In Arq, encryption is not optional.)
      - hollander 10y ago
        You can repeat that string, or use five easy words to get to that 30 character string.
  - hollander 10y ago
    Then you get an accident, and you have amnesia. How about that scenario?
geekit 10y ago
Offtopic: Just a reminder to always remove GPS info from personal photos that are to be uploaded to internet. Photo in post gives me GPS coordinates on Westview Dr in Lake Oswego, OR. If we are so concerned about our privacy or encryption, we should not make such mistakes.
- sccxy 10y ago
  Thanks for posting his home address. Great job (To make a point you don't have to show full coordinates)
  - geekit 10y ago
    Even if I didn't, anyone can still download image and get it. Anyway, I have removed it.
- chromakode 10y ago
  Oops! I appreciate your bringing this to my attention. This is an unexpected downside to my custom git-based publishing scheme -- easy to leave out EXIF wiping in the asset pipeline. In the future, it's really important to attempt such notices privately first. Even if the information is out there, making that knowledge public in a popular spot greatly increases the exposure. I'm sure that with your skillset you could have a direct line to contact me in less than a minute.
maxaf 10y ago
I'd always regretted not keeping my old hard drives when my family left Moscow in 1999. There was the Robotron 1715, which was my first true taste of computing at home. A Compaq-produced 386-SX box was solely responsible for me having learned GW Basic and Pascal before I was a teen. My daughter, who's retracing my steps somewhat, would have loved to see how I had done it. I sense that seeing her dad's attempts at learning to hack might help her understand how harmful current iPad-esque walled gardens are. Not surprisingly, when I visited Moscow 15 years later, my old hardware was gone, and not a shred of knowledge remained of where it might had gone. My relatives conceded that it might have been stolen. I hope the thieving computer obsolescence club or history museum will enjoy my pieces.
apaprocki 10y ago
I went through a similar exercise but had to deal with long deprecated floppy interface tape drives. It took a number of days to source parts and trial and error to get an OS set up to read from them, but in the end it worked... and my BBS lived again inside a DOS emulator! https://twitter.com/apaprocki/status/550432891201941504 https://twitter.com/apaprocki/status/550432891201941504 In case anyone else has a stack of long forgotten various QIC formats -- don't lose hope, have patience and restore them :)
chinathrow 10y ago
For the truly paranoid, I advise to put an additional copy in cold storage at a bank vault at bank A. Print the GPG private key and passphrase out and put it in a bank vault at bank B. Handle permissions to access these vaults within your family as you like it. Bank A shouldn't be in the same region/area as bank B in case a desaster strikes and floods the vault(s). Then play that scenario through whenever you update your keys/data.
jlgaddis 10y ago
I got my first PC in 1990 at age 11, having used only Commodore computers before that. I wish I had had the foresight to preserve everything I've ever created. Cheap Internet access (i.e. a local dial-up number) didn't come to the area where I grew up (a small, midwestern town) until about 1996. From 1990-6, I was heavily involved in the BBS "scene". Being a teenager with lots of time but little money, I ran a BBS but couldn't afford to pay for all those door games. Instead, I learned to reverse engineer them and write "cracks" (binary patches, basically) to "unlock" those games. In the middle of the night, I'd dial into far away BBSes and upload them everywhere that I could. I'd love to have that source code to look at again today. I did find some of my old paper notebooks in my mother's garage a while back. These are the ones that I took to school with me. Instead of doing schoolwork, however, I'd spend my time in class writing code (on paper, by hand) and then I'd type it all in to the computer when I got home after school. It was very neat to find those and look back at code I wrote ~25 years ago. For the last few years, now that we all have a camera in our pocket at all times, I have kept backups of every picture I've taken but there are probably thousands of photos that I took before then that I'll never see again. With storage as cheap as it is, I've resolved to never again "lose" any of the photos or videos I've taken. It'll be amazing to look back at them in another ~25 years or so.
jlgaddis 10y ago
I'm aware of at least two old PCs of mine that are still in my mother's garage. One is a Pentium 100 that was primarily used as a router/firewall for my local network so there probably isn't much "data" on it. The other was slightly newer and may actually contain some of my old files. I think I'll bring them home the next time I'm there and see if I can find anything significant on them.
746F7475 10y ago
This is strange to me, I've been using computers since I can remember, but I only very late started to do anything productive with them. I've never had anything I've wanted to keep, I mean sure I now have some scripts that are valuable, but nothing old like that. It's also so strange when co-workers talk about how their kids are not just playing games but also creating mods for their games. When I was kid I just drew pictures, played with lego and played video games.
- fapjacks 10y ago
  Modding games is extremely easy these days. I had to walk uphill both ways through two feet of snow just to get some games to work in the early 90s. Buckets of blood, sweat and tears went into my MajorMUD scripts.
  - 746F7475 10y ago
    I have no idea how hard or easy it is, but it still seems like they are doing way more advanced shit than I ever did
    - fapjacks 10y ago
      I wonder how much of this is selection bias by media. There are now twice as many human beings on this planet as when we were kids (totally making assumptions about you here, if you don't mind). That means more gifted kids, in addition to our ever more invasive media. Our various feeds are (unfortunately) full of page after page of people talking about the things they think their kids are doing. However I don't think kids today are especially special snowflakes compared to when we were kids. I'm not saying there aren't gifted kids. I just mean it's probably not right or true to say that gifted kids these days are more amazing than gifted kids of yesteryear. And it doesn't help that every parent on the planet sells their kid as a genius. Even the parents of stupid kids.
oxplot 10y ago
Am I the only one blown away by Google Nearline? From https://cloud.google.com/storage/docs/nearline https://cloud.google.com/storage/docs/nearline: > Data is available in seconds, not hours or days. With ~3 second response times and 1 cent per GB/month pricing This is in contrast to 3 hours of Amazon Glacier [1]. [1]: https://aws.amazon.com/glacier/ https://aws.amazon.com/glacier/

186 points by chaosmachine 10y ago

Is there a good reason to back up raw disk images with block-level deduplication, rather than running file recovery to get filesets, and then doing file-level deduplication? On the one hand, I can imagine "cryonically" preserving a disk image for a later, better filesystem recovery program to come around. (This "cryonic" approach would give even better results by preserving bit-level analogue flux recordings of the disk platters, rather than relying on the output of digital reads from the disk heads.) On the other hand, the longer you leave these disk images as dead blobs of data, the more layers of legacy container+encoding formats you'll have to try to get your system emulating when you finally do want to pull the files off. One day your OS won't have drivers for reading e.g. FAT16, or zfs2, or ReiserFS2.11, nor will it be able to parse out the meaning of an MBR-partitioned disk. Reaching back through Linux kernel archives for something old enough to understand those things, will result in a kernel that won't boot your PC. You'll end up having to do something convoluted with qemu just to get your disk read. Personally, I'd much rather throw out all the intermediate containers I can, as soon as I can: not just extracting files from the disk's filesystem, but further extracting files from any proprietary archive formats on the disk (using the extractor tools probably installed on the same disk), and even canonicalizing containers like AVI by remuxing them into modern extensible formats like MKV. The goal being to give a file-level deduplication process the best possible inputs to work with, most likely to match: not just for space-saving, but because reducing "junk duplicates" helps greatly in actually finding anything in all that mess, let alone organizing it.

toomuchtodo 10y ago
Disk is cheap in online storage services, so its "cheaper" in time to dump the image and pull out whatever you need in the future. > One day your OS won't have drivers for reading e.g. FAT16, or zfs2, or ReiserFS2.11, nor will it be able to parse out the meaning of an MBR-partitioned disk. Maybe in 10-20 years.
- rangibaby 10y ago
  Couldn't you just load them up in a VM anyway?
  - deleted 10y ago
    [deleted]
  - chromakode 10y ago
    Yes, for reading these disk images, I've typically resorted to using VMs anyway, so I can experiment with old code in a sandbox. Actually, one project I've had in the back of my mind is attempting to boot some of those old Linux installs in a VM. Can you imagine re-opening a carbon copy of your old workspace from over a decade ago, right where you left it? I think the exercise would be thought-provoking.
    - marak830 10y ago
      Damn that would be interesting. I should make a backup of my current one and see how it changes year after year. Good idea!
    - evincarofautumn 10y ago
      Since I’m young, a decade ago I probably hadn’t discovered proper version control. I expect I’d find folders of neatly organised, timestamped ZIP files of my code. Perhaps some things are best left in the past.
  - michaelt 10y ago
    A few weeks ago I tried to copy a Windows 7 VM from VirtualBox to VMWare Player. I expected it to be easy, but it was actually difficult enough that I gave up on it after a few days. If your backup plan involves mounting a disk image on a VM, I strongly advise you to test it before you need it :)
    - chromakode 10y ago
      Emphatically agreed. Migrating VMs (IME especially Windows VMs) is a frustrating proposition. Creating new Linux VMs expressly for the purpose of reading particular file formats is a much safer bet. It is unlikely that the ability to conveniently emulate a basic i386 system with block storage will go away in the next few decades. My assumption is that any formats I can read on Linux today will be readable in the future, as long as I have a copy of the source / binaries. This is why I included a copy of lrzip's git tree with my backups -- everything else is in a standard Ubuntu install.
chromakode 10y ago
Interesting point! To be honest, I didn't consider that it might be more difficult to interpret the underlying filesystems and encodings in the future. My goal was to get a lossless (well, as much as possible) archive of the disks since my time and physical access was limited. It is hard to predict what I will want to access in these images over the scale of decades, so my thinking was to leave them as untouched as possible, since I don't know what information will be important. That being said, a good guess would be that the most interesting data will be media files (especially old photos) and documents. For that data, your advice of collecting and re-encoding the files is wise. For the purpose of discovering the media files in these backups, I found my favored approach to be a brute force recursive search for file types. Exploring the original structure of the filesystems was interesting, but my intuition for where the valuable data was usually proved wrong.
hvidgaard 10y ago
You want to convert the data to an open format and keep that around in modern containers, where you can easily transfer it to a different container should that be necessary. BUT for many applications you also want to preserve the environment. For instance, if you write your thesis in LaTeX, whatever installation you have is unlikely to be replicable in 10 years, so keeping a minimal VM that can compile it is preferable. You would need to keep this VM runable and upgrade it to newer formats as things progress.
- todd8 10y ago
  LaTeX is plain text markup so old LaTeX documents can usually just be read without processing. (The major exception is that graphics are hard to visualize as a sequence of draw commands!). Further, ordinary LaTeX from the last few decades can still be processed without difficulty. The older versions of LaTeX are still available online. Of course, things are never as simple as they should be. LaTeX is a markup language that through macro expansion ends up expanding into TeX. Since TeX 3.0 in 1989, Knuth has attempted to keep the TeX system stable. Since then TeX documents should produce the same output, pixel for pixel, as they do now running on the current version 3.14159265 (yes, the version number is converging to pi, there won't ever be a TeX 4.0). Few people, however, produce documents in plain TeX--the LaTeX markup is so much more convenient than the lower level TeX. LaTeX has been slowly evolving and there are some backward compatibility issues, but they are minor. The first release of LaTeX seeing general use was described by Leslie Lamport (the creator of LaTeX) in his 1985 book[1]. That version of LaTeX, 2.09, can still be processed by today's LaTeX 2e in compatibility mode. LaTeX 3 is supposed to supersede LaTeX 2e someday, but it's not clear how many more years that will be. Since LaTeX is open source the distributions from TUG (the TeX User's Group) are easily obtained and they have all the historical versions of LaTeX and TeX available. This all makes LaTeX/TeX seem like one of the best ways to maintain a document's source for the future. A few tips: - Fonts can be a problem because fonts evolve. Either use something like TeX's extensive collection of "built in" Computer Modern fonts or save the font files along with the source of anything that you might want to work on in twenty years. - LaTeX has a wide number of very sophisticated third party extensions. Along with the document source it would be a good idea to keep the contemporaneous versions the extensions used by the document. (These extensions are just files of additional macros.) - If one is only interested in the typeset results of a LaTeX document, use a LaTeX extension like pdfx (or Adobe Acrobat) to generate a PDF/A version of the document (LaTeX programs normally generate pdfs). PDF/A is a pdf specification from Adobe that is intended for archival use and rendering far in the future (fonts are embedded in the output, etc.). - Pandoc can convert LaTeX to a wide number of alternative formats (HTLM etc.) with some success depending on the target language and the document complexity. [1] http://www.amazon.com/Latex-Document-Preparation-System-Users/dp/020115790X/ref=sr_1_2?ie=UTF8&qid=1460728027&sr=8-2&keywords=latex+lamport http://www.amazon.com/Latex-Document-Preparation-System-User...
hendryau 10y ago
Do you know of any good resources on storing/maintaining video files? I recently ripped over a TB of uncompressed video from old 8mm tapes and I'd like to preserve these memories digitally for as long as possible.
- voltagex_ 10y ago
  Lossless x264 (with FLAC audio) in MKV with a recent version of FFMPEG.
  - chinathrow 10y ago
    Better store the sourcecode of the exact version used for transcoding/playback and back it up propperly too. You might need to re-recreate the binaries in dozens of years from now.
cableshaft 10y ago
Just curious, why would you choose MKV over MP4? MP4 has native HTML5 support, so I figured that would have more long-term viability. And as far as I know, they're both open formats (maybe MP4 isn't and is just free to use, I'm not sure).
- Intermernet 10y ago
  > maybe MP4 isn't and is just free to use, I'm not sure Correct. MP4 is not an open format. From [1]: > MPEG-4 contains patented technologies, the use of which requires licensing in countries that acknowledge software algorithm patents. Over two dozen companies claim to have patents covering MPEG-4. MPEG LA licenses patents required for MPEG-4 Part 2 Visual from a wide range of companies (audio is licensed separately) and lists all of its licensors and licensees on the site. New licenses for MPEG-4 System patents are under development and no new licenses are being offered while holders of its old MPEG-4 Systems license are still covered under the terms of that license for the patents listed (MPEG LA – Patent List). > AT&T is trying to sue companies such as Apple Inc. over alleged MPEG-4 patent infringement. The terms of Apple's QuickTime 7 license for users describes in paragraph 14 the terms under Apple's existing MPEG-4 System Patent Portfolio license from MPEG LA. [1]: https://en.wikipedia.org/wiki/MPEG-4#Licensing https://en.wikipedia.org/wiki/MPEG-4#Licensing

peterburkimsher 10y ago

I still have a disk image of the Mac Plus I used until I was 10, when I upgraded to a new Blueberry iBook. The Mac Plus experience can be recreated with Mini vMac, an emulator. Files since 1999 are all still in a folder on my local drive. I've made an effort not to lose the photos and chat logs. My iTunes database dates back to 2002, and I'm disappointed that I lost my old SoundJam MP database that I used for 2 years before that. Unfortunately, my archives continuously come under threat from forced upgrades. Upgrading iPhoto to the Photos app will lose a lot of metadata. The latest versions of iTunes removed USB sync for contacts & calendars, which really bothers me when I want to maintain my files without the cloud. Many old documents are now unreadable (e.g. Clarisworks). I've realised that simplicity is essential for keeping long-term records. Keeping a copy as plain text is important for preservation. Just because "there's an app for that" today doesn't mean that 10 years from now, the app will still work. This is (especially) true for companies that should know better, such as budget tracking tools. I recently went through my old records to compile a list of every rock concert I've been to, along with the location and price when possible. With heavy use of archive.org, my iCal, and my iPhoto library, I figured out all the dates, but it was a major effort. Most of them were in the last 10 years. My dad still has his PhD thesis on a magnetic tape. Anybody who knows how to read that onto a PDP-8 should get in touch!

chromakode 10y ago
I feel your pain. Especially with tricky formats like Clarisworks (or in my case, old Cakewalk and Hash Animation Studio) reviving the data might require emulating the entire OS and environment they existed in. At least for full disk images, the backups contain copies of the software used to create the files. It's really disappointing when software uses unnecessarily complex file formats. Chat logs are an important example: I was dismayed to discover that my Yahoo Messenger chat logs on Windows are stored XOR-obfuscated! [1] There's a lot of compromises to be made between the convenience of making backups and their long term viability. My hope is that by greedily saving all of the bits, there will be enough context to make sense of the data if I really needed to, even if it is an intensive process. A good alternate strategy would be to scoop up the most likely interesting files when processing the backups and re-encode them in more future-ready formats, as derefr suggested. [1] https://cryptome.org/isp-spy/yahoo-chat-spy.pdf https://cryptome.org/isp-spy/yahoo-chat-spy.pdf

khc 10y ago

No comment from him trying to do a restore?

chromakode 10y ago
Thanks for mentioning this. I did test the restore flow (download from Nearline, decrypt, unlrzip, and verify), but didn't note it in the blog. It's critical to actually test the backups! Edit: updated the post to mention this as well.

imrehg 10y ago

I've lost so much data over the years (at least 10-15 years worth of digital photos, collected documents, everything), and multiple times. And still... I only have plans to have organized and scheduled backups, and never really executed on it. It's a bit like a big part of my life never existed (just in my mind, and when lucky in lower resolution on Facebook).

hvidgaard 10y ago
You should get CrashPlan or similar that is more or less a "fire and forget" kind of backup where you can recover most of what you lose.
- zerd 10y ago
  Oh, if only CrashPlan existed back then. I've also lost numerous files due to stupid mistakes and failing hard drives. Can't find any photos from before 2006. Now I use CrashPlan, Dropbox w/packrat, Arq and local backups to ZFS w/snapshots.
- imrehg 10y ago
  Oh, this looks interesting, thanks a lot!
deleted 10y ago
[deleted]

agumonkey 10y ago

I recently put back an old P3 box (laptops don't have Floppy Disk ports) to access an HP Colorado Backup Tape. 400MB only but some interesting stuff like Word HTML page, some old games, the much nicer than I remember QBASIC.EXE[1] and most impressively TURBO.EXE (Turbo Pascal 7 IDE) which clocks at a whopping 600KB. [1] that thing had pseudo modules, a visual tracer, live indent and lint (limited but still).

agumonkey 10y ago
Couldn't resist: http://imgur.com/yKarXLK http://imgur.com/yKarXLK http://imgur.com/oT3u4fR http://imgur.com/oT3u4fR

evincarofautumn 10y ago

“Treat any object that is elevated from the floor like it will fall.” I find this an excellent metaphor for programming. In particular: low-level debugging of high-level code is inevitable, so choose your abstractions carefully.

erelde 10y ago
I want to apply that to politics/History as well. That said, this is the one sentence my 1st grade teacher always kept repeating and stuck with me ever since. (it was, at the time, about pens falling from desks)

k_sze 10y ago

Mandatory XKCD reference: https://xkcd.com/1360/ https://xkcd.com/1360/ My point is. The technical aspect of moving data around is actually not that challenging compared to the "business logic" of organizing your old files.

illumen 10y ago

cd Desktop/old/old/old/old/old/old/old/old/old/

chromakode 10y ago
Some of the images I recovered actually contained directory structures like that, which included some of the oldest files I was able to recover. This approach can work out if the filesystem eventually gets preserved!

simula67 10y ago

Did you consider using OneDrive or Google Drive ? Something like http://www.duplicati.com/ http://www.duplicati.com/

cm2187 10y ago
I'd say the only thing more volatile than a forgotten hard drive is a cloud service. They will come and go, go bust or get shut down, get depreciated and replaced. And if you forget to pay, don't count on finding your old forgotten data in 15 years.
- drdaeman 10y ago
  Glacier, Nearline or B2 are "cloud" services as well, and while they're more stable than ordinary-consumer-grade services (like Google Drive or OneDrive), the it's not like their future is set in stone (affordability, service policies, etc) I believe the way to get the best reliability is to keep multiple replicas of the archive (git-annex's great here) over storage options from as many different vendors as possible, and hope the chance of everyone going down at nearly the same time is too low. And probably schedule some periodic fully-automated checks to minimize the chance some service had degraded and had silently lost the data.
  - cm2187 10y ago
    In that respect my recommendation is to buy some of these massive Seagate archive disks. I don't know how durable they are but given you can have 8TB for less than $200, I would have one or two in independent places. But I would encrypt the data though. But then where do you store long term the key (brain is a bad choice!)? And as technologies evolve, in 5y there will probably be cheap 30TB HDD, so it will make sense to consolidated, etc
    - Intermernet 10y ago
      > where do you store long term the key I know this is clunky, but I'd print out the key (or possible laser-cut the key!) onto durable media and store it in a safe / safety deposit box etc. It's going to be annoying to manually type it in, but it's a reliable method of last-resort.

jbverschoor 10y ago

Oh yeah, ddrescue rescues my ass once. My macbook disk crashed, and my timemachine wasn't working for some time.. It was 200GB of data. What I did was connect the disk to a linux machine, and use ddrescue for a month 24/7 to try and read the bad disk. I lost 0 bytes.

Esau 10y ago

For me, the most important take away from this article is the importance of keeping files in formats that are open. On my Macbook Air, all of my personal files are in text, jpeg/png, pdf, or mp3 formats. Of course, even open formats can fall out of favor, but that doesn't seem to happen as often. I also like that you encrypted your backups, although in doing so you probably made Senator Dianne Feinstein cry.

21 10y ago

Will he in another ten years remember the password? Happend to me. Encrypted an archive, and the funny thing is I did save the password into a password manager, but I didn't label it descriptively enough, so when I tried decrypting the archive I couldn't locate the right password. I could have just tried them all, but I had a copy of the files from the archive in another place so I just deleted the encrypted archive.

cpach 10y ago
I’m a happy Arq user since a few weeks back, but this scenario scares me. I have even considered using a weak password such as ”abc123” in order to prevent this issue.
- aidos 10y ago
  Do you think you'll remember that weak password in 10 years time? Or are you thinking you'll brute force it when the time comes? In which case there's not much reason to encrypt in the first place.
  - cpach 10y ago
    I don’t know, but I definitely think it’s easier to remember that string than a 30 characters long random string. I could even use my given name to make it more memorable. To clarify: I didn’t primarily choose my backup application because it encrypts the backups but because it seemed like a reliable application with a solid user interface. (In Arq, encryption is not optional.)
    - hollander 10y ago
      You can repeat that string, or use five easy words to get to that 30 character string.
- hollander 10y ago
  Then you get an accident, and you have amnesia. How about that scenario?

geekit 10y ago

Offtopic: Just a reminder to always remove GPS info from personal photos that are to be uploaded to internet. Photo in post gives me GPS coordinates on Westview Dr in Lake Oswego, OR. If we are so concerned about our privacy or encryption, we should not make such mistakes.

sccxy 10y ago
Thanks for posting his home address. Great job (To make a point you don't have to show full coordinates)
- geekit 10y ago
  Even if I didn't, anyone can still download image and get it. Anyway, I have removed it.
chromakode 10y ago
Oops! I appreciate your bringing this to my attention. This is an unexpected downside to my custom git-based publishing scheme -- easy to leave out EXIF wiping in the asset pipeline. In the future, it's really important to attempt such notices privately first. Even if the information is out there, making that knowledge public in a popular spot greatly increases the exposure. I'm sure that with your skillset you could have a direct line to contact me in less than a minute.

maxaf 10y ago

I'd always regretted not keeping my old hard drives when my family left Moscow in 1999. There was the Robotron 1715, which was my first true taste of computing at home. A Compaq-produced 386-SX box was solely responsible for me having learned GW Basic and Pascal before I was a teen. My daughter, who's retracing my steps somewhat, would have loved to see how I had done it. I sense that seeing her dad's attempts at learning to hack might help her understand how harmful current iPad-esque walled gardens are. Not surprisingly, when I visited Moscow 15 years later, my old hardware was gone, and not a shred of knowledge remained of where it might had gone. My relatives conceded that it might have been stolen. I hope the thieving computer obsolescence club or history museum will enjoy my pieces.

apaprocki 10y ago

I went through a similar exercise but had to deal with long deprecated floppy interface tape drives. It took a number of days to source parts and trial and error to get an OS set up to read from them, but in the end it worked... and my BBS lived again inside a DOS emulator! https://twitter.com/apaprocki/status/550432891201941504 https://twitter.com/apaprocki/status/550432891201941504 In case anyone else has a stack of long forgotten various QIC formats -- don't lose hope, have patience and restore them :)

chinathrow 10y ago

For the truly paranoid, I advise to put an additional copy in cold storage at a bank vault at bank A. Print the GPG private key and passphrase out and put it in a bank vault at bank B. Handle permissions to access these vaults within your family as you like it. Bank A shouldn't be in the same region/area as bank B in case a desaster strikes and floods the vault(s). Then play that scenario through whenever you update your keys/data.

jlgaddis 10y ago

I got my first PC in 1990 at age 11, having used only Commodore computers before that. I wish I had had the foresight to preserve everything I've ever created. Cheap Internet access (i.e. a local dial-up number) didn't come to the area where I grew up (a small, midwestern town) until about 1996. From 1990-6, I was heavily involved in the BBS "scene". Being a teenager with lots of time but little money, I ran a BBS but couldn't afford to pay for all those door games. Instead, I learned to reverse engineer them and write "cracks" (binary patches, basically) to "unlock" those games. In the middle of the night, I'd dial into far away BBSes and upload them everywhere that I could. I'd love to have that source code to look at again today. I did find some of my old paper notebooks in my mother's garage a while back. These are the ones that I took to school with me. Instead of doing schoolwork, however, I'd spend my time in class writing code (on paper, by hand) and then I'd type it all in to the computer when I got home after school. It was very neat to find those and look back at code I wrote ~25 years ago. For the last few years, now that we all have a camera in our pocket at all times, I have kept backups of every picture I've taken but there are probably thousands of photos that I took before then that I'll never see again. With storage as cheap as it is, I've resolved to never again "lose" any of the photos or videos I've taken. It'll be amazing to look back at them in another ~25 years or so.

jlgaddis 10y ago

I'm aware of at least two old PCs of mine that are still in my mother's garage. One is a Pentium 100 that was primarily used as a router/firewall for my local network so there probably isn't much "data" on it. The other was slightly newer and may actually contain some of my old files. I think I'll bring them home the next time I'm there and see if I can find anything significant on them.

746F7475 10y ago

This is strange to me, I've been using computers since I can remember, but I only very late started to do anything productive with them. I've never had anything I've wanted to keep, I mean sure I now have some scripts that are valuable, but nothing old like that. It's also so strange when co-workers talk about how their kids are not just playing games but also creating mods for their games. When I was kid I just drew pictures, played with lego and played video games.

fapjacks 10y ago
Modding games is extremely easy these days. I had to walk uphill both ways through two feet of snow just to get some games to work in the early 90s. Buckets of blood, sweat and tears went into my MajorMUD scripts.
- 746F7475 10y ago
  I have no idea how hard or easy it is, but it still seems like they are doing way more advanced shit than I ever did
  - fapjacks 10y ago
    I wonder how much of this is selection bias by media. There are now twice as many human beings on this planet as when we were kids (totally making assumptions about you here, if you don't mind). That means more gifted kids, in addition to our ever more invasive media. Our various feeds are (unfortunately) full of page after page of people talking about the things they think their kids are doing. However I don't think kids today are especially special snowflakes compared to when we were kids. I'm not saying there aren't gifted kids. I just mean it's probably not right or true to say that gifted kids these days are more amazing than gifted kids of yesteryear. And it doesn't help that every parent on the planet sells their kid as a genius. Even the parents of stupid kids.

oxplot 10y ago

Am I the only one blown away by Google Nearline? From https://cloud.google.com/storage/docs/nearline https://cloud.google.com/storage/docs/nearline: > Data is available in seconds, not hours or days. With ~3 second response times and 1 cent per GB/month pricing This is in contrast to 3 hours of Amazon Glacier [1]. [1]: https://aws.amazon.com/glacier/ https://aws.amazon.com/glacier/