4 ms·

I really wish we as a developer community would better differentiate between "backup" and "preservation". They are both related to storage, but are fundamentall

I really wish we as a developer community would better differentiate between "backup" and "preservation". They are both related to storage, but are fundamentally different problems. The article touches on the desire to have photos safe for decades, but doesn't really get into the strategy necessary to make that happen. If we care about decades, storing the raw NEF or CR2 file is almost certainly the wrong approach. Those files could easily be as difficult to open in fifteen years as a ClarisWorks document is today (just to choose one example - proprietary formats for professional software from the late 90's are all pretty hard to open today). Also, important metadata is kept in the lrcat file. Unless we expect to always be able to open our copies of Lightroom, we need to migrate that data too. We used to think of "archiving" as putting something on a shelf and forgetting about it, sometimes taking environmental factors like humidity into account. With our digital data, we need a plan for periodically checking it, both for fixity and to make sure at risk file formats are migrated to current best practices. This will require greater awareness for the complexities involved (and the need for open file formats) from everyone from creatives to storage and service providers. By the way, every creative industry is struggling with this right now. Recording studios have session files created in ProTools 4. Designers have really old Quark files, even though they've been working in InDesign for the past five years. Composers have Finale files. A lot of these file formats depend on plugins to be able to display properly. I don't mean to sound discouraging, I think there's plenty of opportunities for viable businesses here. We just need to start using the right terms.

graue 13y ago
In a word, VMs. If you make sure you can open that NEF or CR2 file in a virtual machine, and the virtual machine image is in a standard, open format, you're set. You just have to worry about the image format becoming obsolete and not 10 or 20 different proprietary formats. That said, I used to be a pretty prolific hobbyist musician and I haven't pulled this off myself. I have tons of old songs I can't open because they rely on a fiddly collection of old freeware plugins for freeware music apps from 11 years ago. I also have songs I can't open because they only play in a music tracker I wrote, which is open-source, but only runs on OpenBSD. It is a tricky problem and it seems hard to tackle because the market for professional software inherently favors closed-source, which has no incentive to adopt open, non-siloed formats.
jrochkind1 13y ago
Even without getting to format obsolescence... ...do we know, when you store something in Glacier, does Amazon keep multiple copies, or just ONE on some unspecified media? Do they take a hash fingerprint, and compare it to the copy for bit rot corruption? Even without getting to format obsolescence, the strategy of just keeping the bitstream reliably and verifiably retrievable for 'decades to come' is worth discussing.
hvidgaard 13y ago
One of the only ways to do this, is having everything stored in a fully documented format. For photos, I don't think much exist except the most naive raw format (think bitmap files, just in same colour space as NEF or the like), accompanied with an XML/JSON file with the metadata and a document describing this. Once it is needed, you have to convert it. Sounds about right you could build a business on that idea - first step is to find someone willing to pay for it though.

graue 13y ago

In a word, VMs. If you make sure you can open that NEF or CR2 file in a virtual machine, and the virtual machine image is in a standard, open format, you're set. You just have to worry about the image format becoming obsolete and not 10 or 20 different proprietary formats. That said, I used to be a pretty prolific hobbyist musician and I haven't pulled this off myself. I have tons of old songs I can't open because they rely on a fiddly collection of old freeware plugins for freeware music apps from 11 years ago. I also have songs I can't open because they only play in a music tracker I wrote, which is open-source, but only runs on OpenBSD. It is a tricky problem and it seems hard to tackle because the market for professional software inherently favors closed-source, which has no incentive to adopt open, non-siloed formats.

jrochkind1 13y ago

Even without getting to format obsolescence... ...do we know, when you store something in Glacier, does Amazon keep multiple copies, or just ONE on some unspecified media? Do they take a hash fingerprint, and compare it to the copy for bit rot corruption? Even without getting to format obsolescence, the strategy of just keeping the bitstream reliably and verifiably retrievable for 'decades to come' is worth discussing.

hvidgaard 13y ago

One of the only ways to do this, is having everything stored in a fully documented format. For photos, I don't think much exist except the most naive raw format (think bitmap files, just in same colour space as NEF or the like), accompanied with an XML/JSON file with the metadata and a document describing this. Once it is needed, you have to convert it. Sounds about right you could build a business on that idea - first step is to find someone willing to pay for it though.