7 ms·

"You can smell vinegar when you open the cans, which is one of the byproducts of the decomposition process of these films," Spriggs said. "We know that these fi

by M_Grey 10y ago

"You can smell vinegar when you open the cans, which is one of the byproducts of the decomposition process of these films," Spriggs said. "We know that these films are on the brink of decomposing to the point where they'll become useless. The data that we're collecting now must be preserved in a digital form because no matter how well you treat the films, no matter how well you preserve or store them, they will decompose. They're made out of organic material, and organic material decomposes. So this is it. We got to this project just in time to save the data." That's... incredible. Then it sounds like they used a DNN or something similar to analyze the rate of fireball expansion in the films to refine their earlier estimates of yield. What a beautifully subtle use of tech.

random3 10y ago
It makes you wonder what other valuable films are decomposing elsewhere
- M_Grey 10y ago
  Absolutely. If we had a functional government I'd say they should form a branch of the LoC to digitally preserve it all, post hast. It's hard to know what we'll be losing until it's lost otherwise.
  - pavel_lishin 10y ago
    I wonder how much the Archive team could do on their own, with donations from us and cooperation of the "better to ask forgiveness" kind from the LoC librarians.
    - M_Grey 10y ago
      That would be an excellent outcome, but given the climate around IP in this country I wouldn't want to be the one the government decided to make an example out of.
  - cwingrav 10y ago
    It's the Library of Congress's job to do just this. They are doing it. People submit huge volumes of film to them and they sort through it, digitize it, and preserve it, just like books and media.
- jwarren 10y ago
  There are untold thousands of unlabelled canisters in film archives all around the world. Some of them are undoubtedly very important, but until they're catalogued we won't even know. Unfortunately, most film archives are enormously under-funded. Here in the UK, arts funding (which is where this normally comes from) has dropped dramatically since the financial crisis. When Jeremy Hunt axed the UK Film Council[1], most regional film archives totally lost their funding. When their money disappeared, so did any chance of employing cataloguers. A proper archive will at least be storing their film in environments with the proper humidity and temperature to reduce decomposition as much as possible. The other side of it is that the more decomposition that's happened, the more time you need to invest into restoration. See the 11 year restoration of the 18 minute Voyage dans la Lune for an extreme example[2]. That kind of effort is only possible for a film of enormous historical importance and at least some commercial value. Source: My partner's a film archivist and moving image historian. [1]: https://www.theguardian.com/culture/2010/jul/26/uk-film-council-axed https://www.theguardian.com/culture/2010/jul/26/uk-film-coun... [2]: http://sensesofcinema.com/2012/feature-articles/chromatic-frankensteins-monsters-restoration-colour-and-variants-of-georges-meliess-voyage-dans-la-lune/ http://sensesofcinema.com/2012/feature-articles/chromatic-fr...
  - ghaff 10y ago
    Exactly. There is, from the perspective of being able to digitally preserve it, an essentially infinite quantity of film, audio, and paper in all forms sitting in vaults, archives, libraries, etc. around the world in various states of deterioration. We collectively can (and should) make an effort to preserve significant properties but it's almost certainly unreasonable to think that even a small portion of existing physical artifacts can be digitized.
spookyuser 10y ago
That quote makes it so clear how weak film is as an archival method. Yet I still hear people talking about how digitally shot films have an archival problem. I really can't see how any well thought out digital archive wouldn't be significantly safer than film in a can.
- jl6 10y ago
  It's a different set of risks. Digital is easier to make fireproof (store copies offsite) but vulnerable to new threats such as automated bulk deletion (e.g. a backup script gone wrong).
  - ghaff 10y ago
    It really goes beyond that. Th proverbial prints in the shoebox are subject to fire, flood, analog deteriorations, and just being lost in a move or when cleaning out the attic while digital can be backed up six ways from Sunday. Of course, in practice how many people are one head crash away from losing most of their photos stored on a consumer-grade USB drive? And of course, many people are also subject to cloud companies going out of business, computer equipment being stolen, losing access to a site online, etc. And the situation with gaining access to photos and other things if a family member dies or whatever is another whole issue.
- paulmd 10y ago
  Well, like anything it totally depends on the specifics. A modern B+W film, properly fixed and washed, is essentially forever. Metallic silver doesn't really degrade by itself (50+ year lifespan) - and you can use a toner like selenium to convert the metallic silver into silver selenide which is even more stable, easily 100-200 years+ (note that this is generally done with prints but it also works with film). Color films are generally considerably less permanent because you can't capture color using metallic silver unless you have one negative each for red, green, and blue. You can do that though, the TechniColor "3-strip" process did this and it's actually extremely stable for its time. It was also used for the color series of Magic Lantern glass plates from the Prokudin-Gorsky survey of 1905 [0]. Modern negative films are decent if stored properly, but early color negative films were pretty disastrous in terms of long-term stability. Color slide films in particular are actually quite decent as long as you store them properly (cold, and out of light). Kodachrome (K-14 process), in particular, was remarkably stable because there is actually no dye in the film itself - it's created by reaction during the (incredibly complex) processing, so (again, if properly washed) there is nothing left to continue reacting and "fade" the image out. Properly processed, Kodachrome does actually approach the stability of B+W film. The only thing that really degrades it is exposure to light (i.e. projecting it or leaving it in a window). Many early film bases were problematic though, because it was essentially a parallel effort with basic materials-science research in plastics. Nitrocellulose is super flammable and many early theaters burned down as a result, it was replaced with acetate "safety film" which tends to go vinegary, finally everyone settled on polyester. Nowadays you can have a pretty good expectation of the storage properties of our plastics. Anyway, to make a short story long - you are assuming the question by adding "a well-thought-out digital archive". You can't pick a random instance of film storage and pretend that's representative of a modern, well-thought-out effort at archival. People did all kinds of dumb things in the past - South Carolina's constitution is falling apart because they laminated them back in the 60s and now they're turning to vinegar too [1]. The equivalent in digital terms is a random paper tape from the 60s pulled from someone's attic, or a DVD that was scribbled on with an alcohol-based marker and left on a spindle somewhere. Or someone's VCR bootleg of the first rough-cuts and rotoscope layers of Star Wars (see "Deleted Magic" [2], it's amazing but wow is the quality terrible in some of them). It's always going to be possible to store some relatively stable physical media, that's the easy part. The problem with digital is you then add additional tasks of loading and interpretation on top. Even if it's stable, how do you get that Quadruplex video tape into your computer? Is there a driver that interprets the filesystem? A codec (ideally open-source)? Is there a sync signal in the medium that might degrade (like VHS)? etc etc. Yes, these are not insoluble problems, but they do add a huge threshold to get over. It might be worth doing to save Star Wars but it's not going to happen for Uncle John's photo collection. Consider something like the task of encoding a message to an alien race. You have to provide something like a Rosetta Stone to help make sense of your format. Or you could just send a phonograph which can be read by many different methods (laser measurement is the coolest). It's a pretty solid assumption that there will always be basic instruments for physical measurement like lasers and reflectometers, and that's all you need to interpret a basic medium like B+W film or a phonograph. Although I suppose how dare I assume the visual wavelength of an alien species! /s On the other hand - if you are going at it thoughtfully, i.e. periodically rotating your digital storage medium to whatever the standard is at the time, and you retain sufficient backups - the chances of there ever being a total discontinuity in migration path, or a total loss of knowledge on how to interpret the format is relatively low. The going at it thoughtfully part is the key though. Again, nobody apart from a handful of archival specialists or enthusiasts can really read a random minicomputer tape from the 80s, and that's only 30 years. I personally started noticing bit-rot on a significant fraction of DVDs that I burned within 5 years (at least 20% of discs). Now I use DVDisaster to encode extra parity/recovery data on a rolling basis - disc X contains a 20% parity file for disc X-1, with the most recent kept on my HDDs, so that I can "chain" backwards through them. Still though - imagine the kind of digital files that were made in 1995 with MOV files or whatever. Now compare them to the high-resolution 4K scans that are made from the same films. That's really another problem with digital, not just archival - the bits are what they are, you can't go back and try again when technology is better. [0.0] https://www.flickr.com/photos/prokudin-gorsky/ https://www.flickr.com/photos/prokudin-gorsky/ [0.1] https://www.loc.gov/exhibits/empire/making.html https://www.loc.gov/exhibits/empire/making.html [0.2] http://www.loc.gov/pictures/collection/prok/ http://www.loc.gov/pictures/collection/prok/ [1] http://www.npr.org/2017/02/21/515410087/an-attempt-to-save-south-carolinas-historical-documents-is-destroying-them http://www.npr.org/2017/02/21/515410087/an-attempt-to-save-s... [2] https://www.youtube.com/watch?v=f2r4Nffrc6Y https://www.youtube.com/watch?v=f2r4Nffrc6Y (there is also a Deleted Magic Revisited DVD with better quality and some more footage - finding it is left as an exercise to the reader)
DelTaco 10y ago
Fun fact, SSDs deteriorate faster than HDDs, which deteriorate faster than tape.
JustSomeNobody 10y ago
Reminds me of a story I remember hearing long ago about Tom Scholz of Boston trying to save their old tapes and how fragile they were.
jabl 10y ago
> Then it sounds like they used a DNN or something similar to analyze the rate of fireball expansion in the films to refine their earlier estimates of yield. Back in the day, some rather simple models were used to estimate yield based on photographs, see https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculating_yields_and_controversy https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculati... It's actually quite amazing that even with such a simple model, G.I. Taylor was able to estimate yield's to within 10% of the official figure for the Trinity device. Just because DNN's are terribly trendy doesn't mean that every image analysis task is done with them. :) I think that at least with such a crude model as the Taylor one, the model itself will introduce errors that DNN's per se won't fix. DNN's might help the manual drudgery of analyzing each frame of the videos manually though..

by M_Grey 10y ago

random3 10y ago

It makes you wonder what other valuable films are decomposing elsewhere

M_Grey 10y ago
Absolutely. If we had a functional government I'd say they should form a branch of the LoC to digitally preserve it all, post hast. It's hard to know what we'll be losing until it's lost otherwise.
- pavel_lishin 10y ago
  I wonder how much the Archive team could do on their own, with donations from us and cooperation of the "better to ask forgiveness" kind from the LoC librarians.
  - M_Grey 10y ago
    That would be an excellent outcome, but given the climate around IP in this country I wouldn't want to be the one the government decided to make an example out of.
- cwingrav 10y ago
  It's the Library of Congress's job to do just this. They are doing it. People submit huge volumes of film to them and they sort through it, digitize it, and preserve it, just like books and media.
jwarren 10y ago
There are untold thousands of unlabelled canisters in film archives all around the world. Some of them are undoubtedly very important, but until they're catalogued we won't even know. Unfortunately, most film archives are enormously under-funded. Here in the UK, arts funding (which is where this normally comes from) has dropped dramatically since the financial crisis. When Jeremy Hunt axed the UK Film Council[1], most regional film archives totally lost their funding. When their money disappeared, so did any chance of employing cataloguers. A proper archive will at least be storing their film in environments with the proper humidity and temperature to reduce decomposition as much as possible. The other side of it is that the more decomposition that's happened, the more time you need to invest into restoration. See the 11 year restoration of the 18 minute Voyage dans la Lune for an extreme example[2]. That kind of effort is only possible for a film of enormous historical importance and at least some commercial value. Source: My partner's a film archivist and moving image historian. [1]: https://www.theguardian.com/culture/2010/jul/26/uk-film-council-axed https://www.theguardian.com/culture/2010/jul/26/uk-film-coun... [2]: http://sensesofcinema.com/2012/feature-articles/chromatic-frankensteins-monsters-restoration-colour-and-variants-of-georges-meliess-voyage-dans-la-lune/ http://sensesofcinema.com/2012/feature-articles/chromatic-fr...
- ghaff 10y ago
  Exactly. There is, from the perspective of being able to digitally preserve it, an essentially infinite quantity of film, audio, and paper in all forms sitting in vaults, archives, libraries, etc. around the world in various states of deterioration. We collectively can (and should) make an effort to preserve significant properties but it's almost certainly unreasonable to think that even a small portion of existing physical artifacts can be digitized.

spookyuser 10y ago

That quote makes it so clear how weak film is as an archival method. Yet I still hear people talking about how digitally shot films have an archival problem. I really can't see how any well thought out digital archive wouldn't be significantly safer than film in a can.

jl6 10y ago
It's a different set of risks. Digital is easier to make fireproof (store copies offsite) but vulnerable to new threats such as automated bulk deletion (e.g. a backup script gone wrong).
- ghaff 10y ago
  It really goes beyond that. Th proverbial prints in the shoebox are subject to fire, flood, analog deteriorations, and just being lost in a move or when cleaning out the attic while digital can be backed up six ways from Sunday. Of course, in practice how many people are one head crash away from losing most of their photos stored on a consumer-grade USB drive? And of course, many people are also subject to cloud companies going out of business, computer equipment being stolen, losing access to a site online, etc. And the situation with gaining access to photos and other things if a family member dies or whatever is another whole issue.
paulmd 10y ago
Well, like anything it totally depends on the specifics. A modern B+W film, properly fixed and washed, is essentially forever. Metallic silver doesn't really degrade by itself (50+ year lifespan) - and you can use a toner like selenium to convert the metallic silver into silver selenide which is even more stable, easily 100-200 years+ (note that this is generally done with prints but it also works with film). Color films are generally considerably less permanent because you can't capture color using metallic silver unless you have one negative each for red, green, and blue. You can do that though, the TechniColor "3-strip" process did this and it's actually extremely stable for its time. It was also used for the color series of Magic Lantern glass plates from the Prokudin-Gorsky survey of 1905 [0]. Modern negative films are decent if stored properly, but early color negative films were pretty disastrous in terms of long-term stability. Color slide films in particular are actually quite decent as long as you store them properly (cold, and out of light). Kodachrome (K-14 process), in particular, was remarkably stable because there is actually no dye in the film itself - it's created by reaction during the (incredibly complex) processing, so (again, if properly washed) there is nothing left to continue reacting and "fade" the image out. Properly processed, Kodachrome does actually approach the stability of B+W film. The only thing that really degrades it is exposure to light (i.e. projecting it or leaving it in a window). Many early film bases were problematic though, because it was essentially a parallel effort with basic materials-science research in plastics. Nitrocellulose is super flammable and many early theaters burned down as a result, it was replaced with acetate "safety film" which tends to go vinegary, finally everyone settled on polyester. Nowadays you can have a pretty good expectation of the storage properties of our plastics. Anyway, to make a short story long - you are assuming the question by adding "a well-thought-out digital archive". You can't pick a random instance of film storage and pretend that's representative of a modern, well-thought-out effort at archival. People did all kinds of dumb things in the past - South Carolina's constitution is falling apart because they laminated them back in the 60s and now they're turning to vinegar too [1]. The equivalent in digital terms is a random paper tape from the 60s pulled from someone's attic, or a DVD that was scribbled on with an alcohol-based marker and left on a spindle somewhere. Or someone's VCR bootleg of the first rough-cuts and rotoscope layers of Star Wars (see "Deleted Magic" [2], it's amazing but wow is the quality terrible in some of them). It's always going to be possible to store some relatively stable physical media, that's the easy part. The problem with digital is you then add additional tasks of loading and interpretation on top. Even if it's stable, how do you get that Quadruplex video tape into your computer? Is there a driver that interprets the filesystem? A codec (ideally open-source)? Is there a sync signal in the medium that might degrade (like VHS)? etc etc. Yes, these are not insoluble problems, but they do add a huge threshold to get over. It might be worth doing to save Star Wars but it's not going to happen for Uncle John's photo collection. Consider something like the task of encoding a message to an alien race. You have to provide something like a Rosetta Stone to help make sense of your format. Or you could just send a phonograph which can be read by many different methods (laser measurement is the coolest). It's a pretty solid assumption that there will always be basic instruments for physical measurement like lasers and reflectometers, and that's all you need to interpret a basic medium like B+W film or a phonograph. Although I suppose how dare I assume the visual wavelength of an alien species! /s On the other hand - if you are going at it thoughtfully, i.e. periodically rotating your digital storage medium to whatever the standard is at the time, and you retain sufficient backups - the chances of there ever being a total discontinuity in migration path, or a total loss of knowledge on how to interpret the format is relatively low. The going at it thoughtfully part is the key though. Again, nobody apart from a handful of archival specialists or enthusiasts can really read a random minicomputer tape from the 80s, and that's only 30 years. I personally started noticing bit-rot on a significant fraction of DVDs that I burned within 5 years (at least 20% of discs). Now I use DVDisaster to encode extra parity/recovery data on a rolling basis - disc X contains a 20% parity file for disc X-1, with the most recent kept on my HDDs, so that I can "chain" backwards through them. Still though - imagine the kind of digital files that were made in 1995 with MOV files or whatever. Now compare them to the high-resolution 4K scans that are made from the same films. That's really another problem with digital, not just archival - the bits are what they are, you can't go back and try again when technology is better. [0.0] https://www.flickr.com/photos/prokudin-gorsky/ https://www.flickr.com/photos/prokudin-gorsky/ [0.1] https://www.loc.gov/exhibits/empire/making.html https://www.loc.gov/exhibits/empire/making.html [0.2] http://www.loc.gov/pictures/collection/prok/ http://www.loc.gov/pictures/collection/prok/ [1] http://www.npr.org/2017/02/21/515410087/an-attempt-to-save-south-carolinas-historical-documents-is-destroying-them http://www.npr.org/2017/02/21/515410087/an-attempt-to-save-s... [2] https://www.youtube.com/watch?v=f2r4Nffrc6Y https://www.youtube.com/watch?v=f2r4Nffrc6Y (there is also a Deleted Magic Revisited DVD with better quality and some more footage - finding it is left as an exercise to the reader)

DelTaco 10y ago

Fun fact, SSDs deteriorate faster than HDDs, which deteriorate faster than tape.

JustSomeNobody 10y ago

Reminds me of a story I remember hearing long ago about Tom Scholz of Boston trying to save their old tapes and how fragile they were.

jabl 10y ago

> Then it sounds like they used a DNN or something similar to analyze the rate of fireball expansion in the films to refine their earlier estimates of yield. Back in the day, some rather simple models were used to estimate yield based on photographs, see https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculating_yields_and_controversy https://en.wikipedia.org/wiki/Nuclear_weapon_yield#Calculati... It's actually quite amazing that even with such a simple model, G.I. Taylor was able to estimate yield's to within 10% of the official figure for the Trinity device. Just because DNN's are terribly trendy doesn't mean that every image analysis task is done with them. :) I think that at least with such a crude model as the Taylor one, the model itself will introduce errors that DNN's per se won't fix. DNN's might help the manual drudgery of analyzing each frame of the videos manually though..