8 ms·

Archivists are racing to digitise 300 years of newspapers before they crumble

89 points by samclemens 11y ago

ams6110 11y ago
Hm. Yet to be proven that any digital storage will last anywhere close to that long. At best it will be a continually active process of refreshing the archive and converting the data to whatever storage media and formats are currently in use.
- bitJericho 11y ago
  That's not a problem. The problem is in the analog mediums (paper) that will always disintegrate over time. Digital media will not lose fidelity over time at all. In 10,000 years, as long as the backups were maintained, the data will be as good as the day it was made.
  - cpeterso 11y ago
    Moving the scans to new digital media and data formats is the hard part. It takes effort and money, whereas analog formats need no extra work.
    - acdha 11y ago
      This is only partially true: with online storage, it's relatively easy to make bit-for-bit identical copies even if the physical storage medium changes over the years because there's always an overlap period when a technology falls out of favor. Where it gets expensive is when you neglect to do that and then 50 years from now someone is pulling a Zip disk or LTO tape out of a box and wondering how to read it. In contrast, analog formats will always lose quality as you copy it so you have a strong incentive to make copies which will last as long as possible. If you get the right material it might be transferable in the future with no work – e.g. high-quality photographic prints on archival-quality stock – or you might end up needing to build exotic equipment which can do things like optically scan records to reconstruct an audio waveform (http://irene.lbl.gov http://irene.lbl.gov) or deal with media which has disintegrated (https://www.nedcc.org/audio-preservation/irene-blog/2014/08/12/delaminating/ https://www.nedcc.org/audio-preservation/irene-blog/2014/08/...). One look through e.g. http://britishlibrary.typepad.co.uk/collectioncare/index.html http://britishlibrary.typepad.co.uk/collectioncare/index.htm... should be enough to see limited a time period “no extra work” is valid for. The common theme for both formats is that it's critical to maintain the ability to read and make copies. Once something falls out of common usage the cost to rebuild that capacity go up dramatically because you're no longer enjoying mainstream economies of scale and the work will increasingly require skilled technicians using bespoke tools. This can be particularly bad with digital formats if the use of DRM means that few/no people are legally allowed to create tools during the period where many of the original creators are still available for consultation.
    - rwallace 11y ago
      Paying the rent on a building takes extra work. Keeping the roof in good repair so it doesn't leak takes extra work. Maintaining an atmosphere of 14% oxygen takes extra work. Just because we call it maintaining existing stuff instead of making new stuff doesn't mean it's not needed work.
  - pessimizer 11y ago
    If the backups aren't maintained, the data will be instantly, totally, and irreversibly lost. The maintenance on paper involves not leaving it in the rain or setting it on fire; both failure modes shared with digital.
    - ngoede 11y ago
      And dealing with the fact that paper is typically destructive and hard to copy.
- jon-wood 11y ago
  Digital storage probably won't last that long, but if you store it right and keep rotating media (possibly upgrading to better ones as they become available) then the data itself can last as long as someone is putting time into caring for it.
  - walterbell 11y ago
    Is there archive storage management software for offline media testing, recovery and rotation? I've seen dvdisaster and git-annex.
- walterbell 11y ago
  Fresh paper archives could be maintained in parallel with digital.
bane 11y ago
Out of curiosity, there's a huge number of newspapers already archived on microfilm (unless it's been all thrown away). I'm sure digitizing from microfilm will be a reasonable alternative.
- irv 11y ago
  it's mentioned in the article that microfilm is digitised
jedberg 11y ago
It's interesting that people are currently lamenting the fact that a huge chunk of human history will be lost because it's all digital now, while at the same time we are racing to digitize the past to prevent it from getting lost. It almost seems inevitable that most of the details of human history will be lost regardless....
- digi_owl 11y ago
  Well i think that is inherent in entropy. If i understand the physics right, any kind of order/information will eventually decay into chaos. With the ultimate of that being the heat (gradient) death of the universe.
  - andyjohnson0 11y ago
    The scond law of thermodynamics says that the total entropy of a system always increases. It doesn't prohibit local entropy decreases. So a part of a system (newspaper, archive, planet) can maintain or increase its order.
    - contingencies 11y ago
      Anything can happen. However, it will all dissolve to entropy given time. If you find a way to disprove the notion that all is impermanent, you're sitting on a potential goldmine. But ahead lies a terrible choice. If you inform the engineers, they'll potentially write you a fat check, but the world will be run by engineers, and you will be responsible. If you inform the philosophers, Buddhists, Hindus, Jains, physicists or your local psychiatrist, they'll laugh and explain why you're wrong. I suggest talking to the second bunch first, they're nicer people with better food and drink to discuss the matter over, and are more likely to be based in reality than the engineering/theoretical maths set. Personally I sometimes like to step back and view the rise of computing as a terribly mistaken and fallacious dive by humans in to a theoretically perfect world, built at great effort and expense, where the entropy of reality is fought with layers and layers of artifice. In time, it will all crumble. What are we learning out of all this effort? Surely, something that will stay with us. And yet the Eden of earth before modernity is being spoiled while we collectively swipe, at increasingly higher resolutions, speeds and brightness levels, at an ephemeral iconography of collective abstract mental masturbation. (Note to pre-empt negative responses: As stated, this is a line of thinking I enjoy taking sometimes, not a fixed opinion on reality.)
WalterBright 11y ago
The idea of concentrating all these one-of-a-kind newspapers into one building is crazy. What if it burns down? > At such low oxygen levels, the contents simply can’t go up in flames. Famous last words. > And with standards for the documentation, archiving and accessing of data – official and personal – still being thrashed out, I don't understand why this is a problem. Scan them to pdf files, and put them on web pages. Let google index them.
- jperras 11y ago
  It's a basic principle of fire safety engineering – paper requires a concentration of more than 14.1% oxygen to allow combustion to occur. Considering that paper composes the majority of the mass in that installation, a sustained hypoxic environment at 14% or below is exactly how this system should be designed. If the papers were separated into separate warehouses, they would still all have the same environmental requirements. Additionally, you require N times more budget, where N is the number of warehouses you've constructed (not to mention the difficulty in querying physically distributed warehouses for information). > I don't understand why this is a problem. Scan them to pdf files, and put them on web pages. Let google index them. Before making such broad, sweeping statements, perhaps read up a bit on the principles of information science: https://en.wikipedia.org/wiki/Information_science https://en.wikipedia.org/wiki/Information_science Edit: You are editing your comment every few minutes, so I don't know what to reply to anymore.
  - WalterBright 11y ago
    Ships have hulls to keep the water out, too, but sometimes the water gets in and they sink. I can think of dozens of ways the archive could still burn. Here's just one: large earthquake breaks open the building, cuts electric power, breaks gas lines, fire starts. Firemen are overwhelmed and give priority to saving civilians in other buildings, stacks of newspapers are at the bottom of their list. Another: Fire starts in building next door. Wind whipped flames set the archive on fire from the outside. Archive burns down with everything in it. A third: bunch of militants take over that part of town. Set fire to the archive because they are opposed to history. Not like that has never happened before, like in ISIS controlled Iraq, and the great fire of Alexandria. It's the classic eggs-in-one-basket scenario.
slyall 11y ago
A major Australian/NZ newspaper company shipped all their photos to the US for digitization. The company ( Rogers Photo Archive) doing the digitization then went bankrupt. http://www.theguardian.com/media/2015/jun/08/fairfax-media-photo-archive-stuck-in-us-warehouse-after-digitising-deal-unravels http://www.theguardian.com/media/2015/jun/08/fairfax-media-p... Especially embarrassing the the NZ branch since they got special permission from the government to ship the photos out of the country. Some of the photos ended up on Ebay.
MichaelCrawford 11y ago
I have some newspapers that my grandmother saved from world war II. Those were quite unlike todays newspapers.
p3d4nt 11y ago
Oooh gush. Digitise and immediately place behind a paywall so that the BL can sell subscriptions to personal historians more like. No kid will be able to walk into a public library in Britain and access these archives for free.

89 points by samclemens 11y ago

Hm. Yet to be proven that any digital storage will last anywhere close to that long. At best it will be a continually active process of refreshing the archive and converting the data to whatever storage media and formats are currently in use.

bitJericho 11y ago
That's not a problem. The problem is in the analog mediums (paper) that will always disintegrate over time. Digital media will not lose fidelity over time at all. In 10,000 years, as long as the backups were maintained, the data will be as good as the day it was made.
- cpeterso 11y ago
  Moving the scans to new digital media and data formats is the hard part. It takes effort and money, whereas analog formats need no extra work.
  - acdha 11y ago
    This is only partially true: with online storage, it's relatively easy to make bit-for-bit identical copies even if the physical storage medium changes over the years because there's always an overlap period when a technology falls out of favor. Where it gets expensive is when you neglect to do that and then 50 years from now someone is pulling a Zip disk or LTO tape out of a box and wondering how to read it. In contrast, analog formats will always lose quality as you copy it so you have a strong incentive to make copies which will last as long as possible. If you get the right material it might be transferable in the future with no work – e.g. high-quality photographic prints on archival-quality stock – or you might end up needing to build exotic equipment which can do things like optically scan records to reconstruct an audio waveform (http://irene.lbl.gov http://irene.lbl.gov) or deal with media which has disintegrated (https://www.nedcc.org/audio-preservation/irene-blog/2014/08/12/delaminating/ https://www.nedcc.org/audio-preservation/irene-blog/2014/08/...). One look through e.g. http://britishlibrary.typepad.co.uk/collectioncare/index.html http://britishlibrary.typepad.co.uk/collectioncare/index.htm... should be enough to see limited a time period “no extra work” is valid for. The common theme for both formats is that it's critical to maintain the ability to read and make copies. Once something falls out of common usage the cost to rebuild that capacity go up dramatically because you're no longer enjoying mainstream economies of scale and the work will increasingly require skilled technicians using bespoke tools. This can be particularly bad with digital formats if the use of DRM means that few/no people are legally allowed to create tools during the period where many of the original creators are still available for consultation.
  - rwallace 11y ago
    Paying the rent on a building takes extra work. Keeping the roof in good repair so it doesn't leak takes extra work. Maintaining an atmosphere of 14% oxygen takes extra work. Just because we call it maintaining existing stuff instead of making new stuff doesn't mean it's not needed work.
- pessimizer 11y ago
  If the backups aren't maintained, the data will be instantly, totally, and irreversibly lost. The maintenance on paper involves not leaving it in the rain or setting it on fire; both failure modes shared with digital.
  - ngoede 11y ago
    And dealing with the fact that paper is typically destructive and hard to copy.
jon-wood 11y ago
Digital storage probably won't last that long, but if you store it right and keep rotating media (possibly upgrading to better ones as they become available) then the data itself can last as long as someone is putting time into caring for it.
- walterbell 11y ago
  Is there archive storage management software for offline media testing, recovery and rotation? I've seen dvdisaster and git-annex.
walterbell 11y ago
Fresh paper archives could be maintained in parallel with digital.

bane 11y ago

Out of curiosity, there's a huge number of newspapers already archived on microfilm (unless it's been all thrown away). I'm sure digitizing from microfilm will be a reasonable alternative.

irv 11y ago
it's mentioned in the article that microfilm is digitised

jedberg 11y ago

It's interesting that people are currently lamenting the fact that a huge chunk of human history will be lost because it's all digital now, while at the same time we are racing to digitize the past to prevent it from getting lost. It almost seems inevitable that most of the details of human history will be lost regardless....

digi_owl 11y ago
Well i think that is inherent in entropy. If i understand the physics right, any kind of order/information will eventually decay into chaos. With the ultimate of that being the heat (gradient) death of the universe.
- andyjohnson0 11y ago
  The scond law of thermodynamics says that the total entropy of a system always increases. It doesn't prohibit local entropy decreases. So a part of a system (newspaper, archive, planet) can maintain or increase its order.
  - contingencies 11y ago
    Anything can happen. However, it will all dissolve to entropy given time. If you find a way to disprove the notion that all is impermanent, you're sitting on a potential goldmine. But ahead lies a terrible choice. If you inform the engineers, they'll potentially write you a fat check, but the world will be run by engineers, and you will be responsible. If you inform the philosophers, Buddhists, Hindus, Jains, physicists or your local psychiatrist, they'll laugh and explain why you're wrong. I suggest talking to the second bunch first, they're nicer people with better food and drink to discuss the matter over, and are more likely to be based in reality than the engineering/theoretical maths set. Personally I sometimes like to step back and view the rise of computing as a terribly mistaken and fallacious dive by humans in to a theoretically perfect world, built at great effort and expense, where the entropy of reality is fought with layers and layers of artifice. In time, it will all crumble. What are we learning out of all this effort? Surely, something that will stay with us. And yet the Eden of earth before modernity is being spoiled while we collectively swipe, at increasingly higher resolutions, speeds and brightness levels, at an ephemeral iconography of collective abstract mental masturbation. (Note to pre-empt negative responses: As stated, this is a line of thinking I enjoy taking sometimes, not a fixed opinion on reality.)

WalterBright 11y ago

The idea of concentrating all these one-of-a-kind newspapers into one building is crazy. What if it burns down? > At such low oxygen levels, the contents simply can’t go up in flames. Famous last words. > And with standards for the documentation, archiving and accessing of data – official and personal – still being thrashed out, I don't understand why this is a problem. Scan them to pdf files, and put them on web pages. Let google index them.

jperras 11y ago
It's a basic principle of fire safety engineering – paper requires a concentration of more than 14.1% oxygen to allow combustion to occur. Considering that paper composes the majority of the mass in that installation, a sustained hypoxic environment at 14% or below is exactly how this system should be designed. If the papers were separated into separate warehouses, they would still all have the same environmental requirements. Additionally, you require N times more budget, where N is the number of warehouses you've constructed (not to mention the difficulty in querying physically distributed warehouses for information). > I don't understand why this is a problem. Scan them to pdf files, and put them on web pages. Let google index them. Before making such broad, sweeping statements, perhaps read up a bit on the principles of information science: https://en.wikipedia.org/wiki/Information_science https://en.wikipedia.org/wiki/Information_science Edit: You are editing your comment every few minutes, so I don't know what to reply to anymore.
- WalterBright 11y ago
  Ships have hulls to keep the water out, too, but sometimes the water gets in and they sink. I can think of dozens of ways the archive could still burn. Here's just one: large earthquake breaks open the building, cuts electric power, breaks gas lines, fire starts. Firemen are overwhelmed and give priority to saving civilians in other buildings, stacks of newspapers are at the bottom of their list. Another: Fire starts in building next door. Wind whipped flames set the archive on fire from the outside. Archive burns down with everything in it. A third: bunch of militants take over that part of town. Set fire to the archive because they are opposed to history. Not like that has never happened before, like in ISIS controlled Iraq, and the great fire of Alexandria. It's the classic eggs-in-one-basket scenario.

slyall 11y ago

A major Australian/NZ newspaper company shipped all their photos to the US for digitization. The company ( Rogers Photo Archive) doing the digitization then went bankrupt. http://www.theguardian.com/media/2015/jun/08/fairfax-media-photo-archive-stuck-in-us-warehouse-after-digitising-deal-unravels http://www.theguardian.com/media/2015/jun/08/fairfax-media-p... Especially embarrassing the the NZ branch since they got special permission from the government to ship the photos out of the country. Some of the photos ended up on Ebay.

MichaelCrawford 11y ago

I have some newspapers that my grandmother saved from world war II. Those were quite unlike todays newspapers.

p3d4nt 11y ago

Oooh gush. Digitise and immediately place behind a paywall so that the BL can sell subscriptions to personal historians more like. No kid will be able to walk into a public library in Britain and access these archives for free.