5 ms·

> The Chief Data Officer (CDO) will "(1) be responsible for lifecycle data management" I am very interested in what this type of lifecycle might look like cons

by ajr0 8y ago

> The Chief Data Officer (CDO) will "(1) be responsible for lifecycle data management" I am very interested in what this type of lifecycle might look like considering most data I feel should be kept forever. I wonder how a lifecycle might collide with the challenges that bit rot[0] are dealing with. [0] https://www.theguardian.com/technology/2015/feb/13/what-is-bit-rot-and-is-vint-cerf-right-to-be-worried https://www.theguardian.com/technology/2015/feb/13/what-is-b...

ocdtrekkie 8y ago
This is already getting really exciting. Some government entities (including lower levels like local, county, and state governments) have moved to digitizing their old paper and microfilm records. But if they're expected to maintain many types of records essentially forever, it places a constant burden to continue to update and migrate data in perpetuity, whereas paper or microfilm can sit in a box in a closet for decades. For the most part, common file formats like PDFs, JPGs, and TIFs are likely to be understood for a very, very long time, but you don't just have file storage, you have systems to manage, index, and find those files, and those systems are likely to need constant maintenance.
- talmand 8y ago
  It's not like paper in boxes don't have their own maintenance costs.
  - fixermark 8y ago
    But different and arguably on-average cheaper costs. Paper can rot if it's not kept climate-controlled, but not nearly as fast as bits stored to non-volatile media if the power goes out and there isn't a hot backup.
    - pbhjpbhj 8y ago
      You can keep a whole warehouse worth of paper files on one drive though. 1TB is like 500M pages of text (1000 pallets).
      - ocdtrekkie 8y ago
        Government records aren't "text" though. They're documents, which need to be preserved with any margin notes, markings, signatures, etc. intact. So if you're storing government records, you're storing images. Physical storage space definitely does have a cost, but so does software licensing, hardware maintenance contracts, etc. And a lot of government organizations have physical space in abundance, but IT resources, less so.
        gaius 8y ago
        Don’t know why this is downvoted. When a bank stores a record of a cheque it’s a scanned image for that very reason.
        pbhjpbhj 8y ago
        Oh, for sure, imagine trying to warehouse [images of] every submitted form in a condition that can be read - so, climate controlled at least.
- deleted 8y ago
  [deleted]
- tracker1 8y ago
  I would think that outsourcing this to Amazon and Azure for redundant copies would be a sufficient start. AWS S3 and Azure Blob Storage would go a long way. Structured directories, document metadata complements and having hot indexes for said metadata in a database would be a very good start. A lot of this type of work is already being/been done.
  - ForHackernews 8y ago
    I mean, sure, until Amazon and Azure decide to delete it all [0] to save money, or those companies go bankrupt and shut off their servers. None of these big cloud providers is even 50 years old. "Forever" is a very long time indeed. There's a reason the UK still keeps some records on scrolls of velum: https://www.bbc.co.uk/news/magazine-35569281 https://www.bbc.co.uk/news/magazine-35569281 [0] - https://news.softpedia.com/news/All-Videos-on-Google-Video-Will-Be-Deleted-Next-Month-195479.shtml https://news.softpedia.com/news/All-Videos-on-Google-Video-W...
    - tracker1 8y ago
      I doubt they'd both close shop at the same time... leaving enough time to shift the data to a new second source.
    - tracker1 8y ago
      As an aside, nothing is forever... there are a lot of points in history where records are gone for ever due to fire.
      - ForHackernews 8y ago
        Sure. If I were going for "forever" storage, I would use flame-resistant clay tablets https://www.theatlantic.com/technology/archive/2017/01/human-knowledge-salt-mine/512552/ https://www.theatlantic.com/technology/archive/2017/01/human... or perhaps a laser-etched nickel disc: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ But nothing digital, that's for sure.
      - 8y ago
- swebs 8y ago
  I've seen Blu-ray disc manufacturers claim lifespans of over 100 years with capacities of 100 GB per disc. An entire warehouse of paper and microfilm documents would be able to fit in a shoebox.
  - ocdtrekkie 8y ago
    Note that most record digitization projects are likely looking at live storage (online disk arrays), as it allows constant access to said records. Also note that the discs may last 100 years but having disc readers which can read them may not, and one would need to load all the discs to do a media conversion.

by ajr0 8y ago

ocdtrekkie 8y ago

This is already getting really exciting. Some government entities (including lower levels like local, county, and state governments) have moved to digitizing their old paper and microfilm records. But if they're expected to maintain many types of records essentially forever, it places a constant burden to continue to update and migrate data in perpetuity, whereas paper or microfilm can sit in a box in a closet for decades. For the most part, common file formats like PDFs, JPGs, and TIFs are likely to be understood for a very, very long time, but you don't just have file storage, you have systems to manage, index, and find those files, and those systems are likely to need constant maintenance.

talmand 8y ago
It's not like paper in boxes don't have their own maintenance costs.
- fixermark 8y ago
  But different and arguably on-average cheaper costs. Paper can rot if it's not kept climate-controlled, but not nearly as fast as bits stored to non-volatile media if the power goes out and there isn't a hot backup.
  - pbhjpbhj 8y ago
    You can keep a whole warehouse worth of paper files on one drive though. 1TB is like 500M pages of text (1000 pallets).
    - ocdtrekkie 8y ago
      Government records aren't "text" though. They're documents, which need to be preserved with any margin notes, markings, signatures, etc. intact. So if you're storing government records, you're storing images. Physical storage space definitely does have a cost, but so does software licensing, hardware maintenance contracts, etc. And a lot of government organizations have physical space in abundance, but IT resources, less so.
      - gaius 8y ago
        Don’t know why this is downvoted. When a bank stores a record of a cheque it’s a scanned image for that very reason.
      - pbhjpbhj 8y ago
        Oh, for sure, imagine trying to warehouse [images of] every submitted form in a condition that can be read - so, climate controlled at least.
deleted 8y ago
[deleted]
tracker1 8y ago
I would think that outsourcing this to Amazon and Azure for redundant copies would be a sufficient start. AWS S3 and Azure Blob Storage would go a long way. Structured directories, document metadata complements and having hot indexes for said metadata in a database would be a very good start. A lot of this type of work is already being/been done.
- ForHackernews 8y ago
  I mean, sure, until Amazon and Azure decide to delete it all [0] to save money, or those companies go bankrupt and shut off their servers. None of these big cloud providers is even 50 years old. "Forever" is a very long time indeed. There's a reason the UK still keeps some records on scrolls of velum: https://www.bbc.co.uk/news/magazine-35569281 https://www.bbc.co.uk/news/magazine-35569281 [0] - https://news.softpedia.com/news/All-Videos-on-Google-Video-Will-Be-Deleted-Next-Month-195479.shtml https://news.softpedia.com/news/All-Videos-on-Google-Video-W...
  - tracker1 8y ago
    I doubt they'd both close shop at the same time... leaving enough time to shift the data to a new second source.
  - tracker1 8y ago
    As an aside, nothing is forever... there are a lot of points in history where records are gone for ever due to fire.
    - ForHackernews 8y ago
      Sure. If I were going for "forever" storage, I would use flame-resistant clay tablets https://www.theatlantic.com/technology/archive/2017/01/human-knowledge-salt-mine/512552/ https://www.theatlantic.com/technology/archive/2017/01/human... or perhaps a laser-etched nickel disc: http://rosettaproject.org/disk/concept/ http://rosettaproject.org/disk/concept/ But nothing digital, that's for sure.
    - 8y ago
swebs 8y ago
I've seen Blu-ray disc manufacturers claim lifespans of over 100 years with capacities of 100 GB per disc. An entire warehouse of paper and microfilm documents would be able to fit in a shoebox.
- ocdtrekkie 8y ago
  Note that most record digitization projects are likely looking at live storage (online disk arrays), as it allows constant access to said records. Also note that the discs may last 100 years but having disc readers which can read them may not, and one would need to load all the discs to do a media conversion.