5 ms·

But digital copies ain't always perfect - you forget about bit rot [0] unfortunatelly. One bit flip on stream of compressed picture or archive distorts remainin

But digital copies ain't always perfect - you forget about bit rot [0] unfortunatelly. One bit flip on stream of compressed picture or archive distorts remaining result. Entire digital world is founded on analog physical world with all its imperfections. Of course there are solutions to mitigate them: checksuming, redundancy, error correction. But how often one checks checksum hash on copy? Edit: CRC doesn't count as it is too weak. I agree on open standards and deduplication, I hope content addresable archiving will get more mainstream, IPFS brings big hopes. [0] https://en.m.wikipedia.org/wiki/Data_degradation https://en.m.wikipedia.org/wiki/Data_degradation

altcognito 8y ago
> But how often one checks checksum hash on copy? Your operating system does this.
- imhoguy 8y ago
  Only if you use ECC RAM and ZFS/Btrfs/ReFS. CRC e.g. in TCP is too weak.
  - altcognito 8y ago
    Ah this is right, I was thinking of removable media, cdrom/dvd usually do this.
    - tajen 8y ago
      USBs don’t do this. My md5 checksums fail all the time for big files on USBs. And USBs dont have SMART features.
  - zamadatix 8y ago
    Yes, if one was going to reliably archive information they wouldn't use FAT32 on a single spinning disk. Does this really need to be pointed out? The point is digital gives you ways to have perfect copies, correct loss, and the ability to transfer to the next storage medium with no decay. Things analog could never achieve.
    - jacobush 8y ago
      Hybrid analog could achieve this. Film could have a picture printed on it, but instead of going for maximum resolution (down to grain level) it could have tiny tiny pixels, not visible to the eye, but readable by a computer. A kind of steganography, if you will. The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself. Such a picture could be copied, with loss, in an analog fashion, but could also be copied perfectly in a digital fashion, by a digital reader capable of reading out the steganography information. Even the analog copy could be digitally read if it was of high enough quality. Another, more straightforward, way would be to print on the film, first an analog rendition of the picture, then a digitally encoded pattern of the same image, rinse repeat. But from a future safe perspective, it could be smart to encode the data itself inside the image, as per the first paragraph. Someone copying the images could unwittingly copy the digital data too!
      - adrianN 8y ago
        You might be interested in Optar http://ronja.twibright.com/optar/ http://ronja.twibright.com/optar/
        jacobush 8y ago
        Indeed, thanks.
      - yourapostasy 8y ago
        > The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself. This was an intriguing idea, but after I did some back of the envelope rough numbers, I tentatively concluded microform might not encode densely enough to accomplish that. It already has poor fidelity characteristics for photographic data. Right off the bat you are losing lots of information during the reducing process, and the film itself does not have the kind of qualities photographers look for. Color microform is also much more expensive than black and white. So the color film works well for newsprint photos, but for archival grade purposes, we're likely better off with the original film negatives. For deep storage archives, I could see a use case for microform, and with the encoding you suggested, but not quite at one picture per frame (page). Steganography relies upon imperceptibly embedding information into the picture, so there is a very low encoding density limit because we're working with analog pictures. Optar at 600 ppi uses an entire page to encode 200 Kb, but microfilm is closer to 300 ppi, so at 100 percent density with no picture, we can expect around 100 Kb per page of Optar-encoded binary data. An average smartphone picture size these days is around 5 MB, closer to 15-50 MB for DSLR pictures. Optar-encoding the smartphone picture, we are around 50 pages before we scatter it into the picture to steganographically encode it. Already we can see the picture has to be blown up across more than 50 pages and stitched back together. At imperceptible levels of encoding density, I'm guessing we're looking at 500+ pages, maybe. A standard microfiche holds about 98 pages, so we're talking about 6'ish fiches per picture at cellphone grade, much more for DSLR grade picture encoding. For high quality archival photograph storage, I think current librarians have it right: lots of duplicates of negatives, geographically separated, then service copy negatives off those masters, digitally scan in the most-used pictures off the service copy, and let patrons use the digital copies. Migrate the digital copies until we establish a universally-accepted digital bootstrap mechanism and process for Long Now-span time scales.
deleted 8y ago
[deleted]
Koshkin 8y ago
> bit rot Isn't it specific to magnetic media only? (I don't think optical discs susceptible to bit rot. Especially M-discs which are said to last for a thousand years.)
- 0xcde4c3db 8y ago
  The general concept isn't specific to magnetic media at all, although the underlying mechanisms vary. Any EPROM/EEPROM/Flash will leak charge from its cells and eventually lose data if not periodically refreshed. Even some mask ROMs develop bad bits over time due to latent manufacturing defects. Many CD-Rs and DVD-Rs recorded ~15-20 years ago already have uncorrectable errors (due to the dye degrading?). Pressed discs are nominally immune, but may have manufacturing defects that lead to progressive failure (e.g. "laser rot" on LaserDiscs manufactured by certain plants).
  - mjevans 8y ago
    I was thinking about this the other day. Would a good solution for /really/ long term storage be a kind of WORM ROM based on fuses that could be burned to 'program' a given area? (I think some EPROM might be this, but I'm talking about more storage density than even current top end flash drives.) Obviously you'd only want to use this for the LONG term storage, but if it's cheep enough I could see it being useful in the same use cases that existing optical media were and even in roles like backup / audit logs.
    - dredmorbius 8y ago
      This is a complex problem. https://www.archives.gov/preservation/technical/storage-strategy.html https://www.archives.gov/preservation/technical/storage-stra... https://en.wikipedia.org/wiki/Digital_preservation https://en.wikipedia.org/wiki/Digital_preservation The Internet Archives preferred bmedium for book storage is ... books. https://www.theguardian.com/books/2011/aug/01/internet-archive-books-brewster-kahle https://www.theguardian.com/books/2011/aug/01/internet-archi... Quartz glass is a proposed medium, with a potential life of 300 million years. https://www.scientificamerican.com/article/data-saved-quartz-glass-might-last-300-million-years/ https://www.scientificamerican.com/article/data-saved-quartz...

altcognito 8y ago

> But how often one checks checksum hash on copy? Your operating system does this.

imhoguy 8y ago
Only if you use ECC RAM and ZFS/Btrfs/ReFS. CRC e.g. in TCP is too weak.
- altcognito 8y ago
  Ah this is right, I was thinking of removable media, cdrom/dvd usually do this.
  - tajen 8y ago
    USBs don’t do this. My md5 checksums fail all the time for big files on USBs. And USBs dont have SMART features.
- zamadatix 8y ago
  Yes, if one was going to reliably archive information they wouldn't use FAT32 on a single spinning disk. Does this really need to be pointed out? The point is digital gives you ways to have perfect copies, correct loss, and the ability to transfer to the next storage medium with no decay. Things analog could never achieve.
  - jacobush 8y ago
    Hybrid analog could achieve this. Film could have a picture printed on it, but instead of going for maximum resolution (down to grain level) it could have tiny tiny pixels, not visible to the eye, but readable by a computer. A kind of steganography, if you will. The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself. Such a picture could be copied, with loss, in an analog fashion, but could also be copied perfectly in a digital fashion, by a digital reader capable of reading out the steganography information. Even the analog copy could be digitally read if it was of high enough quality. Another, more straightforward, way would be to print on the film, first an analog rendition of the picture, then a digitally encoded pattern of the same image, rinse repeat. But from a future safe perspective, it could be smart to encode the data itself inside the image, as per the first paragraph. Someone copying the images could unwittingly copy the digital data too!
    - adrianN 8y ago
      You might be interested in Optar http://ronja.twibright.com/optar/ http://ronja.twibright.com/optar/
      - jacobush 8y ago
        Indeed, thanks.
    - yourapostasy 8y ago
      > The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself. This was an intriguing idea, but after I did some back of the envelope rough numbers, I tentatively concluded microform might not encode densely enough to accomplish that. It already has poor fidelity characteristics for photographic data. Right off the bat you are losing lots of information during the reducing process, and the film itself does not have the kind of qualities photographers look for. Color microform is also much more expensive than black and white. So the color film works well for newsprint photos, but for archival grade purposes, we're likely better off with the original film negatives. For deep storage archives, I could see a use case for microform, and with the encoding you suggested, but not quite at one picture per frame (page). Steganography relies upon imperceptibly embedding information into the picture, so there is a very low encoding density limit because we're working with analog pictures. Optar at 600 ppi uses an entire page to encode 200 Kb, but microfilm is closer to 300 ppi, so at 100 percent density with no picture, we can expect around 100 Kb per page of Optar-encoded binary data. An average smartphone picture size these days is around 5 MB, closer to 15-50 MB for DSLR pictures. Optar-encoding the smartphone picture, we are around 50 pages before we scatter it into the picture to steganographically encode it. Already we can see the picture has to be blown up across more than 50 pages and stitched back together. At imperceptible levels of encoding density, I'm guessing we're looking at 500+ pages, maybe. A standard microfiche holds about 98 pages, so we're talking about 6'ish fiches per picture at cellphone grade, much more for DSLR grade picture encoding. For high quality archival photograph storage, I think current librarians have it right: lots of duplicates of negatives, geographically separated, then service copy negatives off those masters, digitally scan in the most-used pictures off the service copy, and let patrons use the digital copies. Migrate the digital copies until we establish a universally-accepted digital bootstrap mechanism and process for Long Now-span time scales.

deleted 8y ago

[deleted]

Koshkin 8y ago

> bit rot Isn't it specific to magnetic media only? (I don't think optical discs susceptible to bit rot. Especially M-discs which are said to last for a thousand years.)

0xcde4c3db 8y ago
The general concept isn't specific to magnetic media at all, although the underlying mechanisms vary. Any EPROM/EEPROM/Flash will leak charge from its cells and eventually lose data if not periodically refreshed. Even some mask ROMs develop bad bits over time due to latent manufacturing defects. Many CD-Rs and DVD-Rs recorded ~15-20 years ago already have uncorrectable errors (due to the dye degrading?). Pressed discs are nominally immune, but may have manufacturing defects that lead to progressive failure (e.g. "laser rot" on LaserDiscs manufactured by certain plants).
- mjevans 8y ago
  I was thinking about this the other day. Would a good solution for /really/ long term storage be a kind of WORM ROM based on fuses that could be burned to 'program' a given area? (I think some EPROM might be this, but I'm talking about more storage density than even current top end flash drives.) Obviously you'd only want to use this for the LONG term storage, but if it's cheep enough I could see it being useful in the same use cases that existing optical media were and even in roles like backup / audit logs.
  - dredmorbius 8y ago
    This is a complex problem. https://www.archives.gov/preservation/technical/storage-strategy.html https://www.archives.gov/preservation/technical/storage-stra... https://en.wikipedia.org/wiki/Digital_preservation https://en.wikipedia.org/wiki/Digital_preservation The Internet Archives preferred bmedium for book storage is ... books. https://www.theguardian.com/books/2011/aug/01/internet-archive-books-brewster-kahle https://www.theguardian.com/books/2011/aug/01/internet-archi... Quartz glass is a proposed medium, with a potential life of 300 million years. https://www.scientificamerican.com/article/data-saved-quartz-glass-might-last-300-million-years/ https://www.scientificamerican.com/article/data-saved-quartz...