4 ms·
Direct link to the Github repo: https://github.com/usnationalarchives/digital-preservation https://github.com/usnationalarchives/digital-preservation A quick s
by kibwen 4y ago
Direct link to the Github repo: https://github.com/usnationalarchives/digital-preservation https://github.com/usnationalarchives/digital-preservation
A quick search doesn't reveal any hits for "compression", I wonder what formats the National Archives uses themselves. Some compression formats include extra metadata for fancy error-correction, but I'm not qualified to evaluate how useful they are or what advanced settings to employ.
- mjevans 4y agoAny compressor should have at least free access to source code, and also be 'restore' tested against a specific version artifact (binary) of that codebase. Depending on the source data, I'm personally comfortable with Zstandard, 7zip / xz, or even gzip depending on a mix of how conservative the use case and how effectively each compresses that data structure.
- duskwuff 4y agoI like zstd as a format, but it's too new for archival use -- it's only been available for five years or so. gzip is easily the winner; while it isn't the best compressor available, it's been around for the last 30 years, and is likely to still be in use another 30 years from now.
- mjevans 4y agoThere's also the option to compress with one format now, and during routine data storage maintenance to use newer better storage layers, like compression. What's most important are integrity checks, redundancy / duplication, and the ability to repair to the source data stream.
- jmillikin 4y agoTypes of data are grouped by category, which have their own documents with recommended file encodings (including compression). For example the document for digital audio[0] has FLAC as a "preferred" format and MP3 as "acceptable". I didn't see any discussion of selecting formats based on recovery of original data from corrupted storage, and given the context I doubt that was given much consideration. When doing data archival it's better to have replication and checksums, then if any file is corrupt it's recovered from replica rather than trying to do a partial recovery. [0] https://github.com/usnationalarchives/digital-preservation/blob/master/Digital_Audio_Formats/NARA_PreservationActionPlan_DigitalAudio_20220714.pdf https://github.com/usnationalarchives/digital-preservation/b...