12 ms·

Xz format considered inadequate for long-term archiving (2016)

248 points by goranmoomin 4y ago

EdSchouten 4y ago
The vast majority of the discussion is around xz’s inability of dealing with corrupted data. That said, couldn’t you argue that that needs to be solved at a lower level (storage, transport)? I’m not convinced the compression algorithm is the right place to tackle this. Just use a file system that does proper integrity checking/resilvering. Also use TLS to transfer data over the network.
- riffraff 4y ago
  The article is about usage of xz for long term archival, so transport is not relevant, the concern seems to be bitrot and forward compatibility. Storage with integrity checking would be the solution to bitrot, but TFA also seems concerned with "how do you unarchive/recover a random file you found?" which seems a somewhat valid concern. And xz does have support for integrity checking, so it seems reasonable to have a discussion on whether that is a good support, rather than on whether it should be there at all.
  - pmoriarty 4y ago
    > xz does have support for integrity checking Archives need not only a way to check their integrity but also error correction, which xz does not have. However, you can easily combine xz with par2, which does provide error correction.
- smaudet 4y ago
  > couldn’t you argue that that needs to be solved at a lower level Self-referentially incorruptible data is the standard to beat here, moving the concern to a different layer doesn't increase the efficiency or integrity of the data itself. It is arguably less efficient, as you now rely on some lower layer of protection in addition to whatever is built into the standard itself. It is less flexible - a properly protected archive format could be scrawled on to the side of a hill, or more reasonably onto an archive medium (BD-disk), and should be able to survive any file-system change, upgrade, etc. Self-repairing hard drives with multiple redundancies are nice, but not cheap, and not wide-spread. It also does nothing for actually protecting the data - I don't care how advanced the lower level storage format is, if you overwrite data e.g. with random zeros (conceivable due a badly behaving program with too much memory access, e.g. a misbehaving virus, or a bad program or kernal driver someone ran with root access, also conceivable due to EM interference or solar radiation causing a program to misbehave), the file system will dutifully overwrite the correct data with incorrect data, including updating whatever relevant checks it has for the file. The only way around this is to maintain a historical archive of all writes ever made, and that should be evidently both absurdly impractical (how do you maintain the integrity of this archive? With a self-referentially incorruptible data archive perhaps?) and expensive. Compared to a single file, which can be backed up, agnostic to the filesystem/hardware/transport/major world-ending events, which can be simply read/recovered, far into the future. There's a pretty clear winner here.
  - EdSchouten 4y ago
    > Self-referentially incorruptible data is the standard to beat here, moving the concern to a different layer doesn't increase the efficiency or integrity of the data itself. I respectfully disagree. By putting it in the layer below, there is the ability to do repairs. For example, consider storing XZ files on a Ceph storage cluster. Ceph supports Reed-Solomon coding. This means that if data corruption occurs, Ceph is capable of automatically repairing the data corruption by recomputing the original file and writing it back to disk once more. Even if XZ were able to recover from some forms of data corruption, is it realistic that such repairs propagate back to the underlying data store? Likely not.
    - smaudet 4y ago
      You are thinking in the wrong direction, I tried to explain but maybe I can be clearer: If you can't read the data in question though, you cannot do the repairs, it doesn't matter if you do Reed-Solomon coding or not. You are thinking about coding for the underlying hardware, which is what the data corruption you are talking about is designed to fix - it does not solve the problem for writes coming from above. To do that, you actually have to decode the data in question and perform a reed-solomon encoding on the actual file inside of the archive, and this only gets worse e.g. if you have nested archives. If the data is self-referentially repairable, however, it doens't matter if the file gets overwritten with e.g. a cat gif, the format will work around that. The filesystem on the other hand will have written the cat gif to the file and updated the Reed Solomon encoding for your file, assuming (incorrectly) the file writes were valid. I suppose you could mandate that for any file to be written to your filesystem it must first be completely decompressed, and then store some encoding information alongside the archive, but this would be inefficient to the extreme, since merely copying a file onto the system would mean you have to decompress the file and then checksum it. At any rate, even if you did decompress the file in question, you have failed to separate the layers like you want to, since now you have mandated the XZ and LZMA algorithms also be baked directly into the filesystem itself. Better not to needlessly couple the filesystem to some compression algorithm, let the compression system handle its own error correction.
      - 4y ago
metadat 4y ago
Correct, xz is no longer particularly useful, mostly annoying. For read-only long-term filesystem-like archives, use squashfs with whatever compression option is convenient. The cool thing about squashfs is you can efficiently list the contents and extract single files. This is why it's an ultimate best option for long-term filesystem-esque archives. https://en.m.wikipedia.org/wiki/SquashFS https://en.m.wikipedia.org/wiki/SquashFS For everything else, or if you want (very) fast compression/decompression and/or general high-ratio compression, use .zstd. https://github.com/facebook/zstd https://github.com/facebook/zstd I've used this combination professionally to great effect. You're welcome :)
- ars 4y ago
  zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.
  - metadat 4y ago
    Do you have any data to support this claim? In my experience, zstd is way better in every way compared to gzip. Additionally, xz is good compression but horribly crazy slow to decompress. Xz also only operates on one single file at a time, which is annoying.
    - jmillikin 4y ago
      I just did a quick test with a copy of GNU M4, which is reasonably representative of a source code archive. $ time xz -9k m4-1.4.19.tar real 0m2.928s user 0m2.871s sys 0m0.056s $ time zstd -19 m4-1.4.19.tar real 0m3.411s user 0m3.380s sys 0m0.032s $ ls -l m4-1.4.19.tar* -rw-rw-r-- 1 john john 14837760 Jul 24 14:40 m4-1.4.19.tar -rw-rw-r-- 1 john john 1674612 Jul 24 14:40 m4-1.4.19.tar.xz -rw-rw-r-- 1 john john 1726155 Jul 24 14:40 m4-1.4.19.tar.zst In this test, XZ was both faster and had better compression than Zstd.
      - metadat 4y ago
        Howdy jmillikin. That is a very small file. See: https://news.ycombinator.com/item?id=25455846#:~:text=xz%20on%20highest%20(normal)%20compression,speeds%20regardless%20of%20compression%20ratio https://news.ycombinator.com/item?id=25455846#:~:text=xz%20o.... > xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio
deleted 4y ago
[deleted]
stop50 4y ago
Interesting, i used xz till today for compression, but i think i will use gzip and zstd ffrom now on.
- ars 4y ago
  I'll continue using xz - the compression ratio is far better than gzip or zstd. Maybe there will be some new future format to switch to, but zstd is not it.
  - stop50 4y ago
    zstd got me already better results, but not all machines i am administering have it installed or even available.
  - petre 4y ago
    Yup. To my surprise xz -1 is faster and compresses better than gzip -6, the default setting. Guess what I'm using now? I'll use zstd when the GNU tar in my distribution supports it directly without resorting to pipes and shell magic. It would be nice if one could specify the compression level on the GNU tar command line. BSD tar already does all of the above.
  - assttoasstmgr 4y ago
    The rest of the sane world will continue to use gzip because the marginal compression savings is not worth the compatibility issues. I have lost count of the number of times I have urgently needed to extract an xz archive on a system only to find out xz isn't available. Most notably Solaris and embedded systems.
    - kouteiheika 4y ago
      > The rest of the sane world will continue to use gzip because the marginal compression savings is not worth the compatibility issues. It is definitely worth it to use a modern compression format. I regularly take backups of one service I run. Originally I used gzip, and then on someone's recommendation I tried zstd. Here are the results: gzip compression time: 204s zstd compression time: 19.1s gzip decompression time: 28.2s zstd decompression time: 6.4s gzip compressed size: 3.0GB zstd compressed size: 1.7GB I ain't ever going back to gzip, especially when it wastes so much disk space and CPU cycles. > sane world [..] Solaris ...I think we're living in a different world. (:
saurik 4y ago
It was really disappointing when dpkg actively deprecated support (which I feel they should never do) for lzma format archives and went all-in on xz. The decompressor now needs the annoying flexibility mentioned in this article and the only benefit of the format--the ability to do random access on the file--is almost entirely defeated by dpkg using it to compress a tar file (which barely supports any form of accelerated access even when uncompressed; like the best you can do is kind of attempt to skip through file headers, which only helps if the files in the archive are large enough) and, to add insult to injury, the files are now all slightly larger to account for the extra headers :/. Regardless, this is a pretty old article and if you search for it you will find a number of discussions that have already happened about it that all have a bunch of comments. https://news.ycombinator.com/item?id=20103255 https://news.ycombinator.com/item?id=20103255 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=12768425
- jks 4y ago
  pixz (https://github.com/vasi/pixz https://github.com/vasi/pixz) is a nice parallel xz that additionally creates an index of tar files so you can decompress individual files. I wonder if dpkg could be extended to do something similar.
jmillikin 4y ago
The article spends a lot of time discussing XZ's behavior when reading corrupt archives, but in practice this is not a case that will ever happen. Say you have a file `m4-1.4.19.tar.xz` in your source archive collection. The version I just downloaded from ftp.gnu.org has the SHA-256 checksum `63aede5c6d33b6d9b13511cd0be2cac046f2e70fd0a07aa9573a04a82783af96`. The GNU site doesn't have checksums for archives, but it does have a PGP signature file, so it's possible to (1) verify the archive signature and (2) store the checksum of the file. If that file is later corrupted, the corruption will be detected via the SHA-256 checksum. There's no reason to worry about whether the file itself has an embedded CRC-whatever, or the behavior with regards to variable-length integer fields. If a bit gets flipped then the checksum won't match, and that copy of the file will be discarded (= replaced with a fresh copy from replicated storage) before it ever hits the XZ code. If this is how the lzip format was designed -- optimized for a use case that doesn't exist -- it's no wonder it sees basically no adoption in favor of more pragmatic formats like XZ or Zstd.
- thrdbndndn 4y ago
  I have no idea why you think that use case does not exist. Your whole idea about archive seems to be it is to ensure a blob doesn't change (same hash). But that's far from the only use of archive. (Hell, even with that, you are assuming you know the correct hash of the file to begin with, which isn't guaranteed.) "Repairing" corrupt archives, as in to get as much as usable data from that archive is a pretty useful thing and I have done it multiple times. For example, an archive can have hundreds of files inside and if you can recover any of them that's better than nothing. It is also one of the reason I still use WinRAR occasionally due to its great recovery record (RR) feature. >replaced with a fresh copy from replicated storage Lots of times you don't have other copy.
  - jmillikin 4y ago
    The process of long-term archival starts with replication. A common approach is two local copies on separate physical media, and one remote copy in a cloud storage service with add-only permissions. This protects against hardware failure, bad software (accidental deletion, malware), natural disasters (flood, fire) and other 99th-percentile disaster conditions. The cloud storage providers will have their own level of replication (AWS S3 has a 99.999999999% durability SLA). If you have only one copy of some important file and you discover it no longer matches the stored checksum, then that's not a question of archival, but of data recovery. There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors.
gfody 4y ago
about 4 years ago I had to choose a compression format for streaming database backups, and so I compared every option supported by 7z and it was xz that was the best compromise between performance and compression ratio
haberman 4y ago
Folks seem to be comparing xz to zstd, but if I am understanding correctly the true competitor to xz is the article author’s “lzip” format, which uses the same LZMA compression as xz but with a much better designed container format (at least according to the author).
- exyi 4y ago
  I'd not say it's necessarily better designed - it's just simpler. Few bytes of some headers, LZMA compressed data, checksum, done. No support for seeks and stuff like that
- loeg 4y ago
  Emphasis on “according to the author,” yeah. And the concerns are silly — FEC can and should be added outside the compressed data (e.g., with par2).
  - brnt 4y ago
    But why outside the compression container? I'd love to have format that let me repair their contents up to the number of parity bits it stored.
    - joveian 4y ago
      I agree that a single file is super convenient. It looks like the in-progress par3 will support that (par3 inside zip, I'd add an mtree file as well for metadata if needed): https://parchive.github.io/doc/Parity_Volume_Set_Specification_v3.0.html https://parchive.github.io/doc/Parity_Volume_Set_Specificati... I don't use par2 right now for backups due to the extra hastle, although I might use par3. It might end up a waste of space, though, since I haven't had an issue with corrupted files in at least 20 years (would be great for optical media that can get scratched). For the longest term archiving I'm not sure any kind of compression is a good idea, I'd think along the lines of uncompressed copy + parity info in a simple container where all files are contiguous (maybe a raw zip file without compression).
latchkey 4y ago
I routinely distribute a ~5meg .xz (~20meg uncompressed) to 20k+ servers across multiple physical data centers on a regular basis. Haven't seen a single failure. It ends up being 1.85megs smaller than the tgz version. Unless someone comes up with a better solution (ie: smaller), I probably won't change that any time soon.
- smaudet 4y ago
  What about lzip? The author claims the compression size is comparable or better.
  - latchkey 4y ago
    I haven't tried that one yet. I'll play around and see what I can find out. 7zip didn't perform as well. tar c -C ./build $(BINARY) | gzip -9 - > $(PKG_NAME_GZ) tar c -C ./build $(BINARY) | xz -z -9e - > $(PKG_NAME)
  - jraph 4y ago
    It should, if the compression algorithm is actually the same, just the format specification that differs.
  - exyi 4y ago
    It will be the same, it also just uses LZMA. It has less headers, but that's not going to make much of a difference on 5M file
    - latchkey 4y ago
      Compressed to 5mb. Original is about 20.
- jraph 4y ago
  You are not using for long term archival though, it seems you are using it for deployment.
  - 4y ago
waynesonfire 4y ago
Wow xz just got owned. How do you recover from this.
- jraph 4y ago
  This article is not new, this is the second time I encounter it, it seems it has been published in 2016. Most users of xz probably won't read it and will keep using it. And it's probably fine for most use cases, though lzip would most likely be better in every scenario if available.
poser-boy 4y ago
I'd recommend checking out zpaq[1], it purposed for backups, and has great compression (even on low setting) for large 100GB+ file collections. However for smaller stuff I use zstd at level 22 in a tar for most things since it's much faster, though a little heavier. [1] http://mattmahoney.net/dc/zpaq.html http://mattmahoney.net/dc/zpaq.html
- Klasiaster 4y ago
  ZPAQ is the name of the tool but ZPAQ is also the name of the container format that gets used. ZPAQ embeds the decompression algorithm in the archive. One could store zstd-compressed blocks in ZPAQ archives as soon as a zpaql decompressor exists (e.g., for brotli there is a slow one implemented in a python subset and compiled to zpaql https://github.com/pothos/zpaqlpy https://github.com/pothos/zpaqlpy). I don't know exactly whether other formats are better for seeking and streaming, but since the baseline is tar, ZPAQ (in the 2.0 spec) is already better as it supports deduplication and files can even be updated append-only, and the compression is not an afterthought wrapped around it but well integrated.
INTPenis 4y ago
xz got a bit of hype there about 10 years ago, I used it until a couple years ago when I noticed how slow it was with huge DB dumps and how much faster zstd was while still having decent compression. So I have no idea about all this low level stuff, I just know that zstd is overall better for sysadmins. But next time I'm doing any sort of scripting that involves compression I'll take a look at squashfs now due to this thread.
Quequau 4y ago
Could anyone recommend a broad scoped evaluation of current compression / archiving formats / algorithms which explores their various merits and failings?
KronisLV 4y ago
Archival formats have always been of interest to me, given the very practical need to store a large amount of backups across any number of storage mediums - documents, pictures, music, sometimes particularly good movies, even the occasional software or game installer. Right now, I've personally settled on using the 7z format: https://en.wikipedia.org/wiki/7z https://en.wikipedia.org/wiki/7z The decompression speeds feel good, the compression ratios also seem better than ZIP and somehow it still feels like a widely supported format, with the 7-Zip program in particular being nice to use: https://en.wikipedia.org/wiki/7-Zip https://en.wikipedia.org/wiki/7-Zip Of course, various archivers on *nix systems also seem to support it, so so far everything feels good. Though of course having the chance of an archive getting corrupt and no longer being properly able to decompress it and read all of those files, versus just using the filesystem and having something like that perhaps occur to a single file still sometimes bothers me. Then again, on a certain level, I guess nothing is permanent and at least it's possible to occasionally test the archives for any errors and look into restoring them from backups, should something like that ever occur. Might just have to automate those tests, though. Yet, for the most part, going with an exceedingly boring option like that seems like a good idea, though the space could definitely use more projects and new algorithms for even better compression ratios, so at the very least it's nice to see attempts to do so!
- petre 4y ago
  7zip also uses LZMA under the hoid, just like xz and lzip, but recovery is also poor.
justinclift 4y ago
(2016)
yellowapple 4y ago
One thing that seems to be unmentioned so far in the conversation: xz is public domain, while lzip is subject to the full-blown GPL (v2 or later). In any case, I don't really bother with compression for my own archival needs. Storage is cheap, and encrypted data is kinda hard to reasonably compress anyway.
- brnt 4y ago
  Why would you not compress first, then encrypt?
  - yellowapple 4y ago
    Because I usually think about encryption long before I think about compression; the latter's a bit of an afterthought. Ain't the most logical answer in the world, and if I planned from the outset to both compress and encrypt then I'd do it in that order, but compression usually doesn't cross my mind for archival (whereas encryption is pretty much the default for any data I have that's worth archiving).
  - ChadNauseam 4y ago
    This can lead to side-channel attacks, see https://en.m.wikipedia.org/wiki/CRIME https://en.m.wikipedia.org/wiki/CRIME
- Karellen 4y ago
  Isn't that a limitation of the implementation, which can be worked around by creating a new implementation with whatever licensing you want based on the format specification? OTOH, a limitation of the spec cannot be worked around by any new implementation. (OT - isn't it generally recommended to compress before encrypting? Encrypting is CPU-intensive so the less you have to encrypt the better, also length can be a side-channel, and don't some encryption methods leak the existence of patterns in the source data which compression will eliminate?)
stjohnswarts 4y ago
At least it isn't considered harmful
throwmemoney 4y ago
No one seems to mention lrzip ?
sharmin123 4y ago
tex0 4y ago
This serves as another example to me that governance and conflict resolution in the Debian project is really poor. Maintainers are free to do whatever they want, even if it doesn't make any sense at all.
- rlpb 4y ago
  xz makes sense for Debian. The article author links to a Debian mailing list thread as if that somehow provides evidence that Debian made a bad decision. I just read the thread and that is the opposite of that thread's conclusion. The consensus in that thread is that Debian's existing use of xz is just fine for its own purposes. Debian uses external cryptographic verification for its packages and apt archives so it does not need additional robustness in the compression format.
OliverJones 4y ago
We're talking about long-term archiving here. That means centuries. My brother the archaelogical archivist of ancient (~2000 years BCE) mesopotamian artifacts has a lot to say about archival formats. His raw material is mostly fired clay tablets. Those archives keep working, partially, even if broken. That's good, because many of them are in fact broken when found. But their ordering and other metadata about where they were found is written in archaeologists' notebooks, and many of those notebooks are now over a century old. Paper deteriorates. If a lost flake of paper from a page in the notebook rendered the whole notebook useless, that would be a disastrous outcome for that archive. A decade ago I suggested digitizing the notebooks and storing the bits on CD-ROMs. He laughed, saying "we don't know enough about the long-term viability of CD-ROMs and their readers." Now, when they get around it it, they're digitizing the notebooks and storing them on PDFs on backed-up RAID10 volumes. But they're also printing them on acid-free paper and putting them with the originals in the vaults where they store old books. My point: planning for centuries long archiving is difficult. Formats with redundancy, at least with forward error correction codes, are very helpful. Formats that can be rendered useless by a few bit-flips, not so much.
- jeofken 4y ago
  What about archiving them in torrent format - in this way, as long as there is one nerd who values history out there, there will be a copy
  - zxspectrum1982 4y ago
    Bittorrent is not archival, it's distribution.
    - cogburnd02 4y ago
      Can't distribution be archival? Thomas Jefferson: "Let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident."
rakoo 4y ago
A format for archives, that is contractually built to last, has an impressive test suite, is easily browseable, where blobs can be retrieved individually if needed and is already known everywhere ? Sounds like SQlite, yet again: https://www2.sqlite.org/sqlar.html https://www2.sqlite.org/sqlar.html
- chrisseaton 4y ago
  > A format for archives, that is contractually built to last Who would you be contracting to build a file format?
  - rakoo 4y ago
    Not the file format itself but sqlite: Airbus is using it for its A350 line, so it is guaranteed they will do what it takes to keep it working as long as the plane exists.
jl6 4y ago
I disagree with the premise of the article. Archive formats are all inadequate for long-term resilience and making them adequate would be a violation of the “do one thing and do it right” principle. To support resilience, you don’t need an alternative to xz, you need hashes and forward error correction. Specifically, compress your file using xz for high compression ratio, optionally encrypt it, then take a SHA-256 hash to be used for detecting errors, then generate parity files using PAR[1] or zfec[2] to be used for correcting errors. [1] https://wiki.archlinux.org/title/Parchive https://wiki.archlinux.org/title/Parchive [2] https://github.com/tahoe-lafs/zfec https://github.com/tahoe-lafs/zfec
gorgoiler 4y ago
Can you serialise a ZFS filesystem into a disk image? I feel like ZFS is the leader in data integrity, redundancy, and compression?
- seized 4y ago
  Yes, you can snapshot to a file.
fay59 4y ago
This is fairly old. When it came up last time, there were robust arguments that xz was characterized unfairly, and that the author’s format wasn’t very good at recovering in most cases either.
sbf501 4y ago
How about just not compressing things for archival? A few bit errors in uncompressed files would end up as just a few bad characters. Whereas a few errors in an uncorrectable compression format might render the entire content useless. Sure they files are huge, but were talking about long term archival. In fact, if the documents are that important, have RAID-style redundancy and multiple-bit ECC in multiple geographic locations as well.
- jffry 4y ago
  The format of the uncompressed files matters just as much in terms of bit error resilience. Compression or not, you can always use an additional tool to produce and store extra parity data which can be used to both correct and repair bit errors, which seems like the correct answer for digital archival.
danbmil99 4y ago
Here's a thought: vinyl. While I haven't done intensive research on this, it occurs to me that plastic lasts a long time. Vinyl records are a format that seems fit for long-term archiving. The format is so obvious that it could be reverse engineered by any future civilization. So at least they'll know something about our taste in music.
m463 4y ago
I was surprised to find that .tar format has no checksums/crc. You need .tar.gz to get a crc during compression.

248 points by goranmoomin 4y ago

The vast majority of the discussion is around xz’s inability of dealing with corrupted data. That said, couldn’t you argue that that needs to be solved at a lower level (storage, transport)? I’m not convinced the compression algorithm is the right place to tackle this. Just use a file system that does proper integrity checking/resilvering. Also use TLS to transfer data over the network.

riffraff 4y ago
The article is about usage of xz for long term archival, so transport is not relevant, the concern seems to be bitrot and forward compatibility. Storage with integrity checking would be the solution to bitrot, but TFA also seems concerned with "how do you unarchive/recover a random file you found?" which seems a somewhat valid concern. And xz does have support for integrity checking, so it seems reasonable to have a discussion on whether that is a good support, rather than on whether it should be there at all.
- pmoriarty 4y ago
  > xz does have support for integrity checking Archives need not only a way to check their integrity but also error correction, which xz does not have. However, you can easily combine xz with par2, which does provide error correction.
smaudet 4y ago
> couldn’t you argue that that needs to be solved at a lower level Self-referentially incorruptible data is the standard to beat here, moving the concern to a different layer doesn't increase the efficiency or integrity of the data itself. It is arguably less efficient, as you now rely on some lower layer of protection in addition to whatever is built into the standard itself. It is less flexible - a properly protected archive format could be scrawled on to the side of a hill, or more reasonably onto an archive medium (BD-disk), and should be able to survive any file-system change, upgrade, etc. Self-repairing hard drives with multiple redundancies are nice, but not cheap, and not wide-spread. It also does nothing for actually protecting the data - I don't care how advanced the lower level storage format is, if you overwrite data e.g. with random zeros (conceivable due a badly behaving program with too much memory access, e.g. a misbehaving virus, or a bad program or kernal driver someone ran with root access, also conceivable due to EM interference or solar radiation causing a program to misbehave), the file system will dutifully overwrite the correct data with incorrect data, including updating whatever relevant checks it has for the file. The only way around this is to maintain a historical archive of all writes ever made, and that should be evidently both absurdly impractical (how do you maintain the integrity of this archive? With a self-referentially incorruptible data archive perhaps?) and expensive. Compared to a single file, which can be backed up, agnostic to the filesystem/hardware/transport/major world-ending events, which can be simply read/recovered, far into the future. There's a pretty clear winner here.
- EdSchouten 4y ago
  > Self-referentially incorruptible data is the standard to beat here, moving the concern to a different layer doesn't increase the efficiency or integrity of the data itself. I respectfully disagree. By putting it in the layer below, there is the ability to do repairs. For example, consider storing XZ files on a Ceph storage cluster. Ceph supports Reed-Solomon coding. This means that if data corruption occurs, Ceph is capable of automatically repairing the data corruption by recomputing the original file and writing it back to disk once more. Even if XZ were able to recover from some forms of data corruption, is it realistic that such repairs propagate back to the underlying data store? Likely not.
  - smaudet 4y ago
    You are thinking in the wrong direction, I tried to explain but maybe I can be clearer: If you can't read the data in question though, you cannot do the repairs, it doesn't matter if you do Reed-Solomon coding or not. You are thinking about coding for the underlying hardware, which is what the data corruption you are talking about is designed to fix - it does not solve the problem for writes coming from above. To do that, you actually have to decode the data in question and perform a reed-solomon encoding on the actual file inside of the archive, and this only gets worse e.g. if you have nested archives. If the data is self-referentially repairable, however, it doens't matter if the file gets overwritten with e.g. a cat gif, the format will work around that. The filesystem on the other hand will have written the cat gif to the file and updated the Reed Solomon encoding for your file, assuming (incorrectly) the file writes were valid. I suppose you could mandate that for any file to be written to your filesystem it must first be completely decompressed, and then store some encoding information alongside the archive, but this would be inefficient to the extreme, since merely copying a file onto the system would mean you have to decompress the file and then checksum it. At any rate, even if you did decompress the file in question, you have failed to separate the layers like you want to, since now you have mandated the XZ and LZMA algorithms also be baked directly into the filesystem itself. Better not to needlessly couple the filesystem to some compression algorithm, let the compression system handle its own error correction.
    - 4y ago

metadat 4y ago

Correct, xz is no longer particularly useful, mostly annoying. For read-only long-term filesystem-like archives, use squashfs with whatever compression option is convenient. The cool thing about squashfs is you can efficiently list the contents and extract single files. This is why it's an ultimate best option for long-term filesystem-esque archives. https://en.m.wikipedia.org/wiki/SquashFS https://en.m.wikipedia.org/wiki/SquashFS For everything else, or if you want (very) fast compression/decompression and/or general high-ratio compression, use .zstd. https://github.com/facebook/zstd https://github.com/facebook/zstd I've used this combination professionally to great effect. You're welcome :)

ars 4y ago
zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.
- metadat 4y ago
  Do you have any data to support this claim? In my experience, zstd is way better in every way compared to gzip. Additionally, xz is good compression but horribly crazy slow to decompress. Xz also only operates on one single file at a time, which is annoying.
  - jmillikin 4y ago
    I just did a quick test with a copy of GNU M4, which is reasonably representative of a source code archive. $ time xz -9k m4-1.4.19.tar real 0m2.928s user 0m2.871s sys 0m0.056s $ time zstd -19 m4-1.4.19.tar real 0m3.411s user 0m3.380s sys 0m0.032s $ ls -l m4-1.4.19.tar* -rw-rw-r-- 1 john john 14837760 Jul 24 14:40 m4-1.4.19.tar -rw-rw-r-- 1 john john 1674612 Jul 24 14:40 m4-1.4.19.tar.xz -rw-rw-r-- 1 john john 1726155 Jul 24 14:40 m4-1.4.19.tar.zst In this test, XZ was both faster and had better compression than Zstd.
    - metadat 4y ago
      Howdy jmillikin. That is a very small file. See: https://news.ycombinator.com/item?id=25455846#:~:text=xz%20on%20highest%20(normal)%20compression,speeds%20regardless%20of%20compression%20ratio https://news.ycombinator.com/item?id=25455846#:~:text=xz%20o.... > xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio

deleted 4y ago

[deleted]

stop50 4y ago

Interesting, i used xz till today for compression, but i think i will use gzip and zstd ffrom now on.

ars 4y ago
I'll continue using xz - the compression ratio is far better than gzip or zstd. Maybe there will be some new future format to switch to, but zstd is not it.
- stop50 4y ago
  zstd got me already better results, but not all machines i am administering have it installed or even available.
- petre 4y ago
  Yup. To my surprise xz -1 is faster and compresses better than gzip -6, the default setting. Guess what I'm using now? I'll use zstd when the GNU tar in my distribution supports it directly without resorting to pipes and shell magic. It would be nice if one could specify the compression level on the GNU tar command line. BSD tar already does all of the above.
- assttoasstmgr 4y ago
  The rest of the sane world will continue to use gzip because the marginal compression savings is not worth the compatibility issues. I have lost count of the number of times I have urgently needed to extract an xz archive on a system only to find out xz isn't available. Most notably Solaris and embedded systems.
  - kouteiheika 4y ago
    > The rest of the sane world will continue to use gzip because the marginal compression savings is not worth the compatibility issues. It is definitely worth it to use a modern compression format. I regularly take backups of one service I run. Originally I used gzip, and then on someone's recommendation I tried zstd. Here are the results: gzip compression time: 204s zstd compression time: 19.1s gzip decompression time: 28.2s zstd decompression time: 6.4s gzip compressed size: 3.0GB zstd compressed size: 1.7GB I ain't ever going back to gzip, especially when it wastes so much disk space and CPU cycles. > sane world [..] Solaris ...I think we're living in a different world. (:

saurik 4y ago

It was really disappointing when dpkg actively deprecated support (which I feel they should never do) for lzma format archives and went all-in on xz. The decompressor now needs the annoying flexibility mentioned in this article and the only benefit of the format--the ability to do random access on the file--is almost entirely defeated by dpkg using it to compress a tar file (which barely supports any form of accelerated access even when uncompressed; like the best you can do is kind of attempt to skip through file headers, which only helps if the files in the archive are large enough) and, to add insult to injury, the files are now all slightly larger to account for the extra headers :/. Regardless, this is a pretty old article and if you search for it you will find a number of discussions that have already happened about it that all have a bunch of comments. https://news.ycombinator.com/item?id=20103255 https://news.ycombinator.com/item?id=20103255 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=12768425

jks 4y ago
pixz (https://github.com/vasi/pixz https://github.com/vasi/pixz) is a nice parallel xz that additionally creates an index of tar files so you can decompress individual files. I wonder if dpkg could be extended to do something similar.

jmillikin 4y ago

The article spends a lot of time discussing XZ's behavior when reading corrupt archives, but in practice this is not a case that will ever happen. Say you have a file `m4-1.4.19.tar.xz` in your source archive collection. The version I just downloaded from ftp.gnu.org has the SHA-256 checksum `63aede5c6d33b6d9b13511cd0be2cac046f2e70fd0a07aa9573a04a82783af96`. The GNU site doesn't have checksums for archives, but it does have a PGP signature file, so it's possible to (1) verify the archive signature and (2) store the checksum of the file. If that file is later corrupted, the corruption will be detected via the SHA-256 checksum. There's no reason to worry about whether the file itself has an embedded CRC-whatever, or the behavior with regards to variable-length integer fields. If a bit gets flipped then the checksum won't match, and that copy of the file will be discarded (= replaced with a fresh copy from replicated storage) before it ever hits the XZ code. If this is how the lzip format was designed -- optimized for a use case that doesn't exist -- it's no wonder it sees basically no adoption in favor of more pragmatic formats like XZ or Zstd.

thrdbndndn 4y ago
I have no idea why you think that use case does not exist. Your whole idea about archive seems to be it is to ensure a blob doesn't change (same hash). But that's far from the only use of archive. (Hell, even with that, you are assuming you know the correct hash of the file to begin with, which isn't guaranteed.) "Repairing" corrupt archives, as in to get as much as usable data from that archive is a pretty useful thing and I have done it multiple times. For example, an archive can have hundreds of files inside and if you can recover any of them that's better than nothing. It is also one of the reason I still use WinRAR occasionally due to its great recovery record (RR) feature. >replaced with a fresh copy from replicated storage Lots of times you don't have other copy.
- jmillikin 4y ago
  The process of long-term archival starts with replication. A common approach is two local copies on separate physical media, and one remote copy in a cloud storage service with add-only permissions. This protects against hardware failure, bad software (accidental deletion, malware), natural disasters (flood, fire) and other 99th-percentile disaster conditions. The cloud storage providers will have their own level of replication (AWS S3 has a 99.999999999% durability SLA). If you have only one copy of some important file and you discover it no longer matches the stored checksum, then that's not a question of archival, but of data recovery. There's no plausible mechanism by which a file might suffer a few bitflips and be saved by careful application of a CRC -- bitrot often zeros out entire sectors.

gfody 4y ago

about 4 years ago I had to choose a compression format for streaming database backups, and so I compared every option supported by 7z and it was xz that was the best compromise between performance and compression ratio

haberman 4y ago

Folks seem to be comparing xz to zstd, but if I am understanding correctly the true competitor to xz is the article author’s “lzip” format, which uses the same LZMA compression as xz but with a much better designed container format (at least according to the author).

exyi 4y ago
I'd not say it's necessarily better designed - it's just simpler. Few bytes of some headers, LZMA compressed data, checksum, done. No support for seeks and stuff like that
loeg 4y ago
Emphasis on “according to the author,” yeah. And the concerns are silly — FEC can and should be added outside the compressed data (e.g., with par2).
- brnt 4y ago
  But why outside the compression container? I'd love to have format that let me repair their contents up to the number of parity bits it stored.
  - joveian 4y ago
    I agree that a single file is super convenient. It looks like the in-progress par3 will support that (par3 inside zip, I'd add an mtree file as well for metadata if needed): https://parchive.github.io/doc/Parity_Volume_Set_Specification_v3.0.html https://parchive.github.io/doc/Parity_Volume_Set_Specificati... I don't use par2 right now for backups due to the extra hastle, although I might use par3. It might end up a waste of space, though, since I haven't had an issue with corrupted files in at least 20 years (would be great for optical media that can get scratched). For the longest term archiving I'm not sure any kind of compression is a good idea, I'd think along the lines of uncompressed copy + parity info in a simple container where all files are contiguous (maybe a raw zip file without compression).

latchkey 4y ago

I routinely distribute a ~5meg .xz (~20meg uncompressed) to 20k+ servers across multiple physical data centers on a regular basis. Haven't seen a single failure. It ends up being 1.85megs smaller than the tgz version. Unless someone comes up with a better solution (ie: smaller), I probably won't change that any time soon.

smaudet 4y ago
What about lzip? The author claims the compression size is comparable or better.
- latchkey 4y ago
  I haven't tried that one yet. I'll play around and see what I can find out. 7zip didn't perform as well. tar c -C ./build $(BINARY) | gzip -9 - > $(PKG_NAME_GZ) tar c -C ./build $(BINARY) | xz -z -9e - > $(PKG_NAME)
- jraph 4y ago
  It should, if the compression algorithm is actually the same, just the format specification that differs.
- exyi 4y ago
  It will be the same, it also just uses LZMA. It has less headers, but that's not going to make much of a difference on 5M file
  - latchkey 4y ago
    Compressed to 5mb. Original is about 20.
jraph 4y ago
You are not using for long term archival though, it seems you are using it for deployment.
- 4y ago

waynesonfire 4y ago

Wow xz just got owned. How do you recover from this.

jraph 4y ago
This article is not new, this is the second time I encounter it, it seems it has been published in 2016. Most users of xz probably won't read it and will keep using it. And it's probably fine for most use cases, though lzip would most likely be better in every scenario if available.

poser-boy 4y ago

I'd recommend checking out zpaq[1], it purposed for backups, and has great compression (even on low setting) for large 100GB+ file collections. However for smaller stuff I use zstd at level 22 in a tar for most things since it's much faster, though a little heavier. [1] http://mattmahoney.net/dc/zpaq.html http://mattmahoney.net/dc/zpaq.html

Klasiaster 4y ago
ZPAQ is the name of the tool but ZPAQ is also the name of the container format that gets used. ZPAQ embeds the decompression algorithm in the archive. One could store zstd-compressed blocks in ZPAQ archives as soon as a zpaql decompressor exists (e.g., for brotli there is a slow one implemented in a python subset and compiled to zpaql https://github.com/pothos/zpaqlpy https://github.com/pothos/zpaqlpy). I don't know exactly whether other formats are better for seeking and streaming, but since the baseline is tar, ZPAQ (in the 2.0 spec) is already better as it supports deduplication and files can even be updated append-only, and the compression is not an afterthought wrapped around it but well integrated.

INTPenis 4y ago

xz got a bit of hype there about 10 years ago, I used it until a couple years ago when I noticed how slow it was with huge DB dumps and how much faster zstd was while still having decent compression. So I have no idea about all this low level stuff, I just know that zstd is overall better for sysadmins. But next time I'm doing any sort of scripting that involves compression I'll take a look at squashfs now due to this thread.

Quequau 4y ago

Could anyone recommend a broad scoped evaluation of current compression / archiving formats / algorithms which explores their various merits and failings?

KronisLV 4y ago

Archival formats have always been of interest to me, given the very practical need to store a large amount of backups across any number of storage mediums - documents, pictures, music, sometimes particularly good movies, even the occasional software or game installer. Right now, I've personally settled on using the 7z format: https://en.wikipedia.org/wiki/7z https://en.wikipedia.org/wiki/7z The decompression speeds feel good, the compression ratios also seem better than ZIP and somehow it still feels like a widely supported format, with the 7-Zip program in particular being nice to use: https://en.wikipedia.org/wiki/7-Zip https://en.wikipedia.org/wiki/7-Zip Of course, various archivers on *nix systems also seem to support it, so so far everything feels good. Though of course having the chance of an archive getting corrupt and no longer being properly able to decompress it and read all of those files, versus just using the filesystem and having something like that perhaps occur to a single file still sometimes bothers me. Then again, on a certain level, I guess nothing is permanent and at least it's possible to occasionally test the archives for any errors and look into restoring them from backups, should something like that ever occur. Might just have to automate those tests, though. Yet, for the most part, going with an exceedingly boring option like that seems like a good idea, though the space could definitely use more projects and new algorithms for even better compression ratios, so at the very least it's nice to see attempts to do so!

petre 4y ago
7zip also uses LZMA under the hoid, just like xz and lzip, but recovery is also poor.

justinclift 4y ago

(2016)

yellowapple 4y ago

One thing that seems to be unmentioned so far in the conversation: xz is public domain, while lzip is subject to the full-blown GPL (v2 or later). In any case, I don't really bother with compression for my own archival needs. Storage is cheap, and encrypted data is kinda hard to reasonably compress anyway.

brnt 4y ago
Why would you not compress first, then encrypt?
- yellowapple 4y ago
  Because I usually think about encryption long before I think about compression; the latter's a bit of an afterthought. Ain't the most logical answer in the world, and if I planned from the outset to both compress and encrypt then I'd do it in that order, but compression usually doesn't cross my mind for archival (whereas encryption is pretty much the default for any data I have that's worth archiving).
- ChadNauseam 4y ago
  This can lead to side-channel attacks, see https://en.m.wikipedia.org/wiki/CRIME https://en.m.wikipedia.org/wiki/CRIME
Karellen 4y ago
Isn't that a limitation of the implementation, which can be worked around by creating a new implementation with whatever licensing you want based on the format specification? OTOH, a limitation of the spec cannot be worked around by any new implementation. (OT - isn't it generally recommended to compress before encrypting? Encrypting is CPU-intensive so the less you have to encrypt the better, also length can be a side-channel, and don't some encryption methods leak the existence of patterns in the source data which compression will eliminate?)

stjohnswarts 4y ago

At least it isn't considered harmful

throwmemoney 4y ago

No one seems to mention lrzip ?

sharmin123 4y ago

tex0 4y ago

This serves as another example to me that governance and conflict resolution in the Debian project is really poor. Maintainers are free to do whatever they want, even if it doesn't make any sense at all.

rlpb 4y ago
xz makes sense for Debian. The article author links to a Debian mailing list thread as if that somehow provides evidence that Debian made a bad decision. I just read the thread and that is the opposite of that thread's conclusion. The consensus in that thread is that Debian's existing use of xz is just fine for its own purposes. Debian uses external cryptographic verification for its packages and apt archives so it does not need additional robustness in the compression format.

OliverJones 4y ago

We're talking about long-term archiving here. That means centuries. My brother the archaelogical archivist of ancient (~2000 years BCE) mesopotamian artifacts has a lot to say about archival formats. His raw material is mostly fired clay tablets. Those archives keep working, partially, even if broken. That's good, because many of them are in fact broken when found. But their ordering and other metadata about where they were found is written in archaeologists' notebooks, and many of those notebooks are now over a century old. Paper deteriorates. If a lost flake of paper from a page in the notebook rendered the whole notebook useless, that would be a disastrous outcome for that archive. A decade ago I suggested digitizing the notebooks and storing the bits on CD-ROMs. He laughed, saying "we don't know enough about the long-term viability of CD-ROMs and their readers." Now, when they get around it it, they're digitizing the notebooks and storing them on PDFs on backed-up RAID10 volumes. But they're also printing them on acid-free paper and putting them with the originals in the vaults where they store old books. My point: planning for centuries long archiving is difficult. Formats with redundancy, at least with forward error correction codes, are very helpful. Formats that can be rendered useless by a few bit-flips, not so much.

jeofken 4y ago
What about archiving them in torrent format - in this way, as long as there is one nerd who values history out there, there will be a copy
- zxspectrum1982 4y ago
  Bittorrent is not archival, it's distribution.
  - cogburnd02 4y ago
    Can't distribution be archival? Thomas Jefferson: "Let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident."

rakoo 4y ago

A format for archives, that is contractually built to last, has an impressive test suite, is easily browseable, where blobs can be retrieved individually if needed and is already known everywhere ? Sounds like SQlite, yet again: https://www2.sqlite.org/sqlar.html https://www2.sqlite.org/sqlar.html

chrisseaton 4y ago
> A format for archives, that is contractually built to last Who would you be contracting to build a file format?
- rakoo 4y ago
  Not the file format itself but sqlite: Airbus is using it for its A350 line, so it is guaranteed they will do what it takes to keep it working as long as the plane exists.

jl6 4y ago

I disagree with the premise of the article. Archive formats are all inadequate for long-term resilience and making them adequate would be a violation of the “do one thing and do it right” principle. To support resilience, you don’t need an alternative to xz, you need hashes and forward error correction. Specifically, compress your file using xz for high compression ratio, optionally encrypt it, then take a SHA-256 hash to be used for detecting errors, then generate parity files using PAR[1] or zfec[2] to be used for correcting errors. [1] https://wiki.archlinux.org/title/Parchive https://wiki.archlinux.org/title/Parchive [2] https://github.com/tahoe-lafs/zfec https://github.com/tahoe-lafs/zfec

gorgoiler 4y ago

Can you serialise a ZFS filesystem into a disk image? I feel like ZFS is the leader in data integrity, redundancy, and compression?

seized 4y ago
Yes, you can snapshot to a file.

fay59 4y ago

This is fairly old. When it came up last time, there were robust arguments that xz was characterized unfairly, and that the author’s format wasn’t very good at recovering in most cases either.

sbf501 4y ago

How about just not compressing things for archival? A few bit errors in uncompressed files would end up as just a few bad characters. Whereas a few errors in an uncorrectable compression format might render the entire content useless. Sure they files are huge, but were talking about long term archival. In fact, if the documents are that important, have RAID-style redundancy and multiple-bit ECC in multiple geographic locations as well.

jffry 4y ago
The format of the uncompressed files matters just as much in terms of bit error resilience. Compression or not, you can always use an additional tool to produce and store extra parity data which can be used to both correct and repair bit errors, which seems like the correct answer for digital archival.

danbmil99 4y ago

Here's a thought: vinyl. While I haven't done intensive research on this, it occurs to me that plastic lasts a long time. Vinyl records are a format that seems fit for long-term archiving. The format is so obvious that it could be reverse engineered by any future civilization. So at least they'll know something about our taste in music.

m463 4y ago

I was surprised to find that .tar format has no checksums/crc. You need .tar.gz to get a crc during compression.