5 ms·

This article is originally from 2016, not from 2022, as has also been pointed out when it was last posted circa 11 hours ago. 2024: https://news.ycombinator.co

This article is originally from 2016, not from 2022, as has also been pointed out when it was last posted circa 11 hours ago. 2024: https://news.ycombinator.com/item?id=39868810 https://news.ycombinator.com/item?id=39868810 2022: https://news.ycombinator.com/item?id=32210438 https://news.ycombinator.com/item?id=32210438 2019: https://news.ycombinator.com/item?id=20103255 https://news.ycombinator.com/item?id=20103255 2018: https://news.ycombinator.com/item?id=16884832 https://news.ycombinator.com/item?id=16884832 2016: https://news.ycombinator.com/item?id=12768425 https://news.ycombinator.com/item?id=12768425 It is utterly unrelated to the ongoing issue (which I'm 100% sure is the reason this is on the front page for a second time within 12 hours right now). The author is also the developer of lzip, while it does have some valid points, one should apply a necessary level of skepticism ("Competitors product is really bad and unfit", says vendor).

tex0 3y ago
Thank you for reposting. I don't want to start bashing XZ, but I honestly wonder why it's been picked up so much despite the valid criticism. As if the compression rate increase at medium levels were so significant over bzip2 or grip.
- adeon 3y ago
  If I piggyback off one of the commenters on the 11 hours ago post: https://news.ycombinator.com/item?id=39868810#39869769 https://news.ycombinator.com/item?id=39868810#39869769 > I think none of these issues really matter. I think that's probably it. I.e. it's a nobody-cares issue. Looking at the older posts, there's some serious issues with lzip too (looks like corruption-related) that got a less than stellar response from the author. Just quickly tested lzip vs zstd -9 on a 100 megabyte text file. zstd is almost as good but many times faster. I wonder if the lzip author's work got obsolete. I skimmed the manual of lzip a bit. The author likes to talk a lot about how everything is done correctly in lzip and there's this line line "The lzip format specification has been reviewed carefully and is believed to be free from design errors. " If you type lzip --help it talks about how it's better than bzip2 or gzip. Maybe they are right but ugh comes off real arrogant.
  - Maxious 3y ago
    Phil Katz was real arrogant but also real right
  - Bulat_Ziganshin 3y ago
    1. both lzip and xz are using lzma compression library internally, so there is no difference in their compression ratio/speed 2. lzma compression is LZ + markov chains, while zstd is LZ + order-0 entropy coder (similar to zlib, rar and many other popular algorithms) markov chains are higher-order entropy coding, i.e. one using context of previous data. it's slower, but sometimes gives better compression. but text files don't get any benefit from it. OTOH, various binary formats, like executables or databases, get significantly better compression ratio. in my tests lzma-compressed binary files are ~~10% smaller on average. so, many claims that zstd and lzma provides the same compression ratio, are based on testing on text or other files that don't benefit from higher-order entropy coding. of course, I imply maximum-compression setting and equal dictionary size, in order to make fair comparison of compression formats rather than particular implementations. (I'm author of freearc and several LZ-based compressors, so more or less expert in this area :)
- wongarsu 3y ago
  Because xz archives the best compression ratios. If you care about archive size above all else it's the best choice. If you use "medium levels" I agree there's no point, zstd is superior in that regime (achieving faster compression and decompression for these compression ratios)
- lifthrasiir 3y ago
  Any compressed format is okay as long as you have a working compressor and decompressor, where the compressor can reliably compress an input and the decompressor can reliably decompress the compressed input. Everything else is not as relevant, including the exact file format (because you have the known-good decompressor).
- JoshTriplett 3y ago
  xz is not worth using at medium levels. At its highest levels, it sometimes just barely has an edge over the highest levels of zstd, and it existed before zstd did. These days, zstd is the obvious choice.
Lockal 3y ago
While you are saying that it is utterly unrelated, I'll respond with "that not only related, this is a root cause of current situation". We have a single massively used implementation (also known as reference), which does not even follow xz format specification. It is autohell-based, which allowed m4 injection. In other places it is CMake-based, but uses autohell-based code, inserting hidden dots where they could just use https://cmake.org/cmake/help/latest/module/CheckSymbolExists.html https://cmake.org/cmake/help/latest/module/CheckSymbolExists... "Poor design" is still there, as a core of library. It is as cool, 64-bit square roots and 128-bit divides in Bcachefs[1] It is as cool as lbzip2[2] with their: (((0xffffaa50 >> ((as < 0x20 ? as : (as >> 4)) & 0x1e)) & 0x3) + (as < 0x20 ? 1 : 5)) Spoiler: this lbzip2 code produces corrupted files in some cases, should we care if it is a backdoor? Or as usual, disable optimizations, disable valgrind, disable fuzzers and say that everything is ok? [1] https://www.phoronix.com/news/Linux-6.9-Bcachefs-Attempt https://www.phoronix.com/news/Linux-6.9-Bcachefs-Attempt [2] https://github.com/kjn/lbzip2/blob/b6dc48a7b9bfe6b340ed1f6d72133608ad57144b/src/encode.c#L1123 https://github.com/kjn/lbzip2/blob/b6dc48a7b9bfe6b340ed1f6d7...

tex0 3y ago

Thank you for reposting. I don't want to start bashing XZ, but I honestly wonder why it's been picked up so much despite the valid criticism. As if the compression rate increase at medium levels were so significant over bzip2 or grip.

adeon 3y ago
If I piggyback off one of the commenters on the 11 hours ago post: https://news.ycombinator.com/item?id=39868810#39869769 https://news.ycombinator.com/item?id=39868810#39869769 > I think none of these issues really matter. I think that's probably it. I.e. it's a nobody-cares issue. Looking at the older posts, there's some serious issues with lzip too (looks like corruption-related) that got a less than stellar response from the author. Just quickly tested lzip vs zstd -9 on a 100 megabyte text file. zstd is almost as good but many times faster. I wonder if the lzip author's work got obsolete. I skimmed the manual of lzip a bit. The author likes to talk a lot about how everything is done correctly in lzip and there's this line line "The lzip format specification has been reviewed carefully and is believed to be free from design errors. " If you type lzip --help it talks about how it's better than bzip2 or gzip. Maybe they are right but ugh comes off real arrogant.
- Maxious 3y ago
  Phil Katz was real arrogant but also real right
- Bulat_Ziganshin 3y ago
  1. both lzip and xz are using lzma compression library internally, so there is no difference in their compression ratio/speed 2. lzma compression is LZ + markov chains, while zstd is LZ + order-0 entropy coder (similar to zlib, rar and many other popular algorithms) markov chains are higher-order entropy coding, i.e. one using context of previous data. it's slower, but sometimes gives better compression. but text files don't get any benefit from it. OTOH, various binary formats, like executables or databases, get significantly better compression ratio. in my tests lzma-compressed binary files are ~~10% smaller on average. so, many claims that zstd and lzma provides the same compression ratio, are based on testing on text or other files that don't benefit from higher-order entropy coding. of course, I imply maximum-compression setting and equal dictionary size, in order to make fair comparison of compression formats rather than particular implementations. (I'm author of freearc and several LZ-based compressors, so more or less expert in this area :)
wongarsu 3y ago
Because xz archives the best compression ratios. If you care about archive size above all else it's the best choice. If you use "medium levels" I agree there's no point, zstd is superior in that regime (achieving faster compression and decompression for these compression ratios)
lifthrasiir 3y ago
Any compressed format is okay as long as you have a working compressor and decompressor, where the compressor can reliably compress an input and the decompressor can reliably decompress the compressed input. Everything else is not as relevant, including the exact file format (because you have the known-good decompressor).
JoshTriplett 3y ago
xz is not worth using at medium levels. At its highest levels, it sometimes just barely has an edge over the highest levels of zstd, and it existed before zstd did. These days, zstd is the obvious choice.

Lockal 3y ago

While you are saying that it is utterly unrelated, I'll respond with "that not only related, this is a root cause of current situation". We have a single massively used implementation (also known as reference), which does not even follow xz format specification. It is autohell-based, which allowed m4 injection. In other places it is CMake-based, but uses autohell-based code, inserting hidden dots where they could just use https://cmake.org/cmake/help/latest/module/CheckSymbolExists.html https://cmake.org/cmake/help/latest/module/CheckSymbolExists... "Poor design" is still there, as a core of library. It is as cool, 64-bit square roots and 128-bit divides in Bcachefs[1] It is as cool as lbzip2[2] with their: (((0xffffaa50 >> ((as < 0x20 ? as : (as >> 4)) & 0x1e)) & 0x3) + (as < 0x20 ? 1 : 5)) Spoiler: this lbzip2 code produces corrupted files in some cases, should we care if it is a backdoor? Or as usual, disable optimizations, disable valgrind, disable fuzzers and say that everything is ok? [1] https://www.phoronix.com/news/Linux-6.9-Bcachefs-Attempt https://www.phoronix.com/news/Linux-6.9-Bcachefs-Attempt [2] https://github.com/kjn/lbzip2/blob/b6dc48a7b9bfe6b340ed1f6d72133608ad57144b/src/encode.c#L1123 https://github.com/kjn/lbzip2/blob/b6dc48a7b9bfe6b340ed1f6d7...