3 ms·

The format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to

by jl6 3y ago

The format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to learn here is not about making technical improvements, it’s to do something about improving the way we manage and maintain core open source infrastructure. Ref XKCD 2347.

lifthrasiir 3y ago
No, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster. [1] https://lists.debian.org/debian-devel-announce/2011/08/msg00001.html https://lists.debian.org/debian-devel-announce/2011/08/msg00...
- jl6 3y ago
  "almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature. If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
  - lifthrasiir 3y ago
    I will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice. [1] https://mattmahoney.net/dc/zpaq.html https://mattmahoney.net/dc/zpaq.html
pgeorgi 3y ago
The format is fine, except for https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html
- mort96 3y ago
  Yeah, that title is click bait. The format is perfectly fine for long term archival. In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...
  - jl6 3y ago
    That attack article puts forward a dubious scenario where parts of multiple copies of a file become corrupted in different locations and you need to recover the original using both of the damaged copies. If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario. A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.
dist-epoch 3y ago
It's not, zstd can match LZMA2 compression ratio, while being much faster.
- jl6 3y ago
  Much faster: yes. While matching LZMA2 compression ratio: no.

by jl6 3y ago

lifthrasiir 3y ago

No, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster. [1] https://lists.debian.org/debian-devel-announce/2011/08/msg00001.html https://lists.debian.org/debian-devel-announce/2011/08/msg00...

jl6 3y ago
"almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature. If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.
- lifthrasiir 3y ago
  I will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice. [1] https://mattmahoney.net/dc/zpaq.html https://mattmahoney.net/dc/zpaq.html

pgeorgi 3y ago

The format is fine, except for https://www.nongnu.org/lzip/xz_inadequate.html https://www.nongnu.org/lzip/xz_inadequate.html

mort96 3y ago
Yeah, that title is click bait. The format is perfectly fine for long term archival. In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...
- jl6 3y ago
  That attack article puts forward a dubious scenario where parts of multiple copies of a file become corrupted in different locations and you need to recover the original using both of the damaged copies. If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario. A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.

dist-epoch 3y ago

It's not, zstd can match LZMA2 compression ratio, while being much faster.

jl6 3y ago
Much faster: yes. While matching LZMA2 compression ratio: no.