3 ms·

Migrating to zstd instead of Lzip seems an odd decision. Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd woul

Migrating to zstd instead of Lzip seems an odd decision. Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger. Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions.

Sakos 3y ago
It looks like Lzip only has a single maintainer and there's no public repository to track commits or contributors (at least I couldn't find any). I don't think it's a viable option.
fabian2k 3y ago
I think it makes sense to use a commonly used format here, the maintenance aspects are very important as this incident has shown. And I don't think the tradeoffs between compressed size and speed are entirely obvious. Trading off slightly larger files for easier maintenance would seem like a defensible choice to me.
arp242 3y ago
It doesn't even have a git/mercurial/CVS repo; just .tar.gz on some random website. And installation instructions are: tar -xf lzip[version].tar.gz cd lzip[version] && ./configure && make check The entire thing seems entirely the work by one person. Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.
- rurban 3y ago
  Absolutely not. Extecatly this recipe brought us this desaster. The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
  - throwaway7356 3y ago
    > because such a malicious m4 would have been detected By the xz maintainer? I very much doubt that.
pwdisswordfishc 3y ago
It's not like the algorithm itself is the most valuable part here, as opposed to active maintainership.
lifthrasiir 3y ago
> Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger. Absolutely false because Zstandard didn't exist when xz was initially chosen. And the actual ratio difference is not that huge, generally less than 10% for most use cases, while the fastest level of xz is slower than most levels of zstd. (In fact, `dpkg-deb` uses `xz -6` by default because `-9` would be really expensive.) Given that zstd is also faster to decompress than xz as well, the ~10% increase is justifiable. > Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions. Lzip is not a better-designed format, it is just needlessly minimal for its use cases, much like xz's needless extensibility. I have detailed my own criticisms in the other comment [1]. [1] https://news.ycombinator.com/item?id=39873122 https://news.ycombinator.com/item?id=39873122
dist-epoch 3y ago
zstd at high compression levels (19+) with large dictionaries (32 MB) can match LZMA2 compression level.
- silotis 3y ago
  Almost. I tested this a few years ago and IIRC zstd at its highest settings is typically about 5% larger than LZMA. So there is a difference, but probably not enough for most people to care.
  - dralley 3y ago
    I tested recently and (on that selection of data) zstd had better maximum compression at the highest level but took about 3x as long to compress at that level. At most other levels it's faster than xz with slightly worse compression, but not by much.

Sakos 3y ago

It looks like Lzip only has a single maintainer and there's no public repository to track commits or contributors (at least I couldn't find any). I don't think it's a viable option.

fabian2k 3y ago

I think it makes sense to use a commonly used format here, the maintenance aspects are very important as this incident has shown. And I don't think the tradeoffs between compressed size and speed are entirely obvious. Trading off slightly larger files for easier maintenance would seem like a defensible choice to me.

arp242 3y ago

It doesn't even have a git/mercurial/CVS repo; just .tar.gz on some random website. And installation instructions are: tar -xf lzip[version].tar.gz cd lzip[version] && ./configure && make check The entire thing seems entirely the work by one person. Yeah, this is going to massively improve things. I foresee no plausible way we can gave a repeat if we switch to this lzip thing. Absolutely watertight. We learned from past events. Problem solved.

rurban 3y ago
Absolutely not. Extecatly this recipe brought us this desaster. The backdoor was in the released tar only, the sources were fine, because such a malicious m4 would have been detected. The compressed test bin containing the backdoor was in git, but such unobvious things do get attention.
- throwaway7356 3y ago
  > because such a malicious m4 would have been detected By the xz maintainer? I very much doubt that.

pwdisswordfishc 3y ago

It's not like the algorithm itself is the most valuable part here, as opposed to active maintainership.

lifthrasiir 3y ago

> Presumably xz was chosen to begin with because of its high level of compression, migrating to zstd would make every single package considerably larger. Absolutely false because Zstandard didn't exist when xz was initially chosen. And the actual ratio difference is not that huge, generally less than 10% for most use cases, while the fastest level of xz is slower than most levels of zstd. (In fact, `dpkg-deb` uses `xz -6` by default because `-9` would be really expensive.) Given that zstd is also faster to decompress than xz as well, the ~10% increase is justifiable. > Lzip uses the same compression algorithm as xz, but is otherwise a simpler and better-designed format, without any of xz's bizarre countainer-inside-container design decisions. Lzip is not a better-designed format, it is just needlessly minimal for its use cases, much like xz's needless extensibility. I have detailed my own criticisms in the other comment [1]. [1] https://news.ycombinator.com/item?id=39873122 https://news.ycombinator.com/item?id=39873122

dist-epoch 3y ago

zstd at high compression levels (19+) with large dictionaries (32 MB) can match LZMA2 compression level.

silotis 3y ago
Almost. I tested this a few years ago and IIRC zstd at its highest settings is typically about 5% larger than LZMA. So there is a difference, but probably not enough for most people to care.
- dralley 3y ago
  I tested recently and (on that selection of data) zstd had better maximum compression at the highest level but took about 3x as long to compress at that level. At most other levels it's faster than xz with slightly worse compression, but not by much.