6 ms·

Zlib – a spiffy yet delicately unobtrusive compression library

46 points by maydemir 5y ago

wongarsu 5y ago
The algorithm behind zlib is showing it's age, after all rfc1950 is 25 years old, and the deflate algorithm dates back to 33 year old pkzip. By now you get better compression at better speed from zstd, or vastly better compression at slower speed from lzma2, for basically anything you throw at it. But we still use zlib nearly everywhere, not least because it just works, no matter what language or hardware you use. It's a great piece of software and probably one of the most ubiquitous libraries out there.
- willis936 5y ago
  LZ dates back 45 years but is still high performance. I can only imagine that we've been saddled with deflate because it's computationally light. Well we have computation now and still enjoy using storage efficiently, so are (slowly, lazily) moving off deflate.
  - user-the-name 5y ago
    "LZ" is a vague family of algorithms, with a massively wide spectrum of performance and efficiency. Nearly every single modern compression algorithm falls into this family, with the only remotely popular exceptions I can think of being bzip2 and PPMd.
MauranKilom 5y ago
I feel obliged to point this out here: Do not build zlib with the hand-optimized assembly code. The ASM code is a contribution and does not receive any support by the library authors. There are bugs in there and it will read/write out of bounds given just the right data - we've been bit by it. Note also [0]. The default Makefile gets this right - it stays far away from any hand-rolled assembly (or contributions in general). But the contributed Visual Studio solutions default to building with the ASM: [1] [2]. Make sure to choose ReleaseWithoutAsm, or set up your own configuration! You can check any zlib build for whether it was built with ASM via (zlibCompileFlags() & (1 << 9)): [3] Needless to say, an OOB in a library as fundamental (and as user-data-facing) as zlib is a serious vulnerability. I didn't find any distribution or major software that uses a vulnerable zlib build in a day of investigation (well, I think Telegram Desktop in a debug build might be vulnerable [4]), but if you are on Windows you might accidentally make one. Beware! I'll also point interested readers towards zlib-ng [5] (without any endorsement or affiliation). [0]: https://github.com/madler/zlib/blob/master/inffast.c#L12 https://github.com/madler/zlib/blob/master/inffast.c#L12 [1]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/contrib/vstudio/vc14/testzlib.vcxproj#L362 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [2]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/contrib/vstudio/vc14/testzlib.vcxproj#L414 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [3]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/zlib.h#L1185 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [4]: https://github.com/telegramdesktop/tdesktop/blob/6a3ad52aef0b48ea53aa9533335b1d791547c1dd/Telegram/build/prepare/prepare.py#L459 https://github.com/telegramdesktop/tdesktop/blob/6a3ad52aef0... [5]: https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng
- ihattendorf 5y ago
  I'm surprised they don't rip it out if they don't want to spend the time to fix it. Or at least ensure nothing they provide builds it by default and you need to pass a `-DUSE_INSECURE_ASM` flag/`ReleaseWithInsecureAsm` in VS.
  - MauranKilom 5y ago
    Well, the VS solutions (and the asm code) are both "contrib", so Mark Adler does not assume responsibility for either (which is fair - he has been stewarding this library for 30 something years already...). It could be made clearer that contrib does not come with the same implied fitness for purpose as the main library, but then open source licenses already disavow any and all such fitness in the first place...
- injinj 5y ago
  It looks like the input word is using lodsd instruction, loading a 4 byte word, instead of a ptr increment, which could cause a 3 byte oob read if the last byte was at address 1. I believe this is used in millions of builds, since it's been around for 20 years and Windows has a culture of winzip style software not packaged by Microsoft, but a lone programmer trying to get by with donations. Do you have a specific exploit description? Mark Adler can remove or mark it unsafe on the github repo given a reason.
dharbin 5y ago
As ubiquitous as zlib is, I wonder why it’s not implemented in hardware like many audio and video codecs are.
- wbl 5y ago
  It can be. But searching a large dictionary for the longest match is a pain. Audio and video codecs tend to have a much smaller working set or lots of parallelism for hardware depending on what stage of the process they are in.
- erk__ 5y ago
  It is implemented in hardware on IBM's mainframes, although not upstreamed in the main zlib repo [0]. It gives a massive speedup in my experience up to 20x for decompression and 220x (!) for compression [0]: https://github.com/madler/zlib/pull/410 https://github.com/madler/zlib/pull/410
- wolf550e 5y ago
  deflate is implemented in hardware on game consoles. PS5, in addition to zlib, has hardware [0][2] decompressor for RAD Oodle Kraken [1], which is of course much better than zlib. If at all possible, especially if you control both ends, please use zstandard[3] instead of zlib/deflate/gzip/zip for free general purpose compression. It is more advanced by about 30 years of technology development. It always spends less time to produce smaller files which are then decompressed insanely fast. There is no tradeoff, it is always better. It has a very wide range of compression levels you can choose from if the good default doesn't fit your need. It also supports multithreading, custom dictionaries and long range compression out of the box. 0 - https://www.tweaktown.com/news/71340/understanding-the-ps5s-ssd-deep-dive-into-next-gen-storage-tech/index.html https://www.tweaktown.com/news/71340/understanding-the-ps5s-... 1 - http://www.radgametools.com/oodlekraken.htm http://www.radgametools.com/oodlekraken.htm 2 - https://twitter.com/rygorous/status/1240341184867758085 https://twitter.com/rygorous/status/1240341184867758085 3 - https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
  - mananaysiempre 5y ago
imaurer 5y ago
Is zlib the most installed software component in the world? Or is it curl or sqlite. Interesting blog post about this topic: https://daniel.haxx.se/blog/2021/10/21/the-most-used-software-components-in-the-world/ https://daniel.haxx.se/blog/2021/10/21/the-most-used-softwar...
- deleted 5y ago
  [deleted]
- klmr 5y ago
  Next week on HN: “curl: A command line tool and library for transferring data with URL syntax”. Anyway, the submission led to several very interesting technical comments so I don’t hate it.
zmodem 5y ago
I'm never quite sure about the state of zlib. There are commits going in [0] but at the same time pull requests with fixes aren't seeing much traction (e.g. [1]). Optimization work happens in forks like Intel's [2], Cloudflare's [3], and Chromium's [4], the latter of which also has a lot of added tests and fuzzers [5]. And there's zlib-ng [6] which tries to create a home for many of the improvements, but is at the same time yet another fork. [0] https://github.com/madler/zlib/commits/develop https://github.com/madler/zlib/commits/develop [1] https://github.com/madler/zlib/pull/525 https://github.com/madler/zlib/pull/525 [2] https://github.com/jtkukunas/zlib https://github.com/jtkukunas/zlib [3] https://github.com/cloudflare/zlib https://github.com/cloudflare/zlib [4] https://source.chromium.org/chromium/chromium/src/+/main:third_party/zlib/ https://source.chromium.org/chromium/chromium/src/+/main:thi... [5] https://source.chromium.org/chromium/chromium/src/+/main:third_party/zlib/contrib/tests/fuzzers/ https://source.chromium.org/chromium/chromium/src/+/main:thi... [6] https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng (edit: line breaks)
- mananaysiempre 5y ago
  Going away from Zlib proper but staying with its format, there’s libdeflate[1], an implementation that’s substantially faster on modern machines but can’t stream (although I see no reason why it couldn’t, probably just for simplicity due to its origins as a baseline fast compressor[2]), and of course Zopfli[3], a glacially slow compressor that’s somewhat (and occasionally a lot) better. [1] https://github.com/ebiggers/libdeflate https://github.com/ebiggers/libdeflate [2] https://encode.su/threads/2482-libdeflate-a-new-optimized-DEFLATE-implementation https://encode.su/threads/2482-libdeflate-a-new-optimized-DE... [3] https://github.com/google/zopfli https://github.com/google/zopfli
- melissalobos 5y ago
  I have tried using zlib-ng on a project before, but I found it had too many issues when trying to deploy it. Almost every distro packs zlib, and zlib is so fundamental you can't just wholesale replace it, so you wind up with an almost the same library and regular zlib. I can't remember the exact details(maybe something with RPATH?), but we had to wind up just using the system zlib due to deployment issues.
tzs 5y ago
Zlib is great. I used to use it all the time ~25 years ago. I don't directly use it much now but only because it has been a long time since I worked on anything where I had to deal with compression myself. The only time I ever had a problem with it was on a project to implement a virtual CD-ROM drive for a client. The CD-ROM images needed to be compressed, and the client wanted the user of the ripper to be able to choose between more compression and more speed. Zlib supports different compression levels, but the speed increase from less compression wasn't as much as the client wanted. I found an implementation of zlib that purported to be faster due to hand optimized assembler code. I tested that and it was faster, perhaps enough that the client would find it acceptable. But we would need a commercial license for it, and it was open source. I contacted the author and he was willing to sell a commercial license to allow us to use it in all our products (generally utility software for Windows and Mac). We had an acceptable price worked out and were about to conclude the deal when he got cold feet. He started thinking that since we wrote Windows software and were a small company, maybe Microsoft would decide to acquire us, and then Microsoft would have a commercial license for his zlib. He wanted to do a contract where the license would terminate if the company changed ownership. That would require getting my boss (the founder and CEO of the company) involved to work out the details and he didn't have time for that. I tried doubling the amount of money we were offering for the license to see if that would change the library author's mind but it did not. My boss told me to find another solution, and to find one quickly because this was an urgent project for the client (for a completely stupid reason [1]). I found one. 25 years later I still can't decide if this was the most awesome hack I ever did or the most profoundly shameful thing I've ever written. I added a slider to the ripper UI that went from 0 to 100. As the ripper ripped sectors, it either stored the sector in the image compressed with zlib at maximum compression level or it stored the sector uncompressed. The slider controlled what percent of the sectors were compressed. The client was thrilled with this. [1] The client was a major software company in Japan. Call them X. There was a small software company Y whose products did not overlap with those of X. X thought that a press release for a new product of Y's plagiarized one of X's press releases, were greatly offended by this, and so decided that they wanted to smash Y by directly competing with one of Y's best sellers. That was a virtual CD-ROM, so X told us to put the several other things we were doing for them on hold and quickly provide them with a virtual CD-ROM product and make it better than Y's.
okokwhatever 5y ago
Object not found! on the rfc doc linked in the README. What a shame RFC docs are so badly shared online this days. Shame on us!
- BenjiWiebe 5y ago
  Looks like the link format changed. https://datatracker.ietf.org/doc/html/rfc1950 https://datatracker.ietf.org/doc/html/rfc1950

46 points by maydemir 5y ago

The algorithm behind zlib is showing it's age, after all rfc1950 is 25 years old, and the deflate algorithm dates back to 33 year old pkzip. By now you get better compression at better speed from zstd, or vastly better compression at slower speed from lzma2, for basically anything you throw at it. But we still use zlib nearly everywhere, not least because it just works, no matter what language or hardware you use. It's a great piece of software and probably one of the most ubiquitous libraries out there.

willis936 5y ago
LZ dates back 45 years but is still high performance. I can only imagine that we've been saddled with deflate because it's computationally light. Well we have computation now and still enjoy using storage efficiently, so are (slowly, lazily) moving off deflate.
- user-the-name 5y ago
  "LZ" is a vague family of algorithms, with a massively wide spectrum of performance and efficiency. Nearly every single modern compression algorithm falls into this family, with the only remotely popular exceptions I can think of being bzip2 and PPMd.

MauranKilom 5y ago

I feel obliged to point this out here: Do not build zlib with the hand-optimized assembly code. The ASM code is a contribution and does not receive any support by the library authors. There are bugs in there and it will read/write out of bounds given just the right data - we've been bit by it. Note also [0]. The default Makefile gets this right - it stays far away from any hand-rolled assembly (or contributions in general). But the contributed Visual Studio solutions default to building with the ASM: [1] [2]. Make sure to choose ReleaseWithoutAsm, or set up your own configuration! You can check any zlib build for whether it was built with ASM via (zlibCompileFlags() & (1 << 9)): [3] Needless to say, an OOB in a library as fundamental (and as user-data-facing) as zlib is a serious vulnerability. I didn't find any distribution or major software that uses a vulnerable zlib build in a day of investigation (well, I think Telegram Desktop in a debug build might be vulnerable [4]), but if you are on Windows you might accidentally make one. Beware! I'll also point interested readers towards zlib-ng [5] (without any endorsement or affiliation). [0]: https://github.com/madler/zlib/blob/master/inffast.c#L12 https://github.com/madler/zlib/blob/master/inffast.c#L12 [1]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/contrib/vstudio/vc14/testzlib.vcxproj#L362 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [2]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/contrib/vstudio/vc14/testzlib.vcxproj#L414 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [3]: https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da3b647f07d718623f/zlib.h#L1185 https://github.com/madler/zlib/blob/cacf7f1d4e3d44d871b605da... [4]: https://github.com/telegramdesktop/tdesktop/blob/6a3ad52aef0b48ea53aa9533335b1d791547c1dd/Telegram/build/prepare/prepare.py#L459 https://github.com/telegramdesktop/tdesktop/blob/6a3ad52aef0... [5]: https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng

ihattendorf 5y ago
I'm surprised they don't rip it out if they don't want to spend the time to fix it. Or at least ensure nothing they provide builds it by default and you need to pass a `-DUSE_INSECURE_ASM` flag/`ReleaseWithInsecureAsm` in VS.
- MauranKilom 5y ago
  Well, the VS solutions (and the asm code) are both "contrib", so Mark Adler does not assume responsibility for either (which is fair - he has been stewarding this library for 30 something years already...). It could be made clearer that contrib does not come with the same implied fitness for purpose as the main library, but then open source licenses already disavow any and all such fitness in the first place...
injinj 5y ago
It looks like the input word is using lodsd instruction, loading a 4 byte word, instead of a ptr increment, which could cause a 3 byte oob read if the last byte was at address 1. I believe this is used in millions of builds, since it's been around for 20 years and Windows has a culture of winzip style software not packaged by Microsoft, but a lone programmer trying to get by with donations. Do you have a specific exploit description? Mark Adler can remove or mark it unsafe on the github repo given a reason.

dharbin 5y ago

As ubiquitous as zlib is, I wonder why it’s not implemented in hardware like many audio and video codecs are.

wbl 5y ago
It can be. But searching a large dictionary for the longest match is a pain. Audio and video codecs tend to have a much smaller working set or lots of parallelism for hardware depending on what stage of the process they are in.
erk__ 5y ago
It is implemented in hardware on IBM's mainframes, although not upstreamed in the main zlib repo [0]. It gives a massive speedup in my experience up to 20x for decompression and 220x (!) for compression [0]: https://github.com/madler/zlib/pull/410 https://github.com/madler/zlib/pull/410
wolf550e 5y ago
deflate is implemented in hardware on game consoles. PS5, in addition to zlib, has hardware [0][2] decompressor for RAD Oodle Kraken [1], which is of course much better than zlib. If at all possible, especially if you control both ends, please use zstandard[3] instead of zlib/deflate/gzip/zip for free general purpose compression. It is more advanced by about 30 years of technology development. It always spends less time to produce smaller files which are then decompressed insanely fast. There is no tradeoff, it is always better. It has a very wide range of compression levels you can choose from if the good default doesn't fit your need. It also supports multithreading, custom dictionaries and long range compression out of the box. 0 - https://www.tweaktown.com/news/71340/understanding-the-ps5s-ssd-deep-dive-into-next-gen-storage-tech/index.html https://www.tweaktown.com/news/71340/understanding-the-ps5s-... 1 - http://www.radgametools.com/oodlekraken.htm http://www.radgametools.com/oodlekraken.htm 2 - https://twitter.com/rygorous/status/1240341184867758085 https://twitter.com/rygorous/status/1240341184867758085 3 - https://facebook.github.io/zstd/ https://facebook.github.io/zstd/
- mananaysiempre 5y ago

imaurer 5y ago

Is zlib the most installed software component in the world? Or is it curl or sqlite. Interesting blog post about this topic: https://daniel.haxx.se/blog/2021/10/21/the-most-used-software-components-in-the-world/ https://daniel.haxx.se/blog/2021/10/21/the-most-used-softwar...

deleted 5y ago
[deleted]
klmr 5y ago
Next week on HN: “curl: A command line tool and library for transferring data with URL syntax”. Anyway, the submission led to several very interesting technical comments so I don’t hate it.

zmodem 5y ago

I'm never quite sure about the state of zlib. There are commits going in [0] but at the same time pull requests with fixes aren't seeing much traction (e.g. [1]). Optimization work happens in forks like Intel's [2], Cloudflare's [3], and Chromium's [4], the latter of which also has a lot of added tests and fuzzers [5]. And there's zlib-ng [6] which tries to create a home for many of the improvements, but is at the same time yet another fork. [0] https://github.com/madler/zlib/commits/develop https://github.com/madler/zlib/commits/develop [1] https://github.com/madler/zlib/pull/525 https://github.com/madler/zlib/pull/525 [2] https://github.com/jtkukunas/zlib https://github.com/jtkukunas/zlib [3] https://github.com/cloudflare/zlib https://github.com/cloudflare/zlib [4] https://source.chromium.org/chromium/chromium/src/+/main:third_party/zlib/ https://source.chromium.org/chromium/chromium/src/+/main:thi... [5] https://source.chromium.org/chromium/chromium/src/+/main:third_party/zlib/contrib/tests/fuzzers/ https://source.chromium.org/chromium/chromium/src/+/main:thi... [6] https://github.com/zlib-ng/zlib-ng https://github.com/zlib-ng/zlib-ng (edit: line breaks)

mananaysiempre 5y ago
Going away from Zlib proper but staying with its format, there’s libdeflate[1], an implementation that’s substantially faster on modern machines but can’t stream (although I see no reason why it couldn’t, probably just for simplicity due to its origins as a baseline fast compressor[2]), and of course Zopfli[3], a glacially slow compressor that’s somewhat (and occasionally a lot) better. [1] https://github.com/ebiggers/libdeflate https://github.com/ebiggers/libdeflate [2] https://encode.su/threads/2482-libdeflate-a-new-optimized-DEFLATE-implementation https://encode.su/threads/2482-libdeflate-a-new-optimized-DE... [3] https://github.com/google/zopfli https://github.com/google/zopfli
melissalobos 5y ago
I have tried using zlib-ng on a project before, but I found it had too many issues when trying to deploy it. Almost every distro packs zlib, and zlib is so fundamental you can't just wholesale replace it, so you wind up with an almost the same library and regular zlib. I can't remember the exact details(maybe something with RPATH?), but we had to wind up just using the system zlib due to deployment issues.

tzs 5y ago

Zlib is great. I used to use it all the time ~25 years ago. I don't directly use it much now but only because it has been a long time since I worked on anything where I had to deal with compression myself. The only time I ever had a problem with it was on a project to implement a virtual CD-ROM drive for a client. The CD-ROM images needed to be compressed, and the client wanted the user of the ripper to be able to choose between more compression and more speed. Zlib supports different compression levels, but the speed increase from less compression wasn't as much as the client wanted. I found an implementation of zlib that purported to be faster due to hand optimized assembler code. I tested that and it was faster, perhaps enough that the client would find it acceptable. But we would need a commercial license for it, and it was open source. I contacted the author and he was willing to sell a commercial license to allow us to use it in all our products (generally utility software for Windows and Mac). We had an acceptable price worked out and were about to conclude the deal when he got cold feet. He started thinking that since we wrote Windows software and were a small company, maybe Microsoft would decide to acquire us, and then Microsoft would have a commercial license for his zlib. He wanted to do a contract where the license would terminate if the company changed ownership. That would require getting my boss (the founder and CEO of the company) involved to work out the details and he didn't have time for that. I tried doubling the amount of money we were offering for the license to see if that would change the library author's mind but it did not. My boss told me to find another solution, and to find one quickly because this was an urgent project for the client (for a completely stupid reason [1]). I found one. 25 years later I still can't decide if this was the most awesome hack I ever did or the most profoundly shameful thing I've ever written. I added a slider to the ripper UI that went from 0 to 100. As the ripper ripped sectors, it either stored the sector in the image compressed with zlib at maximum compression level or it stored the sector uncompressed. The slider controlled what percent of the sectors were compressed. The client was thrilled with this. [1] The client was a major software company in Japan. Call them X. There was a small software company Y whose products did not overlap with those of X. X thought that a press release for a new product of Y's plagiarized one of X's press releases, were greatly offended by this, and so decided that they wanted to smash Y by directly competing with one of Y's best sellers. That was a virtual CD-ROM, so X told us to put the several other things we were doing for them on hold and quickly provide them with a virtual CD-ROM product and make it better than Y's.

okokwhatever 5y ago

Object not found! on the rfc doc linked in the README. What a shame RFC docs are so badly shared online this days. Shame on us!

BenjiWiebe 5y ago
Looks like the link format changed. https://datatracker.ietf.org/doc/html/rfc1950 https://datatracker.ietf.org/doc/html/rfc1950