11 ms·

How are zlib, gzip and zip related?

300 points by damagednoob 3y ago

exposition 3y ago
There's also pzip/punzip (https://github.com/ybirader https://github.com/ybirader) for those wanting more performant (concurrent) zip/unzip. Disclaimer: I'm the author.
kissgyorgy 3y ago
If you are interested in implementation details, how to unpack/decompress them, check out these Python implementations: - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/archive/zip.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/compression/gzip.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/compression/zlib.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle...
Dwedit 3y ago
See a highly upvoted answer in a question about zlib related things, suspect it was probably posted by Mark Adler, and turn out to be correct.
eYrKEC2 3y ago
[flagged]
readyplayernull 3y ago
gzip can be used to (de)compress directories recursively in a variable: FOO=$(tar cf - folderToCompress | gzip | base64) echo $FOO | base64 - d | zcat | tar xf -
melagonster 3y ago
For people who first read this: the sweet part is in the comments :)
- stavros 3y ago
  Hah, imagine asking Mark Adler for gzip history references.
- dcow 3y ago
  What’s even more sad is that the SO community has since consequently destroyed SO as the home for this type of info. This post would now be considered off topic as it’s “not a good format for a Q&A site”. You’d never see it happen today. Truly sad.
  - barrkel 3y ago
    Thing is, it could only be that way in its early days, when the vanguard of users came to it from word of mouth, from following Joel Spolsky or Coding Horror or their joint podcast. The audience is much bigger now and with the long tail of people, the number willing to put effort into good questions is too low, and on-topicness is a simple quality bar which can improve the signal to noise ratio.
    - dcow 3y ago
      Except, I doubt anybody would argue that a lower signal to noise ratio has improved the site. (Plus, has the actual metric even improved and how is it measured?) And, did anybody ever stop to ask whether S:N should even be the champion metric in the first place, at a product level? With a philosophy of “Google is our homepage”, I honestly don’t understand why S:N even matters since search pretty effectively cuts out noise. I guess it makes a mod’s life easier though. The site is less useful today than it’s ever been. The road to hell…
    - Dalewyn 3y ago
      Very broadly, I find the quality/value of a given thing is inversely proportional to how many people are involved. So with regards to the internet: The 90s and early 00s were great, then the internet became mainstream and it all just became Cable TV 2.0.
    - s_dev 3y ago
ctur 3y ago
What a great historical summary. Compression has moved on now but having grown up marveling at PKZip and maximizing usable space on very early computers, as well as compression in modems (v42bis ftw!), this field has always seemed magical. These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. And if you need seekable format, consider squashfs as a reasonable choice. These stand on the shoulders of the giants of zlib and zip but do indeed stand much higher in the modern world.
- lxgr 3y ago
  > These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. I'd agree for new applications, but just like MP3, .gz files (and by extension .tar.gz/.tgz) and zlib streams will probably be around for a long time for compatibility reasons.
- pvorb 3y ago
  I think zlib/gzip still has its place these days. It's still a decent choice for most use cases. If you don't know what usage patterns your program will see, zlib still might be a good choice. Plus, it's supported virtually everywhere, which makes it interesting for long-term storage. Often, using one of the modern alternatives is not worth the hassle.
- michaelrpeskin 3y ago
  I had forgotten about modem compression. Back in the BBS days when you had to upload files to get new files, you usually had a ratio (20 bytes download for every byte you uploaded). I would always use the PKZIP no compression option for the archive to upload because Z-Modem would take care of compression over the wire. So I didn't burn my daily time limit by uploading a large file and I got more credit for my download ratios. I was a silly kid.
  - stavros 3y ago
    That sounds like it can be fooled by making a zip bomb that will compress down to a few KB (by the modem), but will be many MB uncompressed. Sounds great for your ratio, and will upload in a few seconds.
dustypotato 3y ago
Found this hilarious: > This post is packed with so much history and information that I feel like some citations need be added > I am the reference (extracted a part of the conversation)
- FartyMcFarter 3y ago
  "I'm the one who knocks".
  - matheusmoreira 3y ago
    "I am the hype."
- tyingq 3y ago
  Maybe a spoiler, but the "I" in "I am the reference" is Mark Adler: https://en.wikipedia.org/wiki/Mark_Adler https://en.wikipedia.org/wiki/Mark_Adler
  - signaru 3y ago
    It's awesome how he is active on stack overflow for almost anything DEFLATE related. I once tried stuffing deflate compressed vector graphics into PDFs. Among other things, it turns out an Adler-32 checksum is necessary for compliance (some newer PDF viewers will ignore its absence though).
- whalesalad 3y ago
  Reminds me of when I was inadvertently arguing here on HN with the inventor of the actor model about what actors are
  - chupasaurus 3y ago
    Oh, I remember that conversation, it was fun.
  - demondemidi 3y ago
wiredfool 3y ago
The real question is: how are zlib and libz related?
- o11c 3y ago
  zlib is the name of the project. libz is an implementation-detail name of the library on Unix-like systems.
  - pdw 3y ago
    Similar: Xlib and libX11.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
cout 3y ago
Interesting -- I did not realize that the zip format supports lzma, bzip2, and zstd. What software supports those compression methods? Can Windows Explorer read zip files produced with those compression methods? (I have been using 7zip for about 15 years to produce archive files that have an index and can quickly extract a single file and can use multiple cores for compression, but I would love to have an alternative, if one exists).
- ForkMeOnTinder 3y ago
  7zip has a dropdown called "Compression method" in the "Add to Archive" dialog that lets you choose.
- pixl97 3y ago
  Until windows 11, no, windows zip only seems to deal with COMPRESS/DEFLATE zip files.
encom 3y ago
(2013)
emmelaich 3y ago
Fun fact: in a sense. gzip can have multiple files, but not in a specially useful way ... $ echo meow >cat $ echo woof > dog $ gzip cat $ gzip dog $ cat cat.gz dog.gz >animals.gz $ gunzip animals.gz $ cat animals meow woof
- lxgr 3y ago
  Wow, that's surprising (at least to me)! Is there a limit in the default gunzip implementation? I'm aware of the concept of ZIP/tar bombs, but I wouldn't have expected gunzip to ever produce more than one output file, at least when invoked without options.
  - tedunangst 3y ago
    It only produces one output. It's just a stream of data.
    - lxgr 3y ago
      Ah, I somehow imagined a second `cat` in there. That makes more sense, thank you!
  - abhibeckert 3y ago
    The limit is it doesn't do filenames or other metadata — it's limited to contents.
- billyhoffman 3y ago
  WARC files (used by the Internet Archive to power the Wayback machine, among others) use this trick too to have a a compressed file format that is seek-able to individual HTTP request/response records
HexDecOctBin 3y ago
Is there an archive format that supports appending diff's of an existing file, so that multiple versions of the same file are stored? PKZIP has a proprietary extension (supposedly), but I couldn't find any open version of that. (I was thinking of a creating a version control system whose .git directory equivalent is basically an archive file that can easily be emailed, etc.)
- pizza 3y ago
  New versions of zstd allow you to produce patches using the trained dictionary feature
raggi 3y ago
The answer is good, but is missing a key section: Salty form: They're all quite slow compared to modern competitors.
- levzettelin 3y ago
  What are some of those modern competitors?
  - raggi 3y ago
    For zlib compatible workloads, there are cloudflare patches and chromium forks, intel forks, and zlib-ng which are compatible but >50% faster. (I think the cloudflare patches eventually made it into upstream zlib, but you may not see that in your distro for a decade). lz4 and zstd have both been very popular since their release, they're similar and by the same author, though zstd has had more thorough testing and fuzzing, and is more featureful. lz4 maintains an extremely fast decompression speed. Snappy also performs very well, with zstd and snappy having very close performance with tuning to achieve comparable compression levels. In recent years Zstd has started to make heavy inroads in broader usage in OSS with a number of distro package managers moving to it and observing substantial benefits. There are HTTP extensions to make it available which Chrome originally resisted but I believe it's now finally coming there too (https://chromestatus.com/feature/6186023867908096 https://chromestatus.com/feature/6186023867908096). In gaming circles there's also Oodle and friends from RAD tools which are now available in Unreal engine as builtin compression offerings (since 4.27+). You could see the effects of this in for example Ark Survival Evolved (250GB) -> Ark Survival Ascended (75GB, with richer models & textures), and associated improved load times.
  - scq 3y ago
    zstd is over 4x faster than zlib, while having a better compression ratio. http://facebook.github.io/zstd/ http://facebook.github.io/zstd/

300 points by damagednoob 3y ago

There's also pzip/punzip (https://github.com/ybirader https://github.com/ybirader) for those wanting more performant (concurrent) zip/unzip. Disclaimer: I'm the author.

kissgyorgy 3y ago

If you are interested in implementation details, how to unpack/decompress them, check out these Python implementations: - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/archive/zip.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/compression/gzip.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handlers/compression/zlib.py https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

Dwedit 3y ago

See a highly upvoted answer in a question about zlib related things, suspect it was probably posted by Mark Adler, and turn out to be correct.

eYrKEC2 3y ago

[flagged]

readyplayernull 3y ago

melagonster 3y ago

For people who first read this: the sweet part is in the comments :)

stavros 3y ago
Hah, imagine asking Mark Adler for gzip history references.
dcow 3y ago
What’s even more sad is that the SO community has since consequently destroyed SO as the home for this type of info. This post would now be considered off topic as it’s “not a good format for a Q&A site”. You’d never see it happen today. Truly sad.
- barrkel 3y ago
  Thing is, it could only be that way in its early days, when the vanguard of users came to it from word of mouth, from following Joel Spolsky or Coding Horror or their joint podcast. The audience is much bigger now and with the long tail of people, the number willing to put effort into good questions is too low, and on-topicness is a simple quality bar which can improve the signal to noise ratio.
  - dcow 3y ago
    Except, I doubt anybody would argue that a lower signal to noise ratio has improved the site. (Plus, has the actual metric even improved and how is it measured?) And, did anybody ever stop to ask whether S:N should even be the champion metric in the first place, at a product level? With a philosophy of “Google is our homepage”, I honestly don’t understand why S:N even matters since search pretty effectively cuts out noise. I guess it makes a mod’s life easier though. The site is less useful today than it’s ever been. The road to hell…
  - Dalewyn 3y ago
    Very broadly, I find the quality/value of a given thing is inversely proportional to how many people are involved. So with regards to the internet: The 90s and early 00s were great, then the internet became mainstream and it all just became Cable TV 2.0.
  - s_dev 3y ago

ctur 3y ago

What a great historical summary. Compression has moved on now but having grown up marveling at PKZip and maximizing usable space on very early computers, as well as compression in modems (v42bis ftw!), this field has always seemed magical. These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. And if you need seekable format, consider squashfs as a reasonable choice. These stand on the shoulders of the giants of zlib and zip but do indeed stand much higher in the modern world.

lxgr 3y ago
> These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. I'd agree for new applications, but just like MP3, .gz files (and by extension .tar.gz/.tgz) and zlib streams will probably be around for a long time for compatibility reasons.
pvorb 3y ago
I think zlib/gzip still has its place these days. It's still a decent choice for most use cases. If you don't know what usage patterns your program will see, zlib still might be a good choice. Plus, it's supported virtually everywhere, which makes it interesting for long-term storage. Often, using one of the modern alternatives is not worth the hassle.
michaelrpeskin 3y ago
I had forgotten about modem compression. Back in the BBS days when you had to upload files to get new files, you usually had a ratio (20 bytes download for every byte you uploaded). I would always use the PKZIP no compression option for the archive to upload because Z-Modem would take care of compression over the wire. So I didn't burn my daily time limit by uploading a large file and I got more credit for my download ratios. I was a silly kid.
- stavros 3y ago
  That sounds like it can be fooled by making a zip bomb that will compress down to a few KB (by the modem), but will be many MB uncompressed. Sounds great for your ratio, and will upload in a few seconds.

dustypotato 3y ago

Found this hilarious: > This post is packed with so much history and information that I feel like some citations need be added > I am the reference (extracted a part of the conversation)

FartyMcFarter 3y ago
"I'm the one who knocks".
- matheusmoreira 3y ago
  "I am the hype."
tyingq 3y ago
Maybe a spoiler, but the "I" in "I am the reference" is Mark Adler: https://en.wikipedia.org/wiki/Mark_Adler https://en.wikipedia.org/wiki/Mark_Adler
- signaru 3y ago
  It's awesome how he is active on stack overflow for almost anything DEFLATE related. I once tried stuffing deflate compressed vector graphics into PDFs. Among other things, it turns out an Adler-32 checksum is necessary for compliance (some newer PDF viewers will ignore its absence though).
whalesalad 3y ago
Reminds me of when I was inadvertently arguing here on HN with the inventor of the actor model about what actors are
- chupasaurus 3y ago
  Oh, I remember that conversation, it was fun.
- demondemidi 3y ago

wiredfool 3y ago

The real question is: how are zlib and libz related?

o11c 3y ago
zlib is the name of the project. libz is an implementation-detail name of the library on Unix-like systems.
- pdw 3y ago
  Similar: Xlib and libX11.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

cout 3y ago

Interesting -- I did not realize that the zip format supports lzma, bzip2, and zstd. What software supports those compression methods? Can Windows Explorer read zip files produced with those compression methods? (I have been using 7zip for about 15 years to produce archive files that have an index and can quickly extract a single file and can use multiple cores for compression, but I would love to have an alternative, if one exists).

ForkMeOnTinder 3y ago
7zip has a dropdown called "Compression method" in the "Add to Archive" dialog that lets you choose.
pixl97 3y ago
Until windows 11, no, windows zip only seems to deal with COMPRESS/DEFLATE zip files.

encom 3y ago

(2013)

emmelaich 3y ago

Fun fact: in a sense. gzip can have multiple files, but not in a specially useful way ... $ echo meow >cat $ echo woof > dog $ gzip cat $ gzip dog $ cat cat.gz dog.gz >animals.gz $ gunzip animals.gz $ cat animals meow woof

lxgr 3y ago
Wow, that's surprising (at least to me)! Is there a limit in the default gunzip implementation? I'm aware of the concept of ZIP/tar bombs, but I wouldn't have expected gunzip to ever produce more than one output file, at least when invoked without options.
- tedunangst 3y ago
  It only produces one output. It's just a stream of data.
  - lxgr 3y ago
    Ah, I somehow imagined a second `cat` in there. That makes more sense, thank you!
- abhibeckert 3y ago
  The limit is it doesn't do filenames or other metadata — it's limited to contents.
billyhoffman 3y ago
WARC files (used by the Internet Archive to power the Wayback machine, among others) use this trick too to have a a compressed file format that is seek-able to individual HTTP request/response records

HexDecOctBin 3y ago

Is there an archive format that supports appending diff's of an existing file, so that multiple versions of the same file are stored? PKZIP has a proprietary extension (supposedly), but I couldn't find any open version of that. (I was thinking of a creating a version control system whose .git directory equivalent is basically an archive file that can easily be emailed, etc.)

pizza 3y ago
New versions of zstd allow you to produce patches using the trained dictionary feature

raggi 3y ago

The answer is good, but is missing a key section: Salty form: They're all quite slow compared to modern competitors.

levzettelin 3y ago
What are some of those modern competitors?
- raggi 3y ago
  For zlib compatible workloads, there are cloudflare patches and chromium forks, intel forks, and zlib-ng which are compatible but >50% faster. (I think the cloudflare patches eventually made it into upstream zlib, but you may not see that in your distro for a decade). lz4 and zstd have both been very popular since their release, they're similar and by the same author, though zstd has had more thorough testing and fuzzing, and is more featureful. lz4 maintains an extremely fast decompression speed. Snappy also performs very well, with zstd and snappy having very close performance with tuning to achieve comparable compression levels. In recent years Zstd has started to make heavy inroads in broader usage in OSS with a number of distro package managers moving to it and observing substantial benefits. There are HTTP extensions to make it available which Chrome originally resisted but I believe it's now finally coming there too (https://chromestatus.com/feature/6186023867908096 https://chromestatus.com/feature/6186023867908096). In gaming circles there's also Oodle and friends from RAD tools which are now available in Unreal engine as builtin compression offerings (since 4.27+). You could see the effects of this in for example Ark Survival Evolved (250GB) -> Ark Survival Ascended (75GB, with richer models & textures), and associated improved load times.
- scq 3y ago
  zstd is over 4x faster than zlib, while having a better compression ratio. http://facebook.github.io/zstd/ http://facebook.github.io/zstd/