4 ms·

So, I wanted to check so I downloaded linux source tree and ran gzip and zstd. Uncompressed .tar: 1064376320 .xz (downloaded file): 117637692 (88.9%

So, I wanted to check so I downloaded linux source tree and ran gzip and zstd. Uncompressed .tar: 1064376320 .xz (downloaded file): 117637692 (88.9%) .gz: 189299444 (82.2%) .gz (level 9): 186533230 (82.5%) .zst: 178012638 (83.3%) .zst --ultra -22: 119676867 (88.8%) So, even at maximum level, zstd is only about 6% better, or about 37% if you consider .gz as baseline, while being about 10x slower than gzip level 9. There are places where having a smaller compressed file is worth the cost of 10x execution time (and much higher memory usage), but serving web pages is almost certainly not one of them. In other words, the reason gzip is still being widely used is simply because it's good enough for its use cases. It's less like the original T1, and more like complaining that people are still driving Toyota Camry 2011 when Toyota Camry 2021 is a so much better car.

dmw_ng 6y ago
For the pile of XML in front of me just now (originally arrived in a ZIP container aka deflate, part of a much larger set), it's 1.4gb decompressed, 44M gzipped/deflated (default settings), 31mb zstd (default), 5.379s to decompress with gzip, 0.560s to decompress with zstd. Just under an order of magnitude difference in the most frequent operation (decompression). Admittedly compressors generally love XML, this is just one example -- 28% less time on download and 89% less wasted on file open. Multiply by a few tens to a few thousand occurrences per week for 7.6 billion people, and I really struggle to call that a Camry.
- rualca 6y ago
  You're admittedly talking about a difference in compression rate of 13MB in a 1400MB package, and a 5s difference in writing a 1400MB file to disk. Come on, let's be honest here: that's nitpicking. Unless you are running batch processes that store TB of compressed data, no one would even bother to switch app for thosd residual gains. Let's put it this way: would you get any VC fund if your sales pitch was "I can improve compression in a 1400MB package by 13MB and shorten file write times by 5 seconds"? Odds are, you'd be asked why not just gzip it and get it over with.
  - dmw_ng 6y ago
    5.3s at 1.4gb on a laptop is 53ms at 14mb, is >70ms on mobile before adding any increased download time. 14mb is one retina image, even Reddit already weighs 8mb. It's silly to pretend this stuff doesn't matter, you can't even get search ranking without addressing it, for reasons that don't require concocted scenarios to explain.
    - NohatCoder 6y ago
      Your difference might be mainly single threaded vs multi threaded decompression. This doesn't typically transform into a real-world speedup when browsing the web, as one will typically run multiple decompression streams in parallel anyway. Zstd is definitely an improvement on Gzip, but not by that much.
    - rualca 6y ago
      > 5.3s at 1.4gb on a laptop is 53ms at 14mb, is 70ms on mobile before adding any increased download time. Factor in how much time it takes you to unpack those 1.4GB of data in a smartphone. Those 5s amount to which percentage of the total execution time?
  - OskarS 6y ago
    > You're admittedly talking about a difference in compression rate of 13MB in a 1400MB package, and a 5s difference in writing a 1400MB file to disk. > Come on, let's be honest here: that's nitpicking. I agree on the compression size (13 megabytes is really not very much difference) but the decompression speed improvement really is remarkable. It's an order of magnitude of difference! Amortize that over every time you make a webrequest that has to decompress data, it makes a huge difference. I'm mostly in the "gzip is good enough" camp, but a speed improvement of 10x is not nitpicking.
  - afavour 6y ago
    > Let's put it this way: would you get any VC fund if your sales pitch was "I can improve compression in a 1400MB package by 13MB and shorten file write times by 5 seconds"? Odds are, you'd be asked why not just gzip it and get it over with. I agree but have a different takeaway: it’s why VCs aren’t the be all and end all. Such an improvement is worth the time invested to create it. It won’t change the world but spanned across the entire globe it’s a very notable step forward. That a VC can’t get hockey stick growth and a lucrative return out of it doesn’t invalidate the idea.
- NohatCoder 6y ago
  A quick note on compressing XML. XML files typically have a high compression ratio because they consist of mostly redundancy, but compressing redundancy isn't free, you still end up with some overhead in the compressed file due to the redundancy. So if you pack the same data in a denser format, and then compress that, you typically end up saving a good chunk of space relative to the compressed XML file.
telendram 6y ago
This is a deliberately misleading statement. When compressor A reduces file size by 90%, and compressor B reduces same file by 99%, it's not "9% better". It's 10x times better. Because it can fit 10x more data in the same storage budget. The rest of the paragraph get exactly the same kind of bias. Take the absolute slowest mode, which is even guarded by special options, and make a general statement based solely on the speed of the slowest advanced setting, implying everything is slow. This is another willful misrepresentation, as zstd is built primarily for speed, and at its default level, compresses not only more than gzip level 9, but also at more than 10x the speed. Not even mentioning decompression speed which is much faster, nor the integrated multithreading mode which can make the difference even more startling since it is not available in default gzip. In short, this is a troll statement. You have picked a side, and contort representation to correspond to your pre-conceived agenda. At the age of disinformation, when bots and troll farms are deliberately pushing confusion onto unsuspecting readers, we have to call these acts as they are.
- yarcob 6y ago
  > at its default level, compresses not only more than gzip level 9, but also at more than 10x the speed I could also call that a wilful misrepresentation, since gzip level 9 is very slow compared to the default setting and IMO isn't worth the extra execution time. Also, nobody mentions that gzip is already installed everywhere. zstd is fairly new, and it's not installed by default on any of the machines I use. I mean, I'm sure there are some exotic screw drivers that work better than Torx. But I can get Torx screws and drivers in every hardware store and everyone already has compatible bits so why not use whatever works?
- usr1106 6y ago
  Never trust any statistics you have not manipulated yourself... You could have demonstrated the math without your attacks.
- nathell 6y ago
  > This is a deliberately misleading statement > the same kind of bias > another willful misrepresentation These phrases, as well as the whole last paragraph, contribute no meritorical value to the comment, but are aggresively vocal about assuming bad faith and as such violate HN guidelines. Please don't do that.
luckystarr 6y ago
The tradeoff you show between compression ratio and compression speed is a misleading one. The slow compression speed only takes place once (when compressing), while the saved bandwidth due to a better compression ratio is saved continuously.
- OskarS 6y ago
  Not if you're serving things dynamically from a web server, then it happens on the fly, every request.
  - luckystarr 6y ago
    True. But there you can go into optimizations like parallelization of the compression and building an optimized dictionary upfront once to make the process faster.
  - pmarreck 6y ago
    Cache by the ETag and/or If-Modified-Since headers, then

dmw_ng 6y ago

For the pile of XML in front of me just now (originally arrived in a ZIP container aka deflate, part of a much larger set), it's 1.4gb decompressed, 44M gzipped/deflated (default settings), 31mb zstd (default), 5.379s to decompress with gzip, 0.560s to decompress with zstd. Just under an order of magnitude difference in the most frequent operation (decompression). Admittedly compressors generally love XML, this is just one example -- 28% less time on download and 89% less wasted on file open. Multiply by a few tens to a few thousand occurrences per week for 7.6 billion people, and I really struggle to call that a Camry.

rualca 6y ago
You're admittedly talking about a difference in compression rate of 13MB in a 1400MB package, and a 5s difference in writing a 1400MB file to disk. Come on, let's be honest here: that's nitpicking. Unless you are running batch processes that store TB of compressed data, no one would even bother to switch app for thosd residual gains. Let's put it this way: would you get any VC fund if your sales pitch was "I can improve compression in a 1400MB package by 13MB and shorten file write times by 5 seconds"? Odds are, you'd be asked why not just gzip it and get it over with.
- dmw_ng 6y ago
  5.3s at 1.4gb on a laptop is 53ms at 14mb, is >70ms on mobile before adding any increased download time. 14mb is one retina image, even Reddit already weighs 8mb. It's silly to pretend this stuff doesn't matter, you can't even get search ranking without addressing it, for reasons that don't require concocted scenarios to explain.
  - NohatCoder 6y ago
    Your difference might be mainly single threaded vs multi threaded decompression. This doesn't typically transform into a real-world speedup when browsing the web, as one will typically run multiple decompression streams in parallel anyway. Zstd is definitely an improvement on Gzip, but not by that much.
  - rualca 6y ago
    > 5.3s at 1.4gb on a laptop is 53ms at 14mb, is 70ms on mobile before adding any increased download time. Factor in how much time it takes you to unpack those 1.4GB of data in a smartphone. Those 5s amount to which percentage of the total execution time?
- OskarS 6y ago
  > You're admittedly talking about a difference in compression rate of 13MB in a 1400MB package, and a 5s difference in writing a 1400MB file to disk. > Come on, let's be honest here: that's nitpicking. I agree on the compression size (13 megabytes is really not very much difference) but the decompression speed improvement really is remarkable. It's an order of magnitude of difference! Amortize that over every time you make a webrequest that has to decompress data, it makes a huge difference. I'm mostly in the "gzip is good enough" camp, but a speed improvement of 10x is not nitpicking.
- afavour 6y ago
  > Let's put it this way: would you get any VC fund if your sales pitch was "I can improve compression in a 1400MB package by 13MB and shorten file write times by 5 seconds"? Odds are, you'd be asked why not just gzip it and get it over with. I agree but have a different takeaway: it’s why VCs aren’t the be all and end all. Such an improvement is worth the time invested to create it. It won’t change the world but spanned across the entire globe it’s a very notable step forward. That a VC can’t get hockey stick growth and a lucrative return out of it doesn’t invalidate the idea.
NohatCoder 6y ago
A quick note on compressing XML. XML files typically have a high compression ratio because they consist of mostly redundancy, but compressing redundancy isn't free, you still end up with some overhead in the compressed file due to the redundancy. So if you pack the same data in a denser format, and then compress that, you typically end up saving a good chunk of space relative to the compressed XML file.

telendram 6y ago

This is a deliberately misleading statement. When compressor A reduces file size by 90%, and compressor B reduces same file by 99%, it's not "9% better". It's 10x times better. Because it can fit 10x more data in the same storage budget. The rest of the paragraph get exactly the same kind of bias. Take the absolute slowest mode, which is even guarded by special options, and make a general statement based solely on the speed of the slowest advanced setting, implying everything is slow. This is another willful misrepresentation, as zstd is built primarily for speed, and at its default level, compresses not only more than gzip level 9, but also at more than 10x the speed. Not even mentioning decompression speed which is much faster, nor the integrated multithreading mode which can make the difference even more startling since it is not available in default gzip. In short, this is a troll statement. You have picked a side, and contort representation to correspond to your pre-conceived agenda. At the age of disinformation, when bots and troll farms are deliberately pushing confusion onto unsuspecting readers, we have to call these acts as they are.

yarcob 6y ago
> at its default level, compresses not only more than gzip level 9, but also at more than 10x the speed I could also call that a wilful misrepresentation, since gzip level 9 is very slow compared to the default setting and IMO isn't worth the extra execution time. Also, nobody mentions that gzip is already installed everywhere. zstd is fairly new, and it's not installed by default on any of the machines I use. I mean, I'm sure there are some exotic screw drivers that work better than Torx. But I can get Torx screws and drivers in every hardware store and everyone already has compatible bits so why not use whatever works?
usr1106 6y ago
Never trust any statistics you have not manipulated yourself... You could have demonstrated the math without your attacks.
nathell 6y ago
> This is a deliberately misleading statement > the same kind of bias > another willful misrepresentation These phrases, as well as the whole last paragraph, contribute no meritorical value to the comment, but are aggresively vocal about assuming bad faith and as such violate HN guidelines. Please don't do that.

luckystarr 6y ago

The tradeoff you show between compression ratio and compression speed is a misleading one. The slow compression speed only takes place once (when compressing), while the saved bandwidth due to a better compression ratio is saved continuously.

OskarS 6y ago
Not if you're serving things dynamically from a web server, then it happens on the fly, every request.
- luckystarr 6y ago
  True. But there you can go into optimizations like parallelization of the compression and building an optimized dictionary upfront once to make the process faster.
- pmarreck 6y ago
  Cache by the ETag and/or If-Modified-Since headers, then