9 ms·

I am surprised all major browsers hadn't implemented this years ago. Zstd is way better than gzip and brotli.

baybal2 3y ago
[dead]
Y_Y 3y ago
Hardly surprising when you consider gzip's DEFLATE was around in 1990 and developed by one guy as shareware and zstd was produced by a megacorp in 2015.
- lifthrasiir 3y ago
  Zstandard was also mainly designed and produced by a single person, Yann Collet.
  - dralley 3y ago
    Yann also developed LZ4
  - KingMob 3y ago
    Yes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors https://github.com/facebook/zstd/graphs/contributors, 300+ other contributors have made 4500+ commits to the zstd repo. It's not quite as small-scale as 90's-style shareware was.
    - kzrdude 3y ago
      zstd has improved a lot since it was brought in under the facebook roof, but those are incremental changes, it already existed before then. Here's the good old blog about that http://fastcompression.blogspot.com/ http://fastcompression.blogspot.com/
    - zerd 3y ago
      I recommend the CoRecursive episode about how LZ4 and zstd came to be. It started with making games for a HP calculator. https://corecursive.com/data-compression-yann-collet/ https://corecursive.com/data-compression-yann-collet/
- veltas 3y ago
  Yes it's worth taking a minute to appreciate how well DEFLATE and gzip has stood up over the years. It's a brilliant format and tool. I was definitely a little too ready to believe the paper that claimed gzip's ability to classify text was better than deep neutral nets, alas it has some limits after all!
silverquiet 3y ago
How much better is Zstd? Compression seems like one of those things where diminishing returns kick in very quickly. Whenever I need compression, it's always just been basically, "throw gzip at it because it does a lot without much of a performance hit". Basically it does the job.
- lifthrasiir 3y ago
  If you are not too much constrained in I/O rate, Zstandard has no match; it can easily clock more than 1 GB/s with a good compression ratio and also can automatically adapt to the changing I/O rate. Web browsers typically work with much less bandwidth available, though, so both Brotli and Zstandard are virtually identical in the clients.
  - jorvi 3y ago
    It’s not so much about IO rate but access times. If compression and decompression speed are critical, lz4. Zstd for pretty much everything else. There are edge cases where compression time doesn’t matter but decompression time does. This used to be the case for game torrents back in the old days, and UHARC was used for that to great effect. Not sure what the current king is for that purpose.
    - lifthrasiir 3y ago
      Brotli has a higher potential for the compression ratio due to its internal structure, so that edge case would be better served by Brotli than by Zstandard.
    - repiret 3y ago
      One of the systems I maintain at work uses enormous, highly compressible, text files, which need to be compressed once and decompressed many times. Decompression speed isn't critical, it just needs to keep up with processing the decompressed data to avoid being the bottleneck. We optimize primarily for compression ratio. For that system, we haven't found something that beats `xz -9`.
    - 3y ago
silisili 3y ago
Highly dependent on several factors. I was comparing about ten compression algos for compressing json data. Mainly needing something reasonably fast and good compressing. zstd did well, but brotli absolutely crushed it at every metric. Of course, it's a data point of one, but it exists.
everybackdoor 3y ago
[dead]

[dead]

Y_Y 3y ago

Hardly surprising when you consider gzip's DEFLATE was around in 1990 and developed by one guy as shareware and zstd was produced by a megacorp in 2015.

lifthrasiir 3y ago
Zstandard was also mainly designed and produced by a single person, Yann Collet.
- dralley 3y ago
  Yann also developed LZ4
- KingMob 3y ago
  Yes, but they also work for megacorp Facebook, and according to https://github.com/facebook/zstd/graphs/contributors https://github.com/facebook/zstd/graphs/contributors, 300+ other contributors have made 4500+ commits to the zstd repo. It's not quite as small-scale as 90's-style shareware was.
  - kzrdude 3y ago
    zstd has improved a lot since it was brought in under the facebook roof, but those are incremental changes, it already existed before then. Here's the good old blog about that http://fastcompression.blogspot.com/ http://fastcompression.blogspot.com/
  - zerd 3y ago
    I recommend the CoRecursive episode about how LZ4 and zstd came to be. It started with making games for a HP calculator. https://corecursive.com/data-compression-yann-collet/ https://corecursive.com/data-compression-yann-collet/
veltas 3y ago
Yes it's worth taking a minute to appreciate how well DEFLATE and gzip has stood up over the years. It's a brilliant format and tool. I was definitely a little too ready to believe the paper that claimed gzip's ability to classify text was better than deep neutral nets, alas it has some limits after all!

silverquiet 3y ago

How much better is Zstd? Compression seems like one of those things where diminishing returns kick in very quickly. Whenever I need compression, it's always just been basically, "throw gzip at it because it does a lot without much of a performance hit". Basically it does the job.

lifthrasiir 3y ago
If you are not too much constrained in I/O rate, Zstandard has no match; it can easily clock more than 1 GB/s with a good compression ratio and also can automatically adapt to the changing I/O rate. Web browsers typically work with much less bandwidth available, though, so both Brotli and Zstandard are virtually identical in the clients.
- jorvi 3y ago
  It’s not so much about IO rate but access times. If compression and decompression speed are critical, lz4. Zstd for pretty much everything else. There are edge cases where compression time doesn’t matter but decompression time does. This used to be the case for game torrents back in the old days, and UHARC was used for that to great effect. Not sure what the current king is for that purpose.
  - lifthrasiir 3y ago
    Brotli has a higher potential for the compression ratio due to its internal structure, so that edge case would be better served by Brotli than by Zstandard.
  - repiret 3y ago
    One of the systems I maintain at work uses enormous, highly compressible, text files, which need to be compressed once and decompressed many times. Decompression speed isn't critical, it just needs to keep up with processing the decompressed data to avoid being the bottleneck. We optimize primarily for compression ratio. For that system, we haven't found something that beats `xz -9`.
  - 3y ago

silisili 3y ago

Highly dependent on several factors. I was comparing about ten compression algos for compressing json data. Mainly needing something reasonably fast and good compressing. zstd did well, but brotli absolutely crushed it at every metric. Of course, it's a data point of one, but it exists.

everybackdoor 3y ago

[dead]