4 ms·

I love everything about zstd. It compresses well, it compresses and decompresses fast. I wish browsers supported it for content encoding because it's a good de

I love everything about zstd. It compresses well, it compresses and decompresses fast. I wish browsers supported it for content encoding because it's a good deal faster and smaller than gzip, nevermind brotli.

josephg 4y ago
The only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate is 714kb, and zstd has no native rust port presumably because of its size. I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include. The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.
- loeg 4y ago
  > The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios. lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 There's also LZ4_HC.
  - duskwuff 4y ago
    LZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.
- sn0wf1re 4y ago
  I feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.
  - maximilianburke 4y ago
    Exactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.
- coder543 4y ago
  There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd https://github.com/klauspost/compress/tree/master/zstd It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.
- Beltalowda 4y ago
  According to a comment in the Chromium issue, it can be configured/stripped down quite a bit: > While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers. While still larger than 4K, it does seem possible to make much smaller builds.
markdog12 4y ago
Chrome issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1246971 https://bugs.chromium.org/p/chromium/issues/detail?id=124697...
coder543 4y ago
The "nevermind brotli" seems overly dismissive. Browsers should add support for zstd, and I like zstd a lot, but I have found more cases than I would have expected where brotli compresses files better than zstd, even while using a ridiculously high compression level setting on zstd (for example, 19). I can't remember encountering anything that compresses better with gzip than with brotli, so brotli is usually a better choice than gzip to reduce bandwidth, and brotli has been better than zstd on more than a few occasions for me as noted previously. Brotli is very worthwhile, at least for static assets, and choosing to offer only gzip compressed assets to browsers is almost always a waste of bandwidth. If faster to compress is all you care about, then sure, brotli might not be the best choice.
- maximilianburke 4y ago
  My views on brotli are just from my experience with it. I'm looking at my static assets now; our largest JS bundle is 6.3MB uncompressed. With brotli at level 9 it takes 400ms to compress down to 1.34mb, and with zstd at level 13 it squeezes down to 1.31mb in 300ms. If I use zstd at level 19, it takes 1.1s to compress, but size is even further reduced to 1.21mb. Decompression time is identical for both (33ms). So, for our static data the edge goes to zstd. For dynamic data there's no contest. Most of our responses are binary and brotli is not just slow, it's so slow we have to disable it otherwise people assume our services have hung. By far most of the data that leaves our servers is dynamic, so having zstd support in browsers would be a huge win for us because we would spend less time compressing (compared to gzip) and have smaller responses. I was looking at one particularly pathological case last month where we had a 2.7gb response that was quite sparse. We were spending 30s of server time as part of the request in gzip, the end result being 52mb. With zstd at level 12 it was 5s to compress it down to 41mb. I don't even know how long it would have taken with brotli as I stopped timing after 10 minutes. I mean, if brotli wasn't working for people it wouldn't be supported by browsers, but I really really wish zstd was an option.
danudey 4y ago
The OpenZFS project added support for ZSTD a while ago for transparent compression, and it's fantastic. I was seeing substantial compression rates given the type of data we were storing for basically free.

josephg 4y ago

The only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate is 714kb, and zstd has no native rust port presumably because of its size. I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include. The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

loeg 4y ago
> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios. lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 There's also LZ4_HC.
- duskwuff 4y ago
  LZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.
sn0wf1re 4y ago
I feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.
- maximilianburke 4y ago
  Exactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.
coder543 4y ago
There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd https://github.com/klauspost/compress/tree/master/zstd It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.
Beltalowda 4y ago
According to a comment in the Chromium issue, it can be configured/stripped down quite a bit: > While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers. While still larger than 4K, it does seem possible to make much smaller builds.

markdog12 4y ago

Chrome issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1246971 https://bugs.chromium.org/p/chromium/issues/detail?id=124697...

coder543 4y ago

The "nevermind brotli" seems overly dismissive. Browsers should add support for zstd, and I like zstd a lot, but I have found more cases than I would have expected where brotli compresses files better than zstd, even while using a ridiculously high compression level setting on zstd (for example, 19). I can't remember encountering anything that compresses better with gzip than with brotli, so brotli is usually a better choice than gzip to reduce bandwidth, and brotli has been better than zstd on more than a few occasions for me as noted previously. Brotli is very worthwhile, at least for static assets, and choosing to offer only gzip compressed assets to browsers is almost always a waste of bandwidth. If faster to compress is all you care about, then sure, brotli might not be the best choice.

maximilianburke 4y ago
My views on brotli are just from my experience with it. I'm looking at my static assets now; our largest JS bundle is 6.3MB uncompressed. With brotli at level 9 it takes 400ms to compress down to 1.34mb, and with zstd at level 13 it squeezes down to 1.31mb in 300ms. If I use zstd at level 19, it takes 1.1s to compress, but size is even further reduced to 1.21mb. Decompression time is identical for both (33ms). So, for our static data the edge goes to zstd. For dynamic data there's no contest. Most of our responses are binary and brotli is not just slow, it's so slow we have to disable it otherwise people assume our services have hung. By far most of the data that leaves our servers is dynamic, so having zstd support in browsers would be a huge win for us because we would spend less time compressing (compared to gzip) and have smaller responses. I was looking at one particularly pathological case last month where we had a 2.7gb response that was quite sparse. We were spending 30s of server time as part of the request in gzip, the end result being 52mb. With zstd at level 12 it was 5s to compress it down to 41mb. I don't even know how long it would have taken with brotli as I stopped timing after 10 minutes. I mean, if brotli wasn't working for people it wouldn't be supported by browsers, but I really really wish zstd was an option.

danudey 4y ago

The OpenZFS project added support for ZSTD a while ago for transparent compression, and it's fantastic. I was seeing substantial compression rates given the type of data we were storing for basically free.