3 ms·

The only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate

The only downside is that zstd has a (relatively) large binary size. I evaluated using it for one of my projects, but it’s a massive library. The zstd-sys crate is 714kb, and zstd has no native rust port presumably because of its size. I ended up using lz4 instead for my project. Lz4 only added 4kb to my compiled wasm size for an encoder and decoder. And it’s native rust (which makes wasm compilation effortless). Lz4 is a no-brainer to include. The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios.

loeg 4y ago
> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios. lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 There's also LZ4_HC.
- duskwuff 4y ago
  LZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.
sn0wf1re 4y ago
I feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.
- maximilianburke 4y ago
  Exactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.
coder543 4y ago
There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd https://github.com/klauspost/compress/tree/master/zstd It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.
Beltalowda 4y ago
According to a comment in the Chromium issue, it can be configured/stripped down quite a bit: > While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers. While still larger than 4K, it does seem possible to make much smaller builds.

loeg 4y ago

> The downside of lz4 is that it can’t be configured to run at higher & slower compression ratios. lz4 has some level of configurability? https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 https://github.com/lz4/lz4/blob/v1.9.4/lib/lz4frame.h#L194 There's also LZ4_HC.

duskwuff 4y ago
LZ4 (which includes the HC encoder) doesn't have an entropy encoder stage, so it's fairly limited in what compression levels it can achieve.

sn0wf1re 4y ago

I feel that is a valid argument for server-side resources not supporting it, but browsers are already hundreds of MB (if not multiple GB) in size.

maximilianburke 4y ago
Exactly, and that couple hundred KB in the browser could pay for itself in terms of bandwidth saved in almost no time over gzip/br.

coder543 4y ago

There is a reasonably feature complete implementation of Zstd for Go: https://github.com/klauspost/compress/tree/master/zstd https://github.com/klauspost/compress/tree/master/zstd It may not offer the same API 1:1, but it has no interoperability issues that I've encountered. I think no one has bothered to implement it in Rust because most use cases don't mind the added bloat you're talking about, not because implementing it in Rust would be insurmountable. Plus, other comments I've seen suggest that you can actually tune the size of the C zstd library, although I'm not sure if the Rust bindings expose that.

Beltalowda 4y ago

According to a comment in the Chromium issue, it can be configured/stripped down quite a bit: > While it's true that a kitchen-sink version of libzstd does weigh in at 1.5MB, the build is easily configurable and much of that can be shed. If Chromium is just going to ship the decoder, a full build of the decoder is 212kB (162kB stripped, 71kB gzipped). A size-optimized version of that is 99kB (62kB stripped, 26kB gzipped). As of recently, we had someone from the busybox project working to strip that down even more, so we very well may be able to improve further on those numbers. While still larger than 4K, it does seem possible to make much smaller builds.