3 ms·

I’ve used lz4 several times for compressing blobs and storing them for serving real-time traffic. In my opinion LZ4 and Snappy hits fine balance of CPU usage an

I’ve used lz4 several times for compressing blobs and storing them for serving real-time traffic. In my opinion LZ4 and Snappy hits fine balance of CPU usage and network usage when you are sensitive to p99 latencies. ZSTD seems to pay more attention to compression size. I hope to do a real life test instead of sythetic benchmarks. Has someone already used it in prod?

strainer 8y ago
I'm curious how LZ4_HC version (high compression) compares to ZSTD since LZ4 benched about twice as quick for compression and decompression, albeit at the lower ratio. Also reportedly, that decompression speed is not slowed by higher compression setting.
ncmncm 8y ago
Lz4 (by the same author, BTW) wins, hands down, on decode time, but doesn't compress as much. That doesn't always matter. Sometimes decode rate is overwhelmingly most important. When lz4 can decode faster than you can feed it, the benefit decreases, unless you can arrange to do something else while waiting. One nice quality of zstd is that you can adaptively adjust the compression ratio according to load -- i.e., input rate, or available CPU.
matsur 8y ago
One production write up, compressing Kafka messages at Cloudflare: https://blog.cloudflare.com/squeezing-the-firehose/ https://blog.cloudflare.com/squeezing-the-firehose/ “For our data zstd was giving amazing results even on the lowest compression level. Compression ratio was better than even gzip at maximum compression level, while throughput was a lot higher.”
felixhandte 8y ago
Yeah, if you’ve got a need for serious speed, especially in decompression, LZ4 can’t be beat. However, Zstd recently added “negative” compression levels (i.e., faster than level 1). Compression is more or less comparable to LZ4 (try level -4 or -5, via `—fast 4` or `—fast 5`). Decompression speed will be somewhere between normal Zstd and LZ4. This lets you use a single format and library in an extremely wide range of situations, where previously you might have used a combination of LZMA and Zlib and LZ4.

strainer 8y ago

I'm curious how LZ4_HC version (high compression) compares to ZSTD since LZ4 benched about twice as quick for compression and decompression, albeit at the lower ratio. Also reportedly, that decompression speed is not slowed by higher compression setting.

ncmncm 8y ago

Lz4 (by the same author, BTW) wins, hands down, on decode time, but doesn't compress as much. That doesn't always matter. Sometimes decode rate is overwhelmingly most important. When lz4 can decode faster than you can feed it, the benefit decreases, unless you can arrange to do something else while waiting. One nice quality of zstd is that you can adaptively adjust the compression ratio according to load -- i.e., input rate, or available CPU.

matsur 8y ago

One production write up, compressing Kafka messages at Cloudflare: https://blog.cloudflare.com/squeezing-the-firehose/ https://blog.cloudflare.com/squeezing-the-firehose/ “For our data zstd was giving amazing results even on the lowest compression level. Compression ratio was better than even gzip at maximum compression level, while throughput was a lot higher.”

felixhandte 8y ago

Yeah, if you’ve got a need for serious speed, especially in decompression, LZ4 can’t be beat. However, Zstd recently added “negative” compression levels (i.e., faster than level 1). Compression is more or less comparable to LZ4 (try level -4 or -5, via `—fast 4` or `—fast 5`). Decompression speed will be somewhere between normal Zstd and LZ4. This lets you use a single format and library in an extremely wide range of situations, where previously you might have used a combination of LZMA and Zlib and LZ4.