4 ms·

Context: Zstandard is a compression algorithm invented by Yann Collet, with the goal to exceed zlib's compression performance along every dimension (those being

Context: Zstandard is a compression algorithm invented by Yann Collet, with the goal to exceed zlib's compression performance along every dimension (those being compression ratio, compression speed, and decompression speed). Although that bar was cleared long ago, we've continued to work to improve Zstd, and with this release, we've made Zstd just that little bit more hardened/better/faster/stronger.

edflsafoiewq 7y ago
How does it compare in binary size?
- deleted 7y ago
  [deleted]
- felixhandte 7y ago
  On my system, the gzip binary is 100KB and my zstd binary is 1.0MB. There are a number of reasons for this: - By default, the Zstd binary has a lot of additional functionality that gzip doesn't. For example: Zstd includes benchmarking capabilities, the ability to train compression dictionaries, support for legacy (pre-1.0) formats, etc. You can strip all that stuff out with compile flags, and the binary gets a lot smaller. - Zstandard tries to cover a much wider range of compression speeds than zlib does. To do this, the Zstd compressor actually has eight or so different LZ match-finding implementations under the hood (which get inlined in various ways, resulting in >100 actual versions in the binary), and 5+ different entropy encoding implementations. For really size-conscious use-cases (like decompressing Zstd content in a mobile app), you can get a minimal Zstd decompressor library down to ~30KB.
meruru 7y ago
Great work.
jacques_chester 7y ago
As a sorta-kinda testimonial, Concourse just released a version[0] that switches its volume streaming compression from gzip to zstd. It consumed so much less CPU that they were also able to introduce parallel streaming. On a test workload which slings around many large volumes these changes made it approximately 20x faster[1]. [0] https://github.com/concourse/concourse/releases/tag/v5.4.0 https://github.com/concourse/concourse/releases/tag/v5.4.0 [1] https://github.com/concourse/concourse/issues/3992#issuecomment-504036411 https://github.com/concourse/concourse/issues/3992#issuecomm...
zepearl 7y ago
Thanks a lot - zstd is reaaaally fast for the compression ratios that it achieves. I personally use it all the time in Clickhouse tables (the "Yandex Clickhouse" database). I admit that I'm still using "xz" when I focus on hardcore max compression (max compression within an "acceptable" timeframe) when doing specific tests with their focus on max final compression.
TAForObvReasons 7y ago
Is RFC 8478 still compatible with the newest library releases?
- felixhandte 7y ago
  Yes. The 1.0.0 release froze the wire format (almost, see below). All Zstandard releases after that are all interoperable. It’s funny though that you should ask that question now. Just yesterday I did submit an erratum on the RFC [0], changing a tiny detail of the spec. While it is literally, a breaking change, (1) we have not yet changed Zstd to produce outputs taking advantage of the change, and (2) it is not actually a concern, because all existing Zstd decoders ignore the existing spec in this regard. [0] https://www.rfc-editor.org/errata/eid5786 https://www.rfc-editor.org/errata/eid5786

edflsafoiewq 7y ago

How does it compare in binary size?

deleted 7y ago
[deleted]
felixhandte 7y ago
On my system, the gzip binary is 100KB and my zstd binary is 1.0MB. There are a number of reasons for this: - By default, the Zstd binary has a lot of additional functionality that gzip doesn't. For example: Zstd includes benchmarking capabilities, the ability to train compression dictionaries, support for legacy (pre-1.0) formats, etc. You can strip all that stuff out with compile flags, and the binary gets a lot smaller. - Zstandard tries to cover a much wider range of compression speeds than zlib does. To do this, the Zstd compressor actually has eight or so different LZ match-finding implementations under the hood (which get inlined in various ways, resulting in >100 actual versions in the binary), and 5+ different entropy encoding implementations. For really size-conscious use-cases (like decompressing Zstd content in a mobile app), you can get a minimal Zstd decompressor library down to ~30KB.

meruru 7y ago

Great work.

jacques_chester 7y ago

As a sorta-kinda testimonial, Concourse just released a version[0] that switches its volume streaming compression from gzip to zstd. It consumed so much less CPU that they were also able to introduce parallel streaming. On a test workload which slings around many large volumes these changes made it approximately 20x faster[1]. [0] https://github.com/concourse/concourse/releases/tag/v5.4.0 https://github.com/concourse/concourse/releases/tag/v5.4.0 [1] https://github.com/concourse/concourse/issues/3992#issuecomment-504036411 https://github.com/concourse/concourse/issues/3992#issuecomm...

zepearl 7y ago

Thanks a lot - zstd is reaaaally fast for the compression ratios that it achieves. I personally use it all the time in Clickhouse tables (the "Yandex Clickhouse" database). I admit that I'm still using "xz" when I focus on hardcore max compression (max compression within an "acceptable" timeframe) when doing specific tests with their focus on max final compression.

TAForObvReasons 7y ago

Is RFC 8478 still compatible with the newest library releases?

felixhandte 7y ago
Yes. The 1.0.0 release froze the wire format (almost, see below). All Zstandard releases after that are all interoperable. It’s funny though that you should ask that question now. Just yesterday I did submit an erratum on the RFC [0], changing a tiny detail of the spec. While it is literally, a breaking change, (1) we have not yet changed Zstd to produce outputs taking advantage of the change, and (2) it is not actually a concern, because all existing Zstd decoders ignore the existing spec in this regard. [0] https://www.rfc-editor.org/errata/eid5786 https://www.rfc-editor.org/errata/eid5786