11 ms·

Zstandard – Real-time data compression algorithm

286 points by josephscott 9y ago

jdhawk 9y ago
We've been using zstd for a while in production, switching over from rar/gzip/7z for archiving huge XML files. It's speed to compression ratio is really impressive. Hats off to the development team.
loeg 9y ago
We added zstd to largely supersede gzip use at work. It performs better than oldschool gzip in both compression and speed. That said, I'm not sure I'd call it a "real-time" compression algorithm. It's still an factor of 2x slower than lz4 compress and a factor of 4x slower decompress.
- michaelmior 9y ago
  It's not clear exactly what "real-time" is supposed to mean, but I don't think it makes sense to define it in terms of other algorithms. If someone developed an algorithm faster than lz4, would lz4 cease to be real-time?
  - joosters 9y ago
    I thought that 'real-time' is not about how fast a program is. Instead, it's about being able to guarantee that a program will always complete in under a fixed amount of time, under all circumstances. I can't see how that applies to Zstandard either.
    - openasocket 9y ago
      Hard real time certainly means that. I've seen "soft real-time" used to refer to programs where the standard deviation of the runtime given random input is low.
    - michaelmior 9y ago
      Yes, I that would be a correct definition although practically speaking the deadline that is guaranteed is important as well. Part of the problem is that the term "real-time" is overloaded.
    - hinkley 9y ago
      It’s a definition not used much anymore but this relates to on demand (streaming) versus ahead of time data processing. With only a small delay, can I compress and decompress at least one second of data per second, indefinitely. Think live TV versus film.
      - chowells 9y ago
        That's usually called an "online" algorithm, in my experience. It does seem to agree with the way it's used in this context.
        foota 9y ago
        Online isn't quite the right distinction here imo, since that's really more about possibility than speed. An algorithm could be online (doesn't require all the data at once to run) but still not be fast enough to stream.
    - hackcasual 9y ago
      Talking about compression you're generally talking about things that when you factor in network overhead have a neutral or better impact on end-to-end time.
- bgirard 9y ago
  It's really subjective based on your computing resources client CPU + network speed and your server CPU if you can't compress ahead of time. Here's a great page that captures this: https://quixdb.github.io/squash-benchmark/#transfer-plus-processing https://quixdb.github.io/squash-benchmark/#transfer-plus-pro... TL;DR: Your optimal compression algorithm will vary based on these parameters. There's no universal answer.
teej 9y ago
We switched many of our Redshift compression encodings to zstd. Works fantastically on text and JSON.
ComputerGuru 9y ago
zstd is incredible, but just in case the thought hasn't occurred to someone here that may benefit from it: if you're in control of the send and the receive, type-specific compression is hard to beat. For example, if you know you're dealing with text, you can use snappy, if you know you're dealing with images, webp, videos x264 (or x265 if you only care about decode speed and encoded size), etc and then fall back to zstd only when you don't have a specific compressor for the chosen file type.
- halayli 9y ago
  I'd pick zstd over snappy for text
- hackcasual 9y ago
  zstd is in snappy's domain, as a low overhead way of reducing bandwidth usage. You've got devs writing some service that talks in JSON or protobuf, just rub a little zstd on it, and bingo, your bandwidth is reduced.
- Smerity 9y ago
  Agreed that if you have the control (and potentially the time depending on the algorithm), type specific compression is the way to go. Having said that, zstd beats Snappy handily for text ^_^ On enwik8 (100MB of Wikipedia XML encoded articles, mostly just text), zstd gets you to ~36MB, Snappy gets you to ~58MB, while gzip will get you to 36MB. If you turn up the compression dials on zstd, you can get down to 27MB - though instead of 2 seconds to compress it takes 52 seconds on my laptop. Decompression takes ~0.3 seconds for the low or high compression rate.
  - wolf550e 9y ago
    ppmd will get you better compression on text (better than zstd and even better than lzma), but it is slow to compress and decompress. Use `7z a -m0=PPMd demo.7z demo.txt`
  - hokkos 9y ago
    Why not use a XML/XSD specific compression like EXI for that ? https://www.w3.org/TR/exi-primer/ https://www.w3.org/TR/exi-primer/
tzs 9y ago
> Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off This would have been very handy about 17 years ago when I was doing a virtual CD-ROM driver, and needed to store compressed images, and the client wanted a wide range of speed/compression options for the user--much wider than zlib offered. I ended up doing a hack that was either the most disgusting thing I have ever done, or was brilliant. I have not been able to decide which. My hack: I gave them a compression/speed slider that went from 0 to 100. If the slider was set to N, I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks. Repeat until the whole image is stored. The client loved it.
- bgirard 9y ago
  Performance wise it's far from ideal. In your N block you're going to spend a lot of time, per-byte, to compress while leaving trivially compressible data in the remaining blocks. For N=50, you'd get much better compression for the same amount of cycles applying a faster compression to all the bytes.
- deleted 9y ago
  [deleted]
- gopalv 9y ago
  > I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks Zlib is actually highly configurable - it has a huge number of tweaks beyond just that one flag of size. DEFAULT_STRATEGY is pretty much only one possible strategy. You can tweak the LZ77, RLE and Huffman separately to get different compression performance (the levels correspond to different LZ77 lookups and deflate_fast vs deflate_slow). In my work, I use different zlib strategies for different data (i.e already bit-packed integers vs raw strings), which works out pretty well. My current complaint is that the Zlib algorithm doesn't quite detect record boundaries when building a dictionary, which is all only relevant to the compression algorithm, because the output is eventually just a DEFLATE stream (Zopfli is an example of an encoder only Zlib library). Zstandard has a much wider set of options to use already built-in (fast, greedy, lazy, btlazy, btopt, btultra) etc.
cakoose 9y ago
In the "Compression Speed vs Ratio" graph, the Y-axis doesn't start at zero. If it were changed to start at zero, it would be easier to evaluate the magnitude of the compression ratio change at a glance. IMO, that's probably worth the extra whitespace in the graph.
natbobc 9y ago
Why is this site an SPA?!?!?
- natbobc 9y ago
  Genuine question with everything happening in Intel land these days I've turned off JS except for a select few websites. Seems like an interesting project though.
- akx 9y ago
  It's... not? It's a static page with some JavaScript-driven charts.
  - userbinator 9y ago
    It's a page with the majority of the content wrapped in an XMP element with style="display:none;". I'd consider that pretty hostile.
revelation 9y ago
From that benchmark list I'd prefer lz4..
- terrelln 9y ago
  lz4 and zstd are complementary, not competitive. lz4 targets applications where extremely fast decompression speed (and compression speed) are required, at the expense of compression ratio. However, many applications would prefer to spend a bit (or a lot) more CPU in exchange for better compression, while maintaining fast decompression speed.
terrelln 9y ago
Zstd recently added a long range mode that can find matches up to 2 GB in the past using a specialized algorithm [1]. It can be enabled on the command line with `zstd --long` for a 128 MB window, or `--long=windowLog` for a `2^windowLog` byte window. [1] https://github.com/facebook/zstd/releases/tag/v1.3.2 https://github.com/facebook/zstd/releases/tag/v1.3.2
- Willson50 9y ago
  A non-obvious drawback of this is you give up multi-threaded compression (maybe fixed in a later release). You also need to use the same memory size when decompressing.
  - terrelln 9y ago
    We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.
    - ghusbands 9y ago
      I understand this to mean that you believe the pages in the long range aren't accessed frequently. Otherwise, mmap doesn't reduce memory usage so much as just hide it. Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
      - terrelln 9y ago
        Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed. The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.
gateway7 9y ago
That patent clause though. Stick to the fork from before Facebook bought the man and the algorithm.
- terrelln 9y ago
  It was relicensed in v1.3.1. It is now dual licensed under BSD without the patents clause [1] and GPLv2 [2]. GPLv2 was added for inclusion in the Linux kernel. [1] https://github.com/facebook/zstd/blob/dev/LICENSE https://github.com/facebook/zstd/blob/dev/LICENSE [2] https://github.com/facebook/zstd/blob/dev/COPYING https://github.com/facebook/zstd/blob/dev/COPYING
cmurf 9y ago
Supported on Btrfs since kernel 4.14. Like other compression options, it's a mount time option. Edit: Also in squashfs. Here's the git pull request which includes some benchmarks. https://lkml.org/lkml/2017/9/11/369 https://lkml.org/lkml/2017/9/11/369
- terrelln 9y ago
  Upstream SquashFUSE also has zstd support. https://github.com/vasi/squashfuse https://github.com/vasi/squashfuse
2pointsomone 9y ago
That's the best Weissman score I have ever seen.
- 2pointsomone 9y ago
  Come on people, take a joke
  - hodl 9y ago
    I get ya!
desertrider12 9y ago
I wonder how the RAD Game Tools compressors would fit into that benchmark list. In their benchmarks Oodle Selkie has roughly the same ratio as Zlib but is 3-4x faster at decompression than Zstd (not the same benchmark though). http://www.radgametools.com/oodle.htm http://www.radgametools.com/oodle.htm
- blattimwind 9y ago
  Asset compression has a specific optimization function (Charles Bloom of RAD Game Tools has a ton about this on his blog, cbloomrants): maximize output bandwidth given this specific CPU and input bandwidth. Compression speed, for example, doesn't really matter. Sure, it should be something reasonable (e.g. not like PAQ), but if you can bump the above metric by 10 % while compression takes 50 % longer: probably worth it.
  - desertrider12 9y ago
    Aha, he actually addressed zstd in a post. tldr is zstd usually lower ratio and lower speed than mermaid but sometimes better ratio for text.
bajsejohannes 9y ago
Some good discussion in earlier postings too: https://hn.algolia.com/?query=zstandard&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=zstandard&sort=byPopularity&pr...
reacharavindh 9y ago
Anyone using zstd along with an incremental rsync approach? Like gzip --rsyncable ? Say I have 1000 files. I want to compress them and let the cron rsync do it's thing. Next day, if only one file had changed, rsync should pickup only the differential instead of the whole archive. Or is there a better way of doing it?
cjhanks 9y ago
I use the C bindings for ZStandard, it performs very nicely as an algorithm. I wish the documentation was a little easier to use. I am still not sure if the provided examples really properly handle the degenerate cases of /dev/urandom and /dev/zero when using `ZSTD_DStream{In,Out}Size` and friends. In any case - thanks for releasing this, it's been very helpful to me.
jakozaur 9y ago
In a lot of big data workflows, zstd is no brainer replacement of gzip. Always faster decompression, plus you can save tons of storage and data transfer.
lars_francke 9y ago
Last time this was discussed Zstandard didn't have a splittable mode and by the looks of it[1] they still don't. That doesn't make it a bad algorithm it just means that it's not a good choice yet for Hadoop et al. As far as I know no container format has implemented Zstandard yet. Does anyone know any better? It seems like we could use a better alternative to Snappy. [1] https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395
- gopalv 9y ago
  > As far as I know no container format has implemented Zstandard yet. ORC has a reserved future enum for ZSTD [1] (my guess is that FB already uses it). The original headache was the patent licensing scheme[2] for the actual codebase, but now that hadoop has a ZStandard built into libhadoop.so, this is much more easy to ship to Apache. My branch is so far behind now that I probably would implement it faster if I started over & used the libhadoop.so impl as the basis - https://github.com/t3rmin4t0r/orc/tree/zstd https://github.com/t3rmin4t0r/orc/tree/zstd [1] - https://issues.apache.org/jira/browse/ORC-46 https://issues.apache.org/jira/browse/ORC-46 [2] - https://github.com/facebook/zstd/issues/775 https://github.com/facebook/zstd/issues/775
beagle3 9y ago
I used to object to using zstd based on Facebook's onerous license and patent policy; but now that zstd is plain BSD+GPLv2, I endorse it.
waterhouse 9y ago
A curious fact I encountered: First, the zstdcat executable that ships with zstd (actually a symlink to zstd itself, roughly equivalent to "zstd -cqdf") can decompress .gz files. Second, zstd is faster than gzip at decompressing .gz files, taking about 2/3 as long. I'm not kidding. I couldn't believe it myself, but subsequent testing stubbornly bore it out—on one file that was 15 MB after compression, and on a mix of smaller files. I tried compiling gzip from source, using the same compiler I used for zstd, and the results were the same. strace seemed to show zstd read and wrote in chunks 2x the size, but the number of syscalls didn't seem to be nearly enough to explain the difference. It seems to have this "zlibWrapper" subdirectory; its README has some benchmark numbers that I don't fully understand, but some of them seem to match the 2/3 factor I got. I'm wondering if this is a clever tactic to drive adoption—get people to use the zstd executable even when they're still using gzipped files. ;-) Also, the fact that it has support for reading (and, apparently, writing) gzip, lz4, and xz on top of its own format really makes "z standard" an appropriate name.
- emj 9y ago
  A zlib feature is being patent free, that might very well affect speed. I do not know if that is the issue you are describing.
  - waterhouse 9y ago
    So, the zstd executable links to libz.so (says ldd), and I don't think it reimplements the core algorithmic part of gzip. I gather that it's simply more intelligent in how it uses the API... e.g. it says some things about reusing contexts, though that seems to apply only to handling multiple files.
unixhero 9y ago
So like 'middle out'
deleted 9y ago
[deleted]
karmicthreat 9y ago
Not really totally this topic, but I deal with a bunch of 8 Gig images of device firmwares. Is there anything that can make compressing these and de-duplicating these images fast? I have used borg in the past, but archive operations are just seem slow.
- cjalmeida 9y ago
  You may want to store them uncompressed and rely on ZFS block level deduplication. It's similar to borg but may be faster. Also, you may want to play with borg options and use lz4
DyslexicAtheist 9y ago
TL;DR: Facebook acquires "Pied Piper" to gain exclusive rights on the Mean-Jerk-Time (MJT) compression patent; renames it to Zstandard for political correctness.
jokoon 9y ago
When one talks about general data compression, isn't that relevant that it generally applies to text compression, and not video or audio? I guess general data compression works on audio and video, but most of the time you either choose to compress text, audio, video or you create a file format that indexes your data.
powturbo 9y ago
see Compression Benchmark at (https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench)

286 points by josephscott 9y ago

We've been using zstd for a while in production, switching over from rar/gzip/7z for archiving huge XML files. It's speed to compression ratio is really impressive. Hats off to the development team.

loeg 9y ago

We added zstd to largely supersede gzip use at work. It performs better than oldschool gzip in both compression and speed. That said, I'm not sure I'd call it a "real-time" compression algorithm. It's still an factor of 2x slower than lz4 compress and a factor of 4x slower decompress.

michaelmior 9y ago
It's not clear exactly what "real-time" is supposed to mean, but I don't think it makes sense to define it in terms of other algorithms. If someone developed an algorithm faster than lz4, would lz4 cease to be real-time?
- joosters 9y ago
  I thought that 'real-time' is not about how fast a program is. Instead, it's about being able to guarantee that a program will always complete in under a fixed amount of time, under all circumstances. I can't see how that applies to Zstandard either.
  - openasocket 9y ago
    Hard real time certainly means that. I've seen "soft real-time" used to refer to programs where the standard deviation of the runtime given random input is low.
  - michaelmior 9y ago
    Yes, I that would be a correct definition although practically speaking the deadline that is guaranteed is important as well. Part of the problem is that the term "real-time" is overloaded.
  - hinkley 9y ago
    It’s a definition not used much anymore but this relates to on demand (streaming) versus ahead of time data processing. With only a small delay, can I compress and decompress at least one second of data per second, indefinitely. Think live TV versus film.
    - chowells 9y ago
      That's usually called an "online" algorithm, in my experience. It does seem to agree with the way it's used in this context.
      - foota 9y ago
        Online isn't quite the right distinction here imo, since that's really more about possibility than speed. An algorithm could be online (doesn't require all the data at once to run) but still not be fast enough to stream.
  - hackcasual 9y ago
    Talking about compression you're generally talking about things that when you factor in network overhead have a neutral or better impact on end-to-end time.
bgirard 9y ago
It's really subjective based on your computing resources client CPU + network speed and your server CPU if you can't compress ahead of time. Here's a great page that captures this: https://quixdb.github.io/squash-benchmark/#transfer-plus-processing https://quixdb.github.io/squash-benchmark/#transfer-plus-pro... TL;DR: Your optimal compression algorithm will vary based on these parameters. There's no universal answer.

teej 9y ago

We switched many of our Redshift compression encodings to zstd. Works fantastically on text and JSON.

ComputerGuru 9y ago

zstd is incredible, but just in case the thought hasn't occurred to someone here that may benefit from it: if you're in control of the send and the receive, type-specific compression is hard to beat. For example, if you know you're dealing with text, you can use snappy, if you know you're dealing with images, webp, videos x264 (or x265 if you only care about decode speed and encoded size), etc and then fall back to zstd only when you don't have a specific compressor for the chosen file type.

halayli 9y ago
I'd pick zstd over snappy for text
hackcasual 9y ago
zstd is in snappy's domain, as a low overhead way of reducing bandwidth usage. You've got devs writing some service that talks in JSON or protobuf, just rub a little zstd on it, and bingo, your bandwidth is reduced.
Smerity 9y ago
Agreed that if you have the control (and potentially the time depending on the algorithm), type specific compression is the way to go. Having said that, zstd beats Snappy handily for text ^_^ On enwik8 (100MB of Wikipedia XML encoded articles, mostly just text), zstd gets you to ~36MB, Snappy gets you to ~58MB, while gzip will get you to 36MB. If you turn up the compression dials on zstd, you can get down to 27MB - though instead of 2 seconds to compress it takes 52 seconds on my laptop. Decompression takes ~0.3 seconds for the low or high compression rate.
- wolf550e 9y ago
  ppmd will get you better compression on text (better than zstd and even better than lzma), but it is slow to compress and decompress. Use `7z a -m0=PPMd demo.7z demo.txt`
- hokkos 9y ago
  Why not use a XML/XSD specific compression like EXI for that ? https://www.w3.org/TR/exi-primer/ https://www.w3.org/TR/exi-primer/

tzs 9y ago

> Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off This would have been very handy about 17 years ago when I was doing a virtual CD-ROM driver, and needed to store compressed images, and the client wanted a wide range of speed/compression options for the user--much wider than zlib offered. I ended up doing a hack that was either the most disgusting thing I have ever done, or was brilliant. I have not been able to decide which. My hack: I gave them a compression/speed slider that went from 0 to 100. If the slider was set to N, I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks. Repeat until the whole image is stored. The client loved it.

bgirard 9y ago
Performance wise it's far from ideal. In your N block you're going to spend a lot of time, per-byte, to compress while leaving trivially compressible data in the remaining blocks. For N=50, you'd get much better compression for the same amount of cycles applying a faster compression to all the bytes.
deleted 9y ago
[deleted]
gopalv 9y ago
> I would apply zlib at maximum compression to N consecutive blocks, and then apply no compression to 100-N consecutive blocks Zlib is actually highly configurable - it has a huge number of tweaks beyond just that one flag of size. DEFAULT_STRATEGY is pretty much only one possible strategy. You can tweak the LZ77, RLE and Huffman separately to get different compression performance (the levels correspond to different LZ77 lookups and deflate_fast vs deflate_slow). In my work, I use different zlib strategies for different data (i.e already bit-packed integers vs raw strings), which works out pretty well. My current complaint is that the Zlib algorithm doesn't quite detect record boundaries when building a dictionary, which is all only relevant to the compression algorithm, because the output is eventually just a DEFLATE stream (Zopfli is an example of an encoder only Zlib library). Zstandard has a much wider set of options to use already built-in (fast, greedy, lazy, btlazy, btopt, btultra) etc.

cakoose 9y ago

In the "Compression Speed vs Ratio" graph, the Y-axis doesn't start at zero. If it were changed to start at zero, it would be easier to evaluate the magnitude of the compression ratio change at a glance. IMO, that's probably worth the extra whitespace in the graph.

natbobc 9y ago

Why is this site an SPA?!?!?

natbobc 9y ago
Genuine question with everything happening in Intel land these days I've turned off JS except for a select few websites. Seems like an interesting project though.
akx 9y ago
It's... not? It's a static page with some JavaScript-driven charts.
- userbinator 9y ago
  It's a page with the majority of the content wrapped in an XMP element with style="display:none;". I'd consider that pretty hostile.

revelation 9y ago

From that benchmark list I'd prefer lz4..

terrelln 9y ago
lz4 and zstd are complementary, not competitive. lz4 targets applications where extremely fast decompression speed (and compression speed) are required, at the expense of compression ratio. However, many applications would prefer to spend a bit (or a lot) more CPU in exchange for better compression, while maintaining fast decompression speed.

terrelln 9y ago

Zstd recently added a long range mode that can find matches up to 2 GB in the past using a specialized algorithm [1]. It can be enabled on the command line with `zstd --long` for a 128 MB window, or `--long=windowLog` for a `2^windowLog` byte window. [1] https://github.com/facebook/zstd/releases/tag/v1.3.2 https://github.com/facebook/zstd/releases/tag/v1.3.2

Willson50 9y ago
A non-obvious drawback of this is you give up multi-threaded compression (maybe fixed in a later release). You also need to use the same memory size when decompressing.
- terrelln 9y ago
  We intend to improve the interaction of multithreaded compression and long range mode in a future release, as they go hand in hand. We also intend to support mmap in the CLI, which should help reduce the memory usage for very large window sizes when writing to a file.
  - ghusbands 9y ago
    I understand this to mean that you believe the pages in the long range aren't accessed frequently. Otherwise, mmap doesn't reduce memory usage so much as just hide it. Heavy mmap usage (VMWare sometimes uses an mmap to hold a guest's memory, for example) doesn't show up as memory usage in any system monitoring tools, and the system starts to thrash long before apparent memory usage gets high. Maybe someone here has a solution to that?
    - terrelln 9y ago
      Generally, pages in long ranges will be accessed less than those in a short ranges, and pages beyond the window size will never be accessed. You can construct data that doesn't fit this pattern, but its probably a good bet to make. With a 1-2 GB window size, I would expect there to be large chunks of the window that are rarely accessed. The long range matcher has a configurable match size, where it will only look for matches that are at least that large. By default it is 64 B, but by making it larger, say 4 KB, you can ensure that if you are going to force a page fault, you get enough benefit.

gateway7 9y ago

That patent clause though. Stick to the fork from before Facebook bought the man and the algorithm.

terrelln 9y ago
It was relicensed in v1.3.1. It is now dual licensed under BSD without the patents clause [1] and GPLv2 [2]. GPLv2 was added for inclusion in the Linux kernel. [1] https://github.com/facebook/zstd/blob/dev/LICENSE https://github.com/facebook/zstd/blob/dev/LICENSE [2] https://github.com/facebook/zstd/blob/dev/COPYING https://github.com/facebook/zstd/blob/dev/COPYING

cmurf 9y ago

Supported on Btrfs since kernel 4.14. Like other compression options, it's a mount time option. Edit: Also in squashfs. Here's the git pull request which includes some benchmarks. https://lkml.org/lkml/2017/9/11/369 https://lkml.org/lkml/2017/9/11/369

terrelln 9y ago
Upstream SquashFUSE also has zstd support. https://github.com/vasi/squashfuse https://github.com/vasi/squashfuse

2pointsomone 9y ago

That's the best Weissman score I have ever seen.

2pointsomone 9y ago
Come on people, take a joke
- hodl 9y ago
  I get ya!

desertrider12 9y ago

I wonder how the RAD Game Tools compressors would fit into that benchmark list. In their benchmarks Oodle Selkie has roughly the same ratio as Zlib but is 3-4x faster at decompression than Zstd (not the same benchmark though). http://www.radgametools.com/oodle.htm http://www.radgametools.com/oodle.htm

blattimwind 9y ago
Asset compression has a specific optimization function (Charles Bloom of RAD Game Tools has a ton about this on his blog, cbloomrants): maximize output bandwidth given this specific CPU and input bandwidth. Compression speed, for example, doesn't really matter. Sure, it should be something reasonable (e.g. not like PAQ), but if you can bump the above metric by 10 % while compression takes 50 % longer: probably worth it.
- desertrider12 9y ago
  Aha, he actually addressed zstd in a post. tldr is zstd usually lower ratio and lower speed than mermaid but sometimes better ratio for text.

bajsejohannes 9y ago

Some good discussion in earlier postings too: https://hn.algolia.com/?query=zstandard&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=zstandard&sort=byPopularity&pr...

reacharavindh 9y ago

Anyone using zstd along with an incremental rsync approach? Like gzip --rsyncable ? Say I have 1000 files. I want to compress them and let the cron rsync do it's thing. Next day, if only one file had changed, rsync should pickup only the differential instead of the whole archive. Or is there a better way of doing it?

cjhanks 9y ago

I use the C bindings for ZStandard, it performs very nicely as an algorithm. I wish the documentation was a little easier to use. I am still not sure if the provided examples really properly handle the degenerate cases of /dev/urandom and /dev/zero when using `ZSTD_DStream{In,Out}Size` and friends. In any case - thanks for releasing this, it's been very helpful to me.

jakozaur 9y ago

In a lot of big data workflows, zstd is no brainer replacement of gzip. Always faster decompression, plus you can save tons of storage and data transfer.

lars_francke 9y ago

Last time this was discussed Zstandard didn't have a splittable mode and by the looks of it[1] they still don't. That doesn't make it a bad algorithm it just means that it's not a good choice yet for Hadoop et al. As far as I know no container format has implemented Zstandard yet. Does anyone know any better? It seems like we could use a better alternative to Snappy. [1] https://github.com/facebook/zstd/issues/395 https://github.com/facebook/zstd/issues/395

gopalv 9y ago
> As far as I know no container format has implemented Zstandard yet. ORC has a reserved future enum for ZSTD [1] (my guess is that FB already uses it). The original headache was the patent licensing scheme[2] for the actual codebase, but now that hadoop has a ZStandard built into libhadoop.so, this is much more easy to ship to Apache. My branch is so far behind now that I probably would implement it faster if I started over & used the libhadoop.so impl as the basis - https://github.com/t3rmin4t0r/orc/tree/zstd https://github.com/t3rmin4t0r/orc/tree/zstd [1] - https://issues.apache.org/jira/browse/ORC-46 https://issues.apache.org/jira/browse/ORC-46 [2] - https://github.com/facebook/zstd/issues/775 https://github.com/facebook/zstd/issues/775

beagle3 9y ago

I used to object to using zstd based on Facebook's onerous license and patent policy; but now that zstd is plain BSD+GPLv2, I endorse it.

waterhouse 9y ago

A curious fact I encountered: First, the zstdcat executable that ships with zstd (actually a symlink to zstd itself, roughly equivalent to "zstd -cqdf") can decompress .gz files. Second, zstd is faster than gzip at decompressing .gz files, taking about 2/3 as long. I'm not kidding. I couldn't believe it myself, but subsequent testing stubbornly bore it out—on one file that was 15 MB after compression, and on a mix of smaller files. I tried compiling gzip from source, using the same compiler I used for zstd, and the results were the same. strace seemed to show zstd read and wrote in chunks 2x the size, but the number of syscalls didn't seem to be nearly enough to explain the difference. It seems to have this "zlibWrapper" subdirectory; its README has some benchmark numbers that I don't fully understand, but some of them seem to match the 2/3 factor I got. I'm wondering if this is a clever tactic to drive adoption—get people to use the zstd executable even when they're still using gzipped files. ;-) Also, the fact that it has support for reading (and, apparently, writing) gzip, lz4, and xz on top of its own format really makes "z standard" an appropriate name.

emj 9y ago
A zlib feature is being patent free, that might very well affect speed. I do not know if that is the issue you are describing.
- waterhouse 9y ago
  So, the zstd executable links to libz.so (says ldd), and I don't think it reimplements the core algorithmic part of gzip. I gather that it's simply more intelligent in how it uses the API... e.g. it says some things about reusing contexts, though that seems to apply only to handling multiple files.

unixhero 9y ago

So like 'middle out'

deleted 9y ago

[deleted]

karmicthreat 9y ago

Not really totally this topic, but I deal with a bunch of 8 Gig images of device firmwares. Is there anything that can make compressing these and de-duplicating these images fast? I have used borg in the past, but archive operations are just seem slow.

cjalmeida 9y ago
You may want to store them uncompressed and rely on ZFS block level deduplication. It's similar to borg but may be faster. Also, you may want to play with borg options and use lz4

DyslexicAtheist 9y ago

TL;DR: Facebook acquires "Pied Piper" to gain exclusive rights on the Mean-Jerk-Time (MJT) compression patent; renames it to Zstandard for political correctness.

jokoon 9y ago

When one talks about general data compression, isn't that relevant that it generally applies to text compression, and not video or audio? I guess general data compression works on audio and video, but most of the time you either choose to compress text, audio, video or you create a file format that indexes your data.

powturbo 9y ago

see Compression Benchmark at (https://github.com/powturbo/TurboBench https://github.com/powturbo/TurboBench)