5 ms·

RAD's ground breaking lossless compression product benchmarked

80 points by rinesh 10y ago

mariojv 10y ago
Impressive. This reminds me of the series Silicon Valley, though.
deleted 10y ago
[deleted]
aorth 10y ago
Impressive results for RAD's suite of lossless compression products, which is apparently called "Oodle"[0] and was on HN a few months ago[1]. I can't put my finger on it, but something about the blog post bothers me. The "groundbreaking" compression suite is closed source and commercial, and even the data set used for testing in the post isn't public. Con Kolivas (ck), the author of lrzip, commented and inquired about the corpus but the blog post author says he "can't release it". I dunno, it just seems like a weird exchange to me. [0] http://www.radgametools.com/oodlewhatsnew.htm http://www.radgametools.com/oodlewhatsnew.htm [1] https://news.ycombinator.com/item?id=11583898 https://news.ycombinator.com/item?id=11583898
- huhtenberg 10y ago
  > something about the blog post bothers me Yep, same here. Way too much excitement for something that shows no "groundbreaking" advantage over other codecs based on his own benchmark results.
  - Dylan16807 10y ago
    Right. There's a niche where it excels, giving you lz4 decompression speed combined with zlib-esque ratios, but it accomplishes that by having significantly slower compression than either of them. It's not really breaking away from the pack in any way that I can see.
    - jeffreyrogers 10y ago
      I think it's targeted at the game industry, so the slow compression doesn't matter. It just needs to decompress quickly for the user.
  - angersock 10y ago
    Go read Cbloom blog for much better information, and some implementation deets.
  - deleted 10y ago
    [deleted]
- Arnt 10y ago
  The author is a game person and primarily interested in game compression (compression at compiletime, decompression at runtime), so if he's lazy and smart, then his corpus consists of assets from the games he's worked on. That would be both conveniently available and realistic. Of course he can't say in public that he's kept copies of any of those assets. They belong to his previous employers.
- TylerE 10y ago
  Isn't there an argument to be made for keeping the corpus private? Namely to prevent the unscrupulous from 'cheating to the test' - as games companies have a long, long history of doing.
- ksk 10y ago
  You don't explain what exactly is wrong with it being commercial and closed-source. That is the default way businesses make money.
  - sp332 10y ago
    It means the claims are unverifiable. The test set especially should be available.
    - ksk 10y ago
      >It means the claims are unverifiable. The test set especially should be available. Eh..Why should anyone in the world care that a compressor can compress test data well? Except vaguely, it might serve as a motivator to investigate the use of said compressor for their own purpose. But then again, this is simply marketing, for developers. To me, its like saying "the new compiler version is faster" on this collection of source files. Personally, I couldn't care less what the test set is, if the compiler is not faster for my own use case. So yeah, if anyone is interested, they could probably get an evaluation copy, try it out on their own data and decide for themselves.
      - sp332 10y ago
        Of course it would be better, but I thought it was implausible that the code would be public given that the corpus is private. I guess it's possible though.
      - michaelmrose 10y ago
        Then the blog post is a waste of everyone's time and ought to be deleted from hacker news. Either it provides enough information to replicate the benchmarks or its spam. If it provided more freely available information other parties could test immediately and post and this would provide additional feedback to help parties decide if it is actually worthwhile.
        ksk 10y ago
- jeremiep 10y ago
  My litmus test is checking how many "titles" authors list on their pages. The more titles the less likely I am to take their word seriously. Pretty crude heuristic, but seems to work with very few false positives!
  - late2part 10y ago
    I feel your heuristic is simple and brilliant.
vardump 10y ago
RAD's packer is nice for cases when compression time is not the most important aspect. Seems like LZ4 is still unbeatable if you need as fast compression as possible. Think kernel memory page compression (to avoid swapping pages to disk) and compressing data destined to a high bandwidth sink. I'm just guessing, but probably LZ4 has also less power consumption. If your output device can do over ~100 megabytes per second, LZ4 seems to win easily. Optimal compression algorithm selection seems to depend on capacity and bandwidth constraints of output device.
- aktau 10y ago
  I often use a script I hacked together called memo [1] when exploring the commandline. Today I was running a pipeline like this: $ memo expensive-with-lots-of-output | memo -s expensive | less I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. ("memo -s" absorbs standard input and calculates a checksum over it and the command arguments, if that checksum is in the cache, it knows it can just output the cached value instead of running the command). So I started looking around if there was nothing better around. From the Squish benchmarks posted elsewhere on this article [2], I noticed that zstd(1) seemed to make the right trade-offs. I required compression speed > 50MB/s, decompression speed at least as fast, and as high a compression ration as possible. zstd(1) seems to work brilliantly and is very fast, it was easy to compile too, just running make in its directory was sufficient for me. I love those kinds of projects. I've made it the default over lz4. What gave me even more confidence in this change is that zstd is by the author of lz4. [1]: https://github.com/aktau/dotfiles/blob/master/bin/memo https://github.com/aktau/dotfiles/blob/master/bin/memo (currently a version that doesn't prefer zstd(1) yet, I need to git push). [2]: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/
  - vardump 10y ago
    > I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. It probably depend on how much bandwidth you got.
- fulafel 10y ago
  The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2] [1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me... [2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html
Klasiaster 10y ago
For me the most interesting benchmark for most use cases is that from the squash library for it's plugins, because it's interactive and visually well made: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/ And these high end benchmarks for bigger data: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/10gb.html http://mattmahoney.net/dc/10gb.html
deleted 10y ago
[deleted]
DerDangDerDang 10y ago
Lots more stats and graphs on Charles Bloom's blog (RAD employee working on compression) - http://cbloomrants.blogspot.co.uk/ http://cbloomrants.blogspot.co.uk/
daniel_reetz 10y ago
Any insight into which tool was used to produce these graphs?
Daviey 10y ago
Interesting preface to this, http://richg42.blogspot.co.uk/2016/01/zlib-in-serious-danger-of-becoming.html http://richg42.blogspot.co.uk/2016/01/zlib-in-serious-danger... However, the compression is nothing compared to Pied Piper's algorithm.
- squarepegg 10y ago
  Someone had to say it.
ctur 10y ago
I would love to see Kraken taken for a spin by objective third parties in a fully transparent way that compared it to the available alternatives on equal footing. Sadly, a compression benchmark that doesn't detail the compiler, hardware, corpus, or even version of the libraries used is not very compelling. If others cannot reproduce your results, the results aren't of much value beyond marketing.
- jacobolus 10y ago
  Go read cbloom’s posts: http://cbloomrants.blogspot.com http://cbloomrants.blogspot.com http://www.cbloom.com/rants.html http://www.cbloom.com/rants.html http://www.cbloom.com/oodle_arm_report/ http://www.cbloom.com/oodle_arm_report/ Stuff like the compilers used for Oodle are detailed there, and a bunch more benchmarks are published using (I believe) open corpora.
willricketts 10y ago
But what's its Weissman score?
willricketts 10y ago
But what's its Weissman score?
willricketts 10y ago
But what's its Weissman score?
rogerdpack 10y ago
comparison to lzturbo possible? (Unfortunately also closed source, sich).

80 points by rinesh 10y ago

mariojv 10y ago

Impressive. This reminds me of the series Silicon Valley, though.

deleted 10y ago

[deleted]

aorth 10y ago

Impressive results for RAD's suite of lossless compression products, which is apparently called "Oodle"[0] and was on HN a few months ago[1]. I can't put my finger on it, but something about the blog post bothers me. The "groundbreaking" compression suite is closed source and commercial, and even the data set used for testing in the post isn't public. Con Kolivas (ck), the author of lrzip, commented and inquired about the corpus but the blog post author says he "can't release it". I dunno, it just seems like a weird exchange to me. [0] http://www.radgametools.com/oodlewhatsnew.htm http://www.radgametools.com/oodlewhatsnew.htm [1] https://news.ycombinator.com/item?id=11583898 https://news.ycombinator.com/item?id=11583898

huhtenberg 10y ago
> something about the blog post bothers me Yep, same here. Way too much excitement for something that shows no "groundbreaking" advantage over other codecs based on his own benchmark results.
- Dylan16807 10y ago
  Right. There's a niche where it excels, giving you lz4 decompression speed combined with zlib-esque ratios, but it accomplishes that by having significantly slower compression than either of them. It's not really breaking away from the pack in any way that I can see.
  - jeffreyrogers 10y ago
    I think it's targeted at the game industry, so the slow compression doesn't matter. It just needs to decompress quickly for the user.
- angersock 10y ago
  Go read Cbloom blog for much better information, and some implementation deets.
- deleted 10y ago
  [deleted]
Arnt 10y ago
The author is a game person and primarily interested in game compression (compression at compiletime, decompression at runtime), so if he's lazy and smart, then his corpus consists of assets from the games he's worked on. That would be both conveniently available and realistic. Of course he can't say in public that he's kept copies of any of those assets. They belong to his previous employers.
TylerE 10y ago
Isn't there an argument to be made for keeping the corpus private? Namely to prevent the unscrupulous from 'cheating to the test' - as games companies have a long, long history of doing.
ksk 10y ago
You don't explain what exactly is wrong with it being commercial and closed-source. That is the default way businesses make money.
- sp332 10y ago
  It means the claims are unverifiable. The test set especially should be available.
  - ksk 10y ago
    >It means the claims are unverifiable. The test set especially should be available. Eh..Why should anyone in the world care that a compressor can compress test data well? Except vaguely, it might serve as a motivator to investigate the use of said compressor for their own purpose. But then again, this is simply marketing, for developers. To me, its like saying "the new compiler version is faster" on this collection of source files. Personally, I couldn't care less what the test set is, if the compiler is not faster for my own use case. So yeah, if anyone is interested, they could probably get an evaluation copy, try it out on their own data and decide for themselves.
    - sp332 10y ago
      Of course it would be better, but I thought it was implausible that the code would be public given that the corpus is private. I guess it's possible though.
    - michaelmrose 10y ago
      Then the blog post is a waste of everyone's time and ought to be deleted from hacker news. Either it provides enough information to replicate the benchmarks or its spam. If it provided more freely available information other parties could test immediately and post and this would provide additional feedback to help parties decide if it is actually worthwhile.
      - ksk 10y ago
jeremiep 10y ago
My litmus test is checking how many "titles" authors list on their pages. The more titles the less likely I am to take their word seriously. Pretty crude heuristic, but seems to work with very few false positives!
- late2part 10y ago
  I feel your heuristic is simple and brilliant.

vardump 10y ago

RAD's packer is nice for cases when compression time is not the most important aspect. Seems like LZ4 is still unbeatable if you need as fast compression as possible. Think kernel memory page compression (to avoid swapping pages to disk) and compressing data destined to a high bandwidth sink. I'm just guessing, but probably LZ4 has also less power consumption. If your output device can do over ~100 megabytes per second, LZ4 seems to win easily. Optimal compression algorithm selection seems to depend on capacity and bandwidth constraints of output device.

aktau 10y ago
I often use a script I hacked together called memo [1] when exploring the commandline. Today I was running a pipeline like this: $ memo expensive-with-lots-of-output | memo -s expensive | less I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. ("memo -s" absorbs standard input and calculates a checksum over it and the command arguments, if that checksum is in the cache, it knows it can just output the cached value instead of running the command). So I started looking around if there was nothing better around. From the Squish benchmarks posted elsewhere on this article [2], I noticed that zstd(1) seemed to make the right trade-offs. I required compression speed > 50MB/s, decompression speed at least as fast, and as high a compression ration as possible. zstd(1) seems to work brilliantly and is very fast, it was easy to compile too, just running make in its directory was sufficient for me. I love those kinds of projects. I've made it the default over lz4. What gave me even more confidence in this change is that zstd is by the author of lz4. [1]: https://github.com/aktau/dotfiles/blob/master/bin/memo https://github.com/aktau/dotfiles/blob/master/bin/memo (currently a version that doesn't prefer zstd(1) yet, I need to git push). [2]: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/
- vardump 10y ago
  > I noticed that lz4 (the default (de)compressor memo uses) was the bottleneck. It probably depend on how much bandwidth you got.
fulafel 10y ago
The RAD guys have entries in that dept too. "Selkie is 1.3X faster to decode than LZ4 and gets 12% more compression" [1] though "Selkie does not currently have a super fast encoder" [2] [1] http://cbloomrants.blogspot.com/2016/07/introducing-oodle-mermaid-and-selkie.html http://cbloomrants.blogspot.com/2016/07/introducing-oodle-me... [2] http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html http://cbloomrants.blogspot.nl/2016/07/oodle-selkie.html

Klasiaster 10y ago

For me the most interesting benchmark for most use cases is that from the squash library for it's plugins, because it's interactive and visually well made: https://quixdb.github.io/squash-benchmark/unstable/ https://quixdb.github.io/squash-benchmark/unstable/ And these high end benchmarks for bigger data: http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/silesia.html http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/text.html http://mattmahoney.net/dc/10gb.html http://mattmahoney.net/dc/10gb.html

deleted 10y ago

[deleted]

DerDangDerDang 10y ago

Lots more stats and graphs on Charles Bloom's blog (RAD employee working on compression) - http://cbloomrants.blogspot.co.uk/ http://cbloomrants.blogspot.co.uk/

daniel_reetz 10y ago

Any insight into which tool was used to produce these graphs?

Daviey 10y ago

Interesting preface to this, http://richg42.blogspot.co.uk/2016/01/zlib-in-serious-danger-of-becoming.html http://richg42.blogspot.co.uk/2016/01/zlib-in-serious-danger... However, the compression is nothing compared to Pied Piper's algorithm.

squarepegg 10y ago
Someone had to say it.

ctur 10y ago

I would love to see Kraken taken for a spin by objective third parties in a fully transparent way that compared it to the available alternatives on equal footing. Sadly, a compression benchmark that doesn't detail the compiler, hardware, corpus, or even version of the libraries used is not very compelling. If others cannot reproduce your results, the results aren't of much value beyond marketing.

jacobolus 10y ago
Go read cbloom’s posts: http://cbloomrants.blogspot.com http://cbloomrants.blogspot.com http://www.cbloom.com/rants.html http://www.cbloom.com/rants.html http://www.cbloom.com/oodle_arm_report/ http://www.cbloom.com/oodle_arm_report/ Stuff like the compilers used for Oodle are detailed there, and a bunch more benchmarks are published using (I believe) open corpora.

willricketts 10y ago

But what's its Weissman score?

willricketts 10y ago

But what's its Weissman score?

willricketts 10y ago

But what's its Weissman score?

rogerdpack 10y ago

comparison to lzturbo possible? (Unfortunately also closed source, sich).