4 ms·

I don't see comparisons against any version of Zstandard, much less Zstandard 1.4, there. (Besides, regardless of the state of benchmarks today, the momentum i

I don't see comparisons against any version of Zstandard, much less Zstandard 1.4, there. (Besides, regardless of the state of benchmarks today, the momentum is clearly with Zstandard, with Facebook, Intel, and the open source community behind it. The basic lossless compression algorithms haven't changed much since the late 1970s. Making compression fast is mostly just a long slog of engineering hurdles, the kind that big companies are very good at doing.)

yoklov 8y ago
http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-leviathan.html http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-... is versus zstd 1.3.3. There are others too on that blog. Oodle and zstd are both very impressive, it's a shame the former is not free.
_wmd 8y ago
Zstandard uses ANS in some (all?) of its modes, that's very much brand new -- 2014
jacobolus 8y ago
> regardless of the state of benchmarks today, the momentum is clearly with Zstandard, with Facebook, Intel, and the open source community behind it. This argument by hand-wave doesn’t match up with demonstrated progress over the past few years. Irrespective of the skill or insight of its individual engineers, I would be surprised if a schizophrenic hack-it-with-duct-tape kind of engineering culture like Facebook could keep up with a focused and motivated expert like cbloom over the medium term (though I suppose the latter could conceivably at some point lose interest in the domain and switch to building something else).
- pcwalton 8y ago
  I've seen this play out before. The history of jemalloc, particularly how it rapidly outpaced virtually every other allocator while under development at Facebook, suggests otherwise. At this point jemalloc is so good that there is little reason other than NIH to use anything else (unless you want an especially hardened allocator for security and are willing to give up some performance). Even Google uses it in Android. Zstandard is likewise deservedly on track to dominate the lossless compression space.
  - vthriller 8y ago
    > At this point jemalloc is so good Some folks disagree: > [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy. via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-status-of-ruby-memory-trimming-and-how-you-can-help-with-testing.html https://www.joyfulbikeshedding.com/blog/2019-03-29-the-statu...
- mappu 8y ago
  Zstd is from Facebook, sure, but to be more specific its development is lead by Yann Collet (of LZ4 fame) who is unquestionably a focused and motivated expert.

yoklov 8y ago

http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-leviathan.html http://cbloomrants.blogspot.com/2018/06/zstd-is-faster-than-... is versus zstd 1.3.3. There are others too on that blog. Oodle and zstd are both very impressive, it's a shame the former is not free.

_wmd 8y ago

Zstandard uses ANS in some (all?) of its modes, that's very much brand new -- 2014

jacobolus 8y ago

> regardless of the state of benchmarks today, the momentum is clearly with Zstandard, with Facebook, Intel, and the open source community behind it. This argument by hand-wave doesn’t match up with demonstrated progress over the past few years. Irrespective of the skill or insight of its individual engineers, I would be surprised if a schizophrenic hack-it-with-duct-tape kind of engineering culture like Facebook could keep up with a focused and motivated expert like cbloom over the medium term (though I suppose the latter could conceivably at some point lose interest in the domain and switch to building something else).

pcwalton 8y ago
I've seen this play out before. The history of jemalloc, particularly how it rapidly outpaced virtually every other allocator while under development at Facebook, suggests otherwise. At this point jemalloc is so good that there is little reason other than NIH to use anything else (unless you want an especially hardened allocator for security and are willing to give up some performance). Even Google uses it in Android. Zstandard is likewise deservedly on track to dominate the lossless compression space.
- vthriller 8y ago
  > At this point jemalloc is so good Some folks disagree: > [Ruby's] memory usage only reduces when using jemalloc 3; memory usage is still high when using jemalloc 5. Nobody knows why, so that makes the choice of defaulting to jemalloc very dodgy. via https://www.joyfulbikeshedding.com/blog/2019-03-29-the-status-of-ruby-memory-trimming-and-how-you-can-help-with-testing.html https://www.joyfulbikeshedding.com/blog/2019-03-29-the-statu...
mappu 8y ago
Zstd is from Facebook, sure, but to be more specific its development is lead by Yann Collet (of LZ4 fame) who is unquestionably a focused and motivated expert.