4 ms·
It's fantastic to see so much fascinating work in compression these days after a fair amount of stagnation. I know there had been a lot of specialized but codec
by tmd83 11y ago
It's fantastic to see so much fascinating work in compression these days after a fair amount of stagnation. I know there had been a lot of specialized but codec work but the recent discoveries are very much for wide use.
You now have LZ4, Brotil, zstd, snappy, lzfse, lzma all pretty useful practical codec.
Brotil is interesting though. It can be an easy replacement for zlib at level 1 with fairly higher compression then zlib at similar speed.
On the other hand with with lzma it can handily beat lzma but with an even slower compression rate (from the levels that they published in the benchmark) but on the other hand with much higher decompression speed meaning its very good for distribution work. It would be interesting to see the compression ratio, time for levels between 1-9.
It's actually a much easier replacement for zlib then lzma for some. The benchmark shows only level 1,9 and 11. It seems that it can handily beat lzma but at the cost of compression speed (I wonder who use more memory). Then again its decompression speed is so much better making it a perfect choice for distribution.
What truly surprise me though is the work of 'Yann Collet'. A single person so handily beating google's snappy. Even his zstd work looks ground breaking. When I read couple of weeks ago that he was a part time hobby programmer I just didn't know how to be suitably impressed.
Am I reading right that apples lzfse is based of a mix of lz+zstd?
- huhtenberg 11y agoResearch in compression hardly ever stalled, it just wasn't getting much publicity because it wasn't coming from Google. For example, looking at Brotli it appears to be based on ideas very similar to those explored by the PAQ project [1] not several years ago. [1] https://en.wikipedia.org/wiki/PAQ https://en.wikipedia.org/wiki/PAQ
- tmd83 11y agoOhh I know there has been a lot of work, exotic codecs for a long while. I'm not an expert but I have looked up whats out there fairly often. What I'm saying that most of them had been too slow or specialised to be used by most people. But in recent times those exotic codecs has been transformed, tuned to make a lot of practical codecs.
- acdha 11y agoThe web has definitely had a problem with legacy restrictions driving thinking. We lost nearly a decade with Internet Explorer stagnation and I think that caused a lot of people to assume that the odds of something new going mainstream were too low. Now we're in a very different place where people upgrade software much faster than in the past and that's likely to start showing up in projects like this where it's actually plausible to think of something like Brotli seeing widespread usage within 1-2 years.
- aidenn0 11y agoOn top of that Google can unilaterally upgrade the servers that serve the most popular site on the internet and ~40% of all browsers at the same time.
- sp332 11y ago7zip's implementation of zip has been faster and more effective than zlib for a long time now.
- legulere 11y agoI though lzfse is deflate with finite state entropy instead of huffman: http://fastcompression.blogspot.de/2013/12/finite-state-entropy-new-breed-of.html http://fastcompression.blogspot.de/2013/12/finite-state-entr...
- eln1 11y agoFSE is implementation of this new ANS entropy coding, also used e.g. in lzturbo 1.2. It's surprising that even still using huffman, brotli is slowly approaching its performance: http://encode.ru/threads/2313-Brotli?p=44970&viewfull=1#post44970 http://encode.ru/threads/2313-Brotli?p=44970&viewfull=1#post...
- deleted 11y ago[deleted]