3 ms·
I wonder how much of this is that relatively simple techniques work very well on small amounts of data, and that more complex techniques don't perform sufficien
by gwern 12y ago
I wonder how much of this is that relatively simple techniques work very well on small amounts of data, and that more complex techniques don't perform sufficiently better to justify baking them into a compression format?
That is, to some extent I subscribe to the compression-as-intelligence school of thought: compressors are tiny little AIs which try to predict regularities in the bitstreams they are given ( http://prize.hutter1.net/ http://prize.hutter1.net/ http://mattmahoney.net/dc/dce.html http://mattmahoney.net/dc/dce.html http://www.danburfoot.net/research.html http://www.danburfoot.net/research.html ).
But when we look at the state of the art like ZPAQ, the AI techniques used don't seem to be much more complex than, say, a one or two layer neural network which might as well be from the 1970s. You don't see anything fancy like deep networks or other modern staples like random forests.
So this makes me wonder: maybe compression performance has stagnated because we're not willing to provide compression algorithms extremely large amounts of data or runtime, and so simple algorithms really do perform best with the minimal resources we're willing to use for compression. (People are happy to run neural networks on thousands of GPUs with many gigabytes of data and wait weeks for training to finish; can you imagine a compression utility which required that?)
- khitchdee 12y agoI think it's also partly due to the way that the typical dev team develops their algos. They all start with a base version and later push out revisions to their earlier base work. The tendency is therefore to just refine and not go for more game-changing answers. If they executed over longer terms and were more isolated from their markets, we might have seen more range in the changes made.
- vtuulos 12y agoIt naturally follows from the compression-as-intelligence school of thought that building general-purpose compression/intelligence is hard. I very much believe in domain-specific intelligence, and correspondingly domain-specific compression. Here's a practical business use case for lossless compression (in-memory analytics), which I have been developing: http://tuulos.github.io/pydata-2014/ http://tuulos.github.io/pydata-2014/ In contrast to general-purpose encoders, this approach is extremely data-intensive, compressing terabytes per chunk.
- gwern 12y ago> that building general-purpose compression/intelligence is hard. Yes, but my point is. that we seem to be doing better at general-purpose intelligence than at compression despite the apparent equivalence of progress.