4 ms·
Good point. While it won't beat a hand-optimized algorithm for a specific use case, compression with a dictionary (like zstd supports/encourages) is sort of a p
by felixhandte 9y ago
Good point. While it won't beat a hand-optimized algorithm for a specific use case, compression with a dictionary (like zstd supports/encourages) is sort of a partial specialization of the algorithm to the type of data you're compressing.
- beagle3 9y agoProvided that the compression stems from repetition of blocks. If you have a file with 16-bit integers, each either exactly 1 or 2 greater than the previous one, you will have 128K with no repetition that zstd will be unable to compress; However, if you transpose the bytes (all high order bytes, followed by all low-order bytes), the compression will be very significant; similarly, if you replace the numbers with their difference. The correct term from information theory is that it approximates a "universal" compressor with respect to observable markov or fsmx sources.