3 ms·

> If you know the structure of your data, you can easily do a better and faster job of compressing than a generic compression algorithm. I'd change this to: th

by ragle 10y ago

> If you know the structure of your data, you can easily do a better and faster job of compressing than a generic compression algorithm. I'd change this to: the ease of doing a better and faster job of compressing than a generic compression algorithm will be a function of the data's Kolmogorov Complexity. You can easily know your data's structure (e.g. strings no larger than N bytes containing natural language in a line-separated file) and not be able to (easily) do better than a generic compressor due to the complexity of the required compression / reconstruction operations.

brilee 10y ago
Not quite; I would describe it as moving the complexity into the algorithm. For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. The knowledge of that correspondence has been moved into the algorithm.
- ragle 10y ago
  Ah, hate I missed this at the time of comment (got on a flight right after I posted and forgot to check back :/). What you're describing (moving complexity into the algorithm) _is_ an increase in Komogorov Complexity, though... so I think we're saying the same thing? i.e. Kolmogorov complexity K(x|y) (x, given y) is the length of the shortest program that on input y, outputs x. So, when you say "move complexity into the algorithm", this is an increase the length of the shortest program that given input <data to compress> outputs <compressed data>. Or have I missed or misunderstood your point?
- gopalv 10y ago
  > For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. Fewer bits does not always mean better compression, particularly if the data has other patterns which is destroyed by the packing. With Apache ORC, I found out that if you bit-pack data to say 7bits vs leaving it as 8 bits, the 8 bits version compressed much more with Zlib than the 7 bit version. This had to do with the data getting a bit offset into the previous byte sequence, until what was a sequence of repeating bytes turned into a pattern which repeats far less often. Leaving the extra bit in place, helped Zlib dictionary encoding and huffman work much better than trying to save a bit. The final kicker was that the 24 bit sequence was faster to read than a 23 or 21 bit sequence, but purely due to the fact that the word aligned stuff can be decoded in SIMD. I'm no better at guessing what would work - "whatever works ... works, so try them.".
  - delhanty 10y ago
    Your comment, particularly the degradation in byte alignment leading to worse subsequent compression, reminds me of the following Hacker News comment by NelsonMinar from then end of November 2016: https://news.ycombinator.com/item?id=13049894 https://news.ycombinator.com/item?id=13049894 >It's crucial to evaluate encoding space usage in the context of compression. For instance gzip(base16(data)) is often smaller than gzip(base64(data)) for practical data. Even though base64 is more efficient than base16, it breaks up data across byte boundaries which then makes gzip significantly less efficient.

by ragle 10y ago

brilee 10y ago

Not quite; I would describe it as moving the complexity into the algorithm. For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. The knowledge of that correspondence has been moved into the algorithm.

ragle 10y ago
Ah, hate I missed this at the time of comment (got on a flight right after I posted and forgot to check back :/). What you're describing (moving complexity into the algorithm) _is_ an increase in Komogorov Complexity, though... so I think we're saying the same thing? i.e. Kolmogorov complexity K(x|y) (x, given y) is the length of the shortest program that on input y, outputs x. So, when you say "move complexity into the algorithm", this is an increase the length of the shortest program that given input <data to compress> outputs <compressed data>. Or have I missed or misunderstood your point?
gopalv 10y ago
> For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. Fewer bits does not always mean better compression, particularly if the data has other patterns which is destroyed by the packing. With Apache ORC, I found out that if you bit-pack data to say 7bits vs leaving it as 8 bits, the 8 bits version compressed much more with Zlib than the 7 bit version. This had to do with the data getting a bit offset into the previous byte sequence, until what was a sequence of repeating bytes turned into a pattern which repeats far less often. Leaving the extra bit in place, helped Zlib dictionary encoding and huffman work much better than trying to save a bit. The final kicker was that the 24 bit sequence was faster to read than a 23 or 21 bit sequence, but purely due to the fact that the word aligned stuff can be decoded in SIMD. I'm no better at guessing what would work - "whatever works ... works, so try them.".
- delhanty 10y ago
  Your comment, particularly the degradation in byte alignment leading to worse subsequent compression, reminds me of the following Hacker News comment by NelsonMinar from then end of November 2016: https://news.ycombinator.com/item?id=13049894 https://news.ycombinator.com/item?id=13049894 >It's crucial to evaluate encoding space usage in the context of compression. For instance gzip(base16(data)) is often smaller than gzip(base64(data)) for practical data. Even though base64 is more efficient than base16, it breaks up data across byte boundaries which then makes gzip significantly less efficient.