> For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. Fewer

> For example, even though the bitpacking from the post achieves fast good compression, it is unable to compress floats in general; merely 1.0 and 0.0. Fewer bits does not always mean better compression, particularly if the data has other patterns which is destroyed by the packing. With Apache ORC, I found out that if you bit-pack data to say 7bits vs leaving it as 8 bits, the 8 bits version compressed much more with Zlib than the 7 bit version. This had to do with the data getting a bit offset into the previous byte sequence, until what was a sequence of repeating bytes turned into a pattern which repeats far less often. Leaving the extra bit in place, helped Zlib dictionary encoding and huffman work much better than trying to save a bit. The final kicker was that the 24 bit sequence was faster to read than a 23 or 21 bit sequence, but purely due to the fact that the word aligned stuff can be decoded in SIMD. I'm no better at guessing what would work - "whatever works ... works, so try them.".