4 ms·

How does this compare to regular dictionary compression?

• duskwuff 3y ago
  Dictionary compression isn't really applicable to floating-point data, because it isn't made up of commonly repeated "words" like text is.
  • GuB-42 3y ago
    Dictionary compression, combined with a form of entropy coding can work. As shown in the article the exponents often repeat themselves, and the low order bits of the mantissa are often zero. Advanced context modeling compression algorithms can work even better than the scheme presented in the article. These are like mini-AIs, and they can recognize patterns more complex than just a string of unchanged bits, and with the appropriate entropy coder, give excellent compression. Or course the processing cost is huge compare to a few bit manipulations, and wouldn't make much sense unless you need extreme compression.
    • gliptic 3y ago
      But context modelling isn't dictionary compression. Parent is correct that dictionary compression itself (e.g. LZ4) isn't very useful for floating point.