3 ms·
You can't, that's why they said fixed point. Time series data should probably never use floats, for the exact reason that delta-encoding is not effective.
by thedance 7y ago
You can't, that's why they said fixed point. Time series data should probably never use floats, for the exact reason that delta-encoding is not effective.
- scottlamb 7y ago"Can't" is too strong a word. Depending on what your numbers represent, I'd think you could do one or more of: * have a shorter exponent and/or mantissa to make each number shorter and the overall series more repetitive. (maybe even skip the mantissa if you only care about order of magnitude! call it the opposite of fixed-point, or just storing the logarithm of the value, whatever terminology works for you...) * skip the sign bit for a non-negative series (e.g., deltas of a non-decreasing series), * use a variable-width bitstream encoding like exp-Golomb to represent the exponent and/or mantissa, * use run-length encoding if it's repetitive (after delta-encoding), * etc. One byte per value (or less!) is surely possible in some cases. But I've never done it myself and agree fixed-point sounds much more pleasant. One of my projects represents deltas of durations in units of 90,000ths of a second, varint-encoded. I much prefer that to dealing with float seconds.
- thedance 7y agoThere's also the XOR compression scheme described by Facebook: https://www.vldb.org/pvldb/vol8/p1816-teller.pdf https://www.vldb.org/pvldb/vol8/p1816-teller.pdf