3 ms·

Those interested in TSDBs might also be interested in M3DB.io which is Apache 2 licensed, supports a modified TSZ compression implementation with 11x compressio

Those interested in TSDBs might also be interested in M3DB.io which is Apache 2 licensed, supports a modified TSZ compression implementation with 11x compression ratio, replication of data in the cluster, streaming of data between nodes on add/remove and a Kubernetes operator.

tfigment 7y ago
Thanks I learned something new. Its not general purpose and has some strict limitations. As usual this seems to skip concepts like quality/status of sensor (maybe qnan can be used as a sort of sentinel value) but is interesting newer entry in the field.
sa46 7y ago
The compression technique to achieve the 11x compression ratio is pretty clever and is detailed in the Gorilla TSDB paper. Timestamps are encoded with a delta of delta encoding scheme which allows 96% of all timestamps to compress to a single bit within a compressed block. Gorilla paper: https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf
- roskilli 7y ago
  It is definitely a huge step forward for time series data collection at scale, great props to the authors. With M3TSZ vs vanilla TSZ the major difference is that instead of XORing the value component of each datapoint it is determined if the precision of the float value is within a few significant digits and if so, it is turned into an int representation with delta-of-delta of the int value from value to value used to represent the different changes in value rather than just XORing the float bits. Also tracking how many significant digits and dampening the need to encode a change in that from value to value is also performed. We should write up the specific algorithm differences for the value component encoding. You can see the relative code here in "writeIntVal": https://github.com/m3db/m3/blob/b25e111aab06fb7cde9f7e1be8b651946c21f341/src/dbnode/encoding/m3tsz/encoder.go#L198-L229 https://github.com/m3db/m3/blob/b25e111aab06fb7cde9f7e1be8b6...

tfigment 7y ago

Thanks I learned something new. Its not general purpose and has some strict limitations. As usual this seems to skip concepts like quality/status of sensor (maybe qnan can be used as a sort of sentinel value) but is interesting newer entry in the field.

sa46 7y ago

The compression technique to achieve the 11x compression ratio is pretty clever and is detailed in the Gorilla TSDB paper. Timestamps are encoded with a delta of delta encoding scheme which allows 96% of all timestamps to compress to a single bit within a compressed block. Gorilla paper: https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf

roskilli 7y ago
It is definitely a huge step forward for time series data collection at scale, great props to the authors. With M3TSZ vs vanilla TSZ the major difference is that instead of XORing the value component of each datapoint it is determined if the precision of the float value is within a few significant digits and if so, it is turned into an int representation with delta-of-delta of the int value from value to value used to represent the different changes in value rather than just XORing the float bits. Also tracking how many significant digits and dampening the need to encode a change in that from value to value is also performed. We should write up the specific algorithm differences for the value component encoding. You can see the relative code here in "writeIntVal": https://github.com/m3db/m3/blob/b25e111aab06fb7cde9f7e1be8b651946c21f341/src/dbnode/encoding/m3tsz/encoder.go#L198-L229 https://github.com/m3db/m3/blob/b25e111aab06fb7cde9f7e1be8b6...