3 ms·

> 15x native compression That is a rather remarkable compression ratio. Do you have a link to technical details? The lightweight compression outlined in the Go

by sa46 6y ago

> 15x native compression That is a rather remarkable compression ratio. Do you have a link to technical details? The lightweight compression outlined in the Gorilla TSDB and implemented in Uber's m3db is 11x which I thought was best in class: https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf

mfreed 6y ago
Yep, absolutely. Here's a blog post outlining our general approach, which applies type-specific compression based on a column's datatype. (While m3db only supports floats, TimescaleDB supports a whole range of 40+ data types.) https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented-database/ https://blog.timescale.com/blog/building-columnar-compressio... For example, by default we employ: - Gorilla for floats (+ some optimizations for faster decoding) - Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types - Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top) - LZ-based array compression for all other types And the architecture is extensible to support new/updated compression algorithms for different data types. The above are all lossless algorithms; obviously one could drop precision to achieve higher rates. We also wrote a post describing these various approaches in a bit more detail for folks interested in learning (although a summary compared to the Facebook paper): https://blog.timescale.com/blog/time-series-compression-algorithms-explained/ https://blog.timescale.com/blog/time-series-compression-algo... Obviously, your compression rate depends heavily on workload, but the number 15x comes from median performance that users see in the field against their real databases, not just a synthetic or single workload.

by sa46 6y ago

mfreed 6y ago

Yep, absolutely. Here's a blog post outlining our general approach, which applies type-specific compression based on a column's datatype. (While m3db only supports floats, TimescaleDB supports a whole range of 40+ data types.) https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented-database/ https://blog.timescale.com/blog/building-columnar-compressio... For example, by default we employ: - Gorilla for floats (+ some optimizations for faster decoding) - Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types - Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top) - LZ-based array compression for all other types And the architecture is extensible to support new/updated compression algorithms for different data types. The above are all lossless algorithms; obviously one could drop precision to achieve higher rates. We also wrote a post describing these various approaches in a bit more detail for folks interested in learning (although a summary compared to the Facebook paper): https://blog.timescale.com/blog/time-series-compression-algorithms-explained/ https://blog.timescale.com/blog/time-series-compression-algo... Obviously, your compression rate depends heavily on workload, but the number 15x comes from median performance that users see in the field against their real databases, not just a synthetic or single workload.