3 ms·
Yep, absolutely. Here's a blog post outlining our general approach, which applies type-specific compression based on a column's datatype. (While m3db only sup
by mfreed 6y ago
Yep, absolutely.
Here's a blog post outlining our general approach, which applies type-specific compression based on a column's datatype. (While m3db only supports floats, TimescaleDB supports a whole range of 40+ data types.)
https://blog.timescale.com/blog/building-columnar-compression-in-a-row-oriented-database/ https://blog.timescale.com/blog/building-columnar-compressio...
For example, by default we employ:
- Gorilla for floats (+ some optimizations for faster decoding)
- Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types
- Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top)
- LZ-based array compression for all other types
And the architecture is extensible to support new/updated compression algorithms for different data types. The above are all lossless algorithms; obviously one could drop precision to achieve higher rates.
We also wrote a post describing these various approaches in a bit more detail for folks interested in learning (although a summary compared to the Facebook paper): https://blog.timescale.com/blog/time-series-compression-algorithms-explained/ https://blog.timescale.com/blog/time-series-compression-algo...
Obviously, your compression rate depends heavily on workload, but the number 15x comes from median performance that users see in the field against their real databases, not just a synthetic or single workload.