3 ms·
1.2 PB saved using gorilla compression! Didn’t realize how sparse this data was.
by DSingularity 6y ago
1.2 PB saved using gorilla compression! Didn’t realize how sparse this data was.
- cbsmith 6y agoNot necessarily sparse. Just lacking in entropy. You can compress an arithmetic series down to a handful of bytes...
- hodgesrm 6y agoCompression for correlated measurements can be spectacular. ClickHouse can often get 99.9% reduction in data size with the right combination of codecs (delta, double-delta, gorilla, etc.) and compression (e.g., ZSTD). Sort order within tables is also important.
- akulkarni 6y agoCompression isn't just about sparse-ness, but optimizations one can make based on the dataset. E.g., Gorilla uses delta-delta-encoding for time-stamps, and a novel XOR-based compression scheme for floats. For more, here is a primer on Gorilla and other leading time-series algorithms: https://blog.timescale.com/blog/time-series-compression-algorithms-explained/ https://blog.timescale.com/blog/time-series-compression-algo...
- valyala 6y agoBTW, there are algorithms with better compression rate for time series data than Gorilla - see https://medium.com/faun/victoriametrics-achieving-better-compression-for-time-series-data-than-gorilla-317bc1f95932 https://medium.com/faun/victoriametrics-achieving-better-com...