4 ms·
It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we’ve seen compression reduce each po
by chaotic-good 11y ago
It depends greatly on the shape of your data, but with regularly spaced timestamps at second level precision and float64s, we’ve seen compression reduce each point down to around 2.2 bytes per point.
AFAIK you're using float64 compression scheme from "Gorilla" paper. 2.2 bytes per data point is possible with it but only on data that doesn't utilize full double precision (example: small integers converted to float64). You should compare compression algorithm used by TSM storage engine with zlib or any other general purpose compression algorithm, otherwise this number will be meaningless.
- Dylan16807 11y agoHonestly if you have a series of increasing unix timestamps you should be able to do much better than 2.2 bytes each.
- chaotic-good 11y agoEven simple delta+RLE can squeeze increasing unix timestamps up to few bytes. I'm talking about floating point data compression. Even the specialized state of the art algorithms can't guarantee you 2.2 bytes per float in general.