2 ms·
That's great for unstructured data, like data with high cardinality on the dimensions. But for most real world metrics outside analytics, this isn't necessary a
by discordianfish 11y ago
That's great for unstructured data, like data with high cardinality on the dimensions. But for most real world metrics outside analytics, this isn't necessary and a data model like prometheus makes more sense.
If I did the math right, even after compression elasticsearch uses 22 bytes per data point (23m points / 508 megabyte) where prometheus uses about 2.5-3.5 bytes per data point.
Disclosure: Prometheus contributor here
- existencebox 11y agoGiven your background, I'm going to take this opportunity to ask some very noobish questions. (I will be doing my due diligence to read up on Prometheus, I likely should have done so prior, but these are just some off the top of my head that I'm not sure I'd gain a good intuition for until late in the learning process) - How is the bulk of this additional compression derived? Is it explicitly the existence of a data model that lets you use more aggressive/intelligent compression strategies? - Does this come at a cost? (increased CPU overhead, latency at read time, something like that.)
- chaotic-good 11y agoIf you can predict next value in the sequence you can achieve better compression (xor predicted value and actual, not previous and next - http://users.ices.utexas.edu/~burtscher/papers/dcc06.pdf http://users.ices.utexas.edu/~burtscher/papers/dcc06.pdf), but I don't think that Prometheus developers are using this. And BTW, I don't believe in 2.5-3.5 bytes per data-point somewhere outside synthetic benchmarks.