5 ms·
Percentiles become painful when trying to aggregate them. Suppose you record the p99 latency of some service, but you collect these metrics at the rack or data
by Xcelerate 3y ago
Percentiles become painful when trying to aggregate them. Suppose you record the p99 latency of some service, but you collect these metrics at the rack or data center level. Now you ask, what is the overall p99 latency of the service? Not an easy question to answer. Especially if you automatically subsample older time series data in order to store more of it (I've seen people trying to perform a weighted average of subsampled percentile metrics—it turns into a mess).
We need an efficient way to compactly represent the entire distribution of a metric over time so arbitrary aggregations can be performed accurately. There is some research on this topic, but nothing really production-ready that I'm aware of.
- ickyforce 3y ago> but nothing really production-ready that I'm aware of Here are some: http://hdrhistogram.org/ http://hdrhistogram.org/ https://prometheus.io/docs/practices/histograms/ https://prometheus.io/docs/practices/histograms/ (Victoria Metrics is also worth looking at, IIRC they store quantiles differently) https://prestodb.io/docs/current/functions/qdigest.html https://prestodb.io/docs/current/functions/qdigest.html
- wrigby 3y agoSounds like you’re looking for T-digests[1] - most production systems I’ve worked on that do this are using them under the hood. 1: https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest
- brimtown 3y agoYou might be interested in https://www.datadoghq.com/blog/engineering/computing-accurate-percentiles-with-ddsketch/ https://www.datadoghq.com/blog/engineering/computing-accurat... Disclaimer: I work there/built the TypeScript implementation of the library