4 ms·
I think you may be overlooking the challenges that the author is trying to tackle with their proposal. First, the author is looking for a characteristic that c
by jcrites 8y ago
I think you may be overlooking the challenges that the author is trying to tackle with their proposal.
First, the author is looking for a characteristic that can be monitored automatically - for example, alarm if P99 latency is over 2 s. Visualizations while useful don’t help with that.
Second, the author is looking for a solution that can run in soft real time, so that it can be used for system monitoring.
Third, they’re looking for a solution that does not have to aggregate the full raw data set from across the fleet. It is implied that they are working with reasonably large fleets such that full aggregation is impractical; or maybe just too costly or too slow.
If you were able to aggregate the full raw data set in real time and compute the Nth percentile, then that statistic would meet the author’s needs. Their point is that actually computing the Nth percentile is expensive and not commonly done in real-time monitoring (hence the statistic is usually an average of host-level Nth percentile).
The challenge they’ve proposed is to define a statistic that is more useful for alarming while still avoiding the need to aggregate the entire raw data set.
I thought this was a thoughtful article with a clever suggestion. “Percent of requests over threshold” meets these criteria. One criticism of this approach however is that the threshold needs to be known ahead of time, prior to aggregation.
- eutropia 8y agoThe threshold could be computed batchwise as you aggregate the stats out of band, so it would at least be dynamic, if not instantaneous.
- rixed 8y agoBut an histogram can be considered a generalised version of what the article propose so the OP is indeed right that histograms are already a well known solution to that boring problem.
- londons_explore 8y agoThere is no reason to aggregate all raw data across the fleet. Use bucketed histograms. Have a few hundred buckets for times, exponentially distributed. Store counts of numbers of events in each bucket. A few hundred integers per server isn't hard to store and aggregate. Prometheus does this out of the box. Now you can recreate any of the charts you want!
- heinrichhartman 8y ago> Second, the author is looking for a solution that can run in soft real time, so that it can be used for system monitoring. Histograms can absolutely be used for alerting. We have done this at Circonus for ages: https://www.circonus.com/features/analytics/ https://www.circonus.com/features/analytics/ I wrote up the case of latency monitoring 6 weeks ago here: https://www.circonus.com/2018/08/30/latency-slos-done-right/ https://www.circonus.com/2018/08/30/latency-slos-done-right/
- mping 8y agoA probabilistic data structure comes to mind, such as Q-Digest. I wonder why it isn't used more for these kinds of problems