4 ms·
We can do better than percentile latencies
- mkesper 8y agoThe article states that almost everyone is doing percentile latencies wrong by averaging on agent level (thus creating nonsensical data) and proposes using the percentage of requests that are over the threshold instead, a metric that can be averaged properly. He additionally suggests to always use actionable dashboards catered to its users (dev/ops/manager).
- amarant 8y agoI really like the idea of displaying what percentage is over a certain threshold. at my work, we kinda sorta simulate this by having separate alarms for many percentiles (with increasing thresholds). The approach suggested by the article seems to be quite obviously better tbh.
- digikata 8y agoFor diagnosing, I like building up a cumulative distribution function (CDF) plot. If you're collecting data either for percentiles or thresholds you likely have the data already. If you're setting thresholds, it's a useful plot to judge how likely a given threshold might trigger an alarm.
- latch 8y agoWorth watching is How NOT to Measure Latency: https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU The speaker Gil Tene is also the author of the HdrHistogram which addresses this articles point: https://hdrhistogram.github.io/HdrHistogram/ https://hdrhistogram.github.io/HdrHistogram/
- romed 8y agoI don't get hdrhistogram at all. It's really just histogram with a shitload of logarithmic buckets.
- sbanach 8y agoThe compression format is really smart, there's a neat trick to make the logarithm calculation fast, and there's a concurrent thread handoff mechanism so you can swap out a histogram without disturbing the thread you're measuring (though last two probably only in the java version). Those three make it super useful for very low impact performance measurements.
- frankmcsherry 8y agoAbsolutely worth watching. Cannot un-watch.
- WorkLifeBalance 8y agoStop trying to re-invent statistics. Use a box and whisker plot of latency. You quickly get to see the mean, the quartiles, and all the outliers and you get it in a format which is familiar and easy to understand. You can even plot box and whisker plots next to each other for quick meaningful comparisons between different things.
- scott_s 8y agoI've recently grown to like violin plots for latency (https://en.wikipedia.org/wiki/Violin_plot https://en.wikipedia.org/wiki/Violin_plot). I've also added 99%ile tick marks, which with the already present median mark, gives a relatively full picture of latency that is easily digestible.
- vlovich123 8y agoMathematics is not something handed down by the gods. It's possible to encounter not just completely new problems, but also limitations to existing methods for solving a known problem. In this particular case the challenge is aggregating statistics from a very large fleet & having automated alarms. Visualization tools don't help with any of that. More specifically, the reporting tools out there apparently have a very common & persistent flaw of reporting an average of percentiles across agents which is a statistically meaningless metric. It makes no difference how you visualize it - the data is bunk. This article flips it so that agents simply report how many requests they got & how many exceeded the required threshold. This lets them report the percentage of users having a worse experience than the desired SLA. You can also build reliable tools on top of this metric. It's not a universal solution but it's a neat trick to maintain the performance properties of not needing to pull full logs from all agents & still have a meaningful representation of the latency of your users.
- afpx 8y agoReservoir sampling with outliers?
- krona 8y agoCorrect.
- asplake 8y agoMean excess delay?
- imtringued 8y agoJust plot the latency distribution in a histogram [1] and be done with this boring topic. The problem you're complaining about is that aggregating everything into a single number doesn't give you the information you want to know. It's impossible by design. If you want a different type of information you will also need a different type of diagram or statistic. [1] https://rethinkdb.com/assets/images/docs/performance-report/w-a-reads-latency.png https://rethinkdb.com/assets/images/docs/performance-report/...
- jcrites 8y agoI think you may be overlooking the challenges that the author is trying to tackle with their proposal. First, the author is looking for a characteristic that can be monitored automatically - for example, alarm if P99 latency is over 2 s. Visualizations while useful don’t help with that. Second, the author is looking for a solution that can run in soft real time, so that it can be used for system monitoring. Third, they’re looking for a solution that does not have to aggregate the full raw data set from across the fleet. It is implied that they are working with reasonably large fleets such that full aggregation is impractical; or maybe just too costly or too slow. If you were able to aggregate the full raw data set in real time and compute the Nth percentile, then that statistic would meet the author’s needs. Their point is that actually computing the Nth percentile is expensive and not commonly done in real-time monitoring (hence the statistic is usually an average of host-level Nth percentile). The challenge they’ve proposed is to define a statistic that is more useful for alarming while still avoiding the need to aggregate the entire raw data set. I thought this was a thoughtful article with a clever suggestion. “Percent of requests over threshold” meets these criteria. One criticism of this approach however is that the threshold needs to be known ahead of time, prior to aggregation.
- eutropia 8y agoThe threshold could be computed batchwise as you aggregate the stats out of band, so it would at least be dynamic, if not instantaneous.
- rixed 8y agoBut an histogram can be considered a generalised version of what the article propose so the OP is indeed right that histograms are already a well known solution to that boring problem.
- jordanthoms 8y agoThe approach here sounds a bit like the ApDex metric New Relic has been doing for years? Is there something different I'm missing?
- henridf 8y agoI'm not sure which tools the author has tried, but the Prometheus monitoring system supports both histograms and quantiles. There's a good discussion of the respective merits of each at https://prometheus.io/docs/practices/histograms/#quantiles https://prometheus.io/docs/practices/histograms/#quantiles
- deathanatos 8y agoHistograms require you to configure buckets into which your samples are allocated; to allocate the buckets appropriately, you need to know what your expected values are — that is, to measure latency, you need to know your latency. While this can work (I think most of us have a clear idea, or can obtain an idea of what our typical latencies is, and configure buckets around that) it is inelegant. I feel like I would rather have X=percentile, Y=latency, but such a bucketing gives you X=latency, Y=request count. Still useful, but only as informative as you are good at choosing buckets. (There is the histogram_quantile function, but I am unclear that its assumption of linear distribution within buckets really makes much sense, since most things would be long-tail distributions, and thus I would think that once you get past the main "hump" of typical latencies, most samples would cluster towards the lower end of any particular bucket.) I am not clear on how Summaries actually work; they appear to report count and sum of the thing they're monitoring; that is, if one were to use them for latencies (and the docs do indeed suggest this), it would report a value like "3" and "2000ms", indicating that 3 requests took a total of 2000ms together; how is one supposed to derive a latency histogram/profile from that? Prometheus's fatal flaw here, IMO, is that it requires sampling of metrics. That is, things like CPU, which are essentially a continuous function that you're sampling over time. But its collection method/format doesn't seem to really work that well for when you have an event-based metric, such as request latency, which only happens at discrete points. (If no requests are being served, what is the latency? It makes no sense to ask, unlike CPU usage or RAM usage.) To me, ideally, you want to collect up all the samples in a central location and then compute percentiles. Anything else seems to run afoul of the very "doing percentiles on the agents, then 'averaging' percentiles at the monitoring system" critique pointed out in the video posted in this sibling comment: https://news.ycombinator.com/item?id=18194507 https://news.ycombinator.com/item?id=18194507
- vvern 8y agoI agree with the premise but it seems that there are more solutions out there. As other commenters noted, you can collect histograms or hdrhistograms. Those have the problem of needing to be precofiguring and of not being able to be merged unless they are configured the same way. Instead you can use the t-digest (https://github.com/tdunning/t-digest https://github.com/tdunning/t-digest), a very cool online quantile estimation data structure from Ted Dunning (which he has recently improved with the Merging approach). There are a number of implementations out there. It is not unreasonable to serialize them and merge them. Unfortunately there’s no easy way to set this up in Prometheus but making that easy could be a fun project
- deleted 8y ago[deleted]
- deleted 8y ago[deleted]