4 ms·
Using the median in lieu of the average isn't always a good idea either. A service could completely fail to respond almost 50% of the time and you'd still get a
by contravariant 7y ago
Using the median in lieu of the average isn't always a good idea either. A service could completely fail to respond almost 50% of the time and you'd still get a low median. Same holds for perc95, but to a lesser extent.
The main problem is that people try to summarize their data too early. What you want is a measure for how good or bad a single datum is and only then can you summarize the end result. And usually averages aren't a bad choice at that stage.
Averages have some particularly nice properties when dealing with dependent variables, sums of variables, and when you want to minimize the distance between your estimate and the actual value. However to take advantage of that your measure actually needs to make sense. For companies the holy grail is if you can directly express how much money you make / lose because of that single datum, but failing that you'll need to find something that's at least somewhat proportional to it.
- alexhutcheson 7y agoMy normal approach is to measure and set up separate alerts for "error rate" and possibly "timeout rate". You definitely want to know about those, but "mean latency" mixes those metrics with the latency metrics for successful requests, which makes it less sensitive to changes in either one. In general I agree that averages aren't always bad. One additional advantage is that it's often possible to generate robust confidence intervals for averages, but it's often not valid to generate CIs for medians/percentiles without introducing other probably flawed assumptions.
- contravariant 7y agoError rate and timeout rate are good examples of an average that measures the thing you're actually interested in. The whole point is that trying to average the latency and only then try to figure out what it means is backwards.