4 ms·

95% latency seems way more useful as a metric than max() would be. It’s better to spend most of your time improving the experience of most users rather than deb

by chis 4y ago

95% latency seems way more useful as a metric than max() would be. It’s better to spend most of your time improving the experience of most users rather than debugging ultra-rare issues that rarely affect users.

Gh0stRAT 4y ago
It depends how many requests (both frontend and backend) each user triggers and the odds of a big latency spike on at least one of them. For single requests, targeting 95% would probably be just fine back in the days of static sites served by a single Apache instance, but that doesn't describe many modern systems. Nowadays, a single HTTP request to a load balancer can trigger a flurry of blocking queries to fulfil it. (eg a distributed ElasticSearch query that can't combine the results from each node and return until EVERY node has responded) then your worst-case performance will quickly begin to dominate.
jcelerier 4y ago
Up to the day one of these ultra-rare issues falls on the journalist writing a review in a specialist magazine that will make or break your product
winrid 4y ago
Until max() is your largest, top paying customers.
- Akronymus 4y ago
  Or you use microservices where a lot of requests are sent between the services. Which results in max being more and more likely for each user made request. https://youtu.be/_Zoa3xkzgFk https://youtu.be/_Zoa3xkzgFk
dikei 4y ago
95% is really not a lot, it's 5 slow requests out of 100. It takes ~300 requests to load the homepage of Amazon. You need a lot more "9"
- bawolff 4y ago
  But presumably those slow requests arent evenly distributed. say you have some server generated page + static css, js, images. One would presume the static resources are less likely to be in the 95th percentile (although i guess it depends on architecture a lot)
lifthrasiir 4y ago
The effective latency users see is the maximum of all latencies observed during the session. If your session requires 10 observable requests 1 - 0.95^10 = 40% of users will experience >95%ile latency at some point. There are many considerations (I specifically said "observable" because there are tons of ways to hide latencies to users, and the number of observable requests can greatly vary) but it is never the case that the raw per-request latency equals to the user-visible latency.
- kqr 4y ago
  Also worth mentioning the slightly fuzzier problem that the effects on user satisfaction from latency is not linear. A user that experiences four 125 ms page loads followed by one 500 ms page load is less happy than the user experiencing five 200 ms page loads. So the user experience of your system is strongly driven by the tails of the latency distribution. Practically all of the useful information is beyond 95 %, not below it.
kqr 4y ago
Upvoting because this is one of the most common misconceptions around performance I encounter. You have received several good responses already, and I'll add just one more: the latency distribution of many software systems is in practise closer to a power law than a normal distribution. This means the "outliers" dominate almost all ways to look at the distribution, which means you will find the very high percentiles (including max) to be more informative of what going on (on a per-request basis) than the lower percentiles. Another feature of power laws is that it's hard to estimate their shape based on the central values -- one of the more effective methods is to MLE fit a tail exponent to the tail (i.e. what's beyond the 95th percentile) and then extrapolate from that also to central values when necessary. This yields a less biased estimate of the mean, for example.
bitcharmer 4y ago
This heavily depends on the SLA, the use case or the industry. Working on ultra low latency systems for algorithmic trading I can tell you we actually care and hunt down maxes more than the 95th percentile. They tend to be much bigher outliers in HFT and the transactions you typically loose money on.
say_it_as_it_is 4y ago
"99% of users experience ~99.995%’ile response times, " Most of the time, you will experience adequate performance from our service. However, from time to time, you will also experience anomalies. We won't be fixing the sources of anomalous behavior because they're rare.
vlovich123 4y ago
Aside from everything else already mentioned (which is true and accurate and discussed in the article), it’s important to also remember that tails skew your percentiles quite heavily. So fixing your max could easily drop your 95%-ile by a good margin on its own. So if that doesn’t improve the experience of most users, why do you choose to believe that the 95% number represents the experience of most people? In fact, isn’t it possible and likely your entire 95% number is a virtual artifact of your common case vs your max as opposed to it being its own mode with its own root cause? In practice, what you really want to focus on is solving the modes of your system but identifying them can be quite hard (aside from the obvious one of max). %-iles are a useful tool but don’t mistake your tool for the truth or accidentally mistake the measure for the target.

by chis 4y ago

Gh0stRAT 4y ago

It depends how many requests (both frontend and backend) each user triggers and the odds of a big latency spike on at least one of them. For single requests, targeting 95% would probably be just fine back in the days of static sites served by a single Apache instance, but that doesn't describe many modern systems. Nowadays, a single HTTP request to a load balancer can trigger a flurry of blocking queries to fulfil it. (eg a distributed ElasticSearch query that can't combine the results from each node and return until EVERY node has responded) then your worst-case performance will quickly begin to dominate.

jcelerier 4y ago

Up to the day one of these ultra-rare issues falls on the journalist writing a review in a specialist magazine that will make or break your product

winrid 4y ago

Until max() is your largest, top paying customers.

Akronymus 4y ago
Or you use microservices where a lot of requests are sent between the services. Which results in max being more and more likely for each user made request. https://youtu.be/_Zoa3xkzgFk https://youtu.be/_Zoa3xkzgFk

dikei 4y ago

95% is really not a lot, it's 5 slow requests out of 100. It takes ~300 requests to load the homepage of Amazon. You need a lot more "9"

bawolff 4y ago
But presumably those slow requests arent evenly distributed. say you have some server generated page + static css, js, images. One would presume the static resources are less likely to be in the 95th percentile (although i guess it depends on architecture a lot)

lifthrasiir 4y ago

The effective latency users see is the maximum of all latencies observed during the session. If your session requires 10 observable requests 1 - 0.95^10 = 40% of users will experience >95%ile latency at some point. There are many considerations (I specifically said "observable" because there are tons of ways to hide latencies to users, and the number of observable requests can greatly vary) but it is never the case that the raw per-request latency equals to the user-visible latency.

kqr 4y ago
Also worth mentioning the slightly fuzzier problem that the effects on user satisfaction from latency is not linear. A user that experiences four 125 ms page loads followed by one 500 ms page load is less happy than the user experiencing five 200 ms page loads. So the user experience of your system is strongly driven by the tails of the latency distribution. Practically all of the useful information is beyond 95 %, not below it.

kqr 4y ago

Upvoting because this is one of the most common misconceptions around performance I encounter. You have received several good responses already, and I'll add just one more: the latency distribution of many software systems is in practise closer to a power law than a normal distribution. This means the "outliers" dominate almost all ways to look at the distribution, which means you will find the very high percentiles (including max) to be more informative of what going on (on a per-request basis) than the lower percentiles. Another feature of power laws is that it's hard to estimate their shape based on the central values -- one of the more effective methods is to MLE fit a tail exponent to the tail (i.e. what's beyond the 95th percentile) and then extrapolate from that also to central values when necessary. This yields a less biased estimate of the mean, for example.

bitcharmer 4y ago

This heavily depends on the SLA, the use case or the industry. Working on ultra low latency systems for algorithmic trading I can tell you we actually care and hunt down maxes more than the 95th percentile. They tend to be much bigher outliers in HFT and the transactions you typically loose money on.

say_it_as_it_is 4y ago

"99% of users experience ~99.995%’ile response times, " Most of the time, you will experience adequate performance from our service. However, from time to time, you will also experience anomalies. We won't be fixing the sources of anomalous behavior because they're rare.

vlovich123 4y ago

Aside from everything else already mentioned (which is true and accurate and discussed in the article), it’s important to also remember that tails skew your percentiles quite heavily. So fixing your max could easily drop your 95%-ile by a good margin on its own. So if that doesn’t improve the experience of most users, why do you choose to believe that the 95% number represents the experience of most people? In fact, isn’t it possible and likely your entire 95% number is a virtual artifact of your common case vs your max as opposed to it being its own mode with its own root cause? In practice, what you really want to focus on is solving the modes of your system but identifying them can be quite hard (aside from the obvious one of max). %-iles are a useful tool but don’t mistake your tool for the truth or accidentally mistake the measure for the target.