4 ms·

Maybe, but even this still feels incomplete. Missing from this (and ironically missing from the linked article given that it's part of the blog's title) is th

by nkurz 10y ago

Maybe, but even this still feels incomplete. Missing from this (and ironically missing from the linked article given that it's part of the blog's title) is the notion of concurrency. Whether a xx mS 99th percentile latency is good or bad depends very strongly on how many parallel requests are in flight. In a single-threaded situation where a slow response holds up all other requests, a few slow responses might be a disaster, but if it just means that an individual user has to wait half a second longer, it might be a non-issue. I agree with the blog author's point that characterizing the entire latency distribution is more valuable than looking at an individual statistic, but it's difficult for me to understand how this blog post can say avoid talking about the equally important degree of parallelism. I'm probably a broken record, but I think Little's Law is one of the most under-utilized concepts in system analysis and design. http://web.mit.edu/sgraves/www/papers/Little's%20Law-Published.pdf http://web.mit.edu/sgraves/www/papers/Little's%20Law-Publish....

infinite8s 10y ago
Actually concurrency makes it worse. Say your site has an median latency of < second but a p99 of 10 seconds for any particular request, but your site needs to make 100 requests (not unheard of with modern sites) for any particular page. This means that now almost 99% of your users will be affected by the p99 latency. This is usually a big issue because most systems are designed with median or mean latency in mind, so the p99 is usually orders of magnitude greater.
- devbug 10y ago
  Mathmatically, 1-(1-0.99)^100 = ~1.
  - dmit 10y ago
    Not quite. What you calculated is the probability that visitors will not hit the p99 case on all their requests. In other words, the chance of getting a p99 response on all 100 requests is infinitesimal. The chance of getting at least one p99 response out of a hundred is 1 - 0.99^100 ≈ 63%. https://latencytipoftheday.blogspot.com/2014/06/latencytipoftheday-most-page-loads.html https://latencytipoftheday.blogspot.com/2014/06/latencytipof...
    - forgotpwtomain 10y ago
      This is only if it's Markovian - but it almost certainly is not. If you need X requests served for a full page-load (or whichever for the primary functionality of the page), you should be totaling those request times into a single entry for your histogram, otherwise you are calculating the distribution of something potentially useless (e.g. 50% of page_is_broken_without_it.js might actually be in p99 even if that file is 1% of your requests and the generic histogram might look deceptively good).
- nkurz 10y ago
  I agree that this should be kept in mind, but I think your specific conclusion requires a lot of assumptions. The primary one is that all those requests are "independent and identically distributed". If instead your server gets laggy one minute per hour when some background update runs and trashes the file cache, the percentage of affected users would be completely different. I think the blog author is on the right track to suggest that you at least need to look at the full distribution of the latencies. I'd go farther and say that if the metric you care about is total load time for a page with 100 components, lumping together all the individual component requests is probably never going to be able to provide all the information you actually need. But as 'sp527' pointed out below, I may not have been precise enough in my use of "concurrency". From the Little's Law perspective I was emphasizing, we really care about the "number of requests in flight". But what constitutes a request depends on what we are measuring. If if each "page" consists of 100 pipelined HTTP requests, we probably need to define "in flight" as the number of simultaneous users we can support if we want to calculate the metric we actually care about.
vikiomega9 10y ago
Also missing is how timeouts can cap the latency distributions.
- neeleshs 10y ago
  Yes! This one is important. Error rates are very important to track too
matt4077 10y ago
I don't quite follow.. When the 99% perct latency is 100ms, that means 99% of requests where served within that timeframe. In terms of quality of service, isn't that all that matters? I can see that throughput is an important measurement when you're trying to plan infrastructure, and concurrency is an important concept when you want to optimize it. but if I were to guarantee you <100ms latency for the next six months, you could drop your pen and take a vacation, right?
- neeleshs 10y ago
  While 99% latency provides a good measure of QoS, it does not tell me the scale at which it is operating. I may have a service that renders some static content, say a hundred times a day. My 99th percentile latency is < 50ms. Now my observed throughput is literally non-existent. If 99% latency was < 100ms at 10k requests/s throughput, now we are talking!
sp527 10y ago
Parallelism and concurrency aren't the same thing. Normally I wouldn't nitpick something like that, but it seemed important to clarify that in the context of talking about very esoteric distributed system considerations.
- nkurz 10y ago
  In some cases the definitions are different, but aren't they usually used to describe the same concept at different levels of granularity. Which do you think is correct terminology in this context? And which is appropriate for the example in Little's paper of how long an average bottle remains in the wine cellar? And why do you feel it's esoteric? I was trying to argue that the knowing the relationship between latency, throughput, and concurrency/parallelism is essential for proper design of just about any situation where performance is a concern. Rather than esoteric, I see it as fundamental --- although I suppose that depends on what one is trying to do.
  - sp527 10y ago
    Sorry I didn't offer additional clarity. Parallelism generally implies a task that can be split into smaller chunks, which are then processed more or less simultaneously. Concurrency, by contrast, refers to a situation in which multiple executions of the same task can occur concurrently. I think it's important to get that right when talking about dist. systems simply to avoid potential confusion.

by nkurz 10y ago

infinite8s 10y ago

Actually concurrency makes it worse. Say your site has an median latency of < second but a p99 of 10 seconds for any particular request, but your site needs to make 100 requests (not unheard of with modern sites) for any particular page. This means that now almost 99% of your users will be affected by the p99 latency. This is usually a big issue because most systems are designed with median or mean latency in mind, so the p99 is usually orders of magnitude greater.

devbug 10y ago
Mathmatically, 1-(1-0.99)^100 = ~1.
- dmit 10y ago
  Not quite. What you calculated is the probability that visitors will not hit the p99 case on all their requests. In other words, the chance of getting a p99 response on all 100 requests is infinitesimal. The chance of getting at least one p99 response out of a hundred is 1 - 0.99^100 ≈ 63%. https://latencytipoftheday.blogspot.com/2014/06/latencytipoftheday-most-page-loads.html https://latencytipoftheday.blogspot.com/2014/06/latencytipof...
  - forgotpwtomain 10y ago
    This is only if it's Markovian - but it almost certainly is not. If you need X requests served for a full page-load (or whichever for the primary functionality of the page), you should be totaling those request times into a single entry for your histogram, otherwise you are calculating the distribution of something potentially useless (e.g. 50% of page_is_broken_without_it.js might actually be in p99 even if that file is 1% of your requests and the generic histogram might look deceptively good).
nkurz 10y ago
I agree that this should be kept in mind, but I think your specific conclusion requires a lot of assumptions. The primary one is that all those requests are "independent and identically distributed". If instead your server gets laggy one minute per hour when some background update runs and trashes the file cache, the percentage of affected users would be completely different. I think the blog author is on the right track to suggest that you at least need to look at the full distribution of the latencies. I'd go farther and say that if the metric you care about is total load time for a page with 100 components, lumping together all the individual component requests is probably never going to be able to provide all the information you actually need. But as 'sp527' pointed out below, I may not have been precise enough in my use of "concurrency". From the Little's Law perspective I was emphasizing, we really care about the "number of requests in flight". But what constitutes a request depends on what we are measuring. If if each "page" consists of 100 pipelined HTTP requests, we probably need to define "in flight" as the number of simultaneous users we can support if we want to calculate the metric we actually care about.

vikiomega9 10y ago

Also missing is how timeouts can cap the latency distributions.

neeleshs 10y ago
Yes! This one is important. Error rates are very important to track too

matt4077 10y ago

I don't quite follow.. When the 99% perct latency is 100ms, that means 99% of requests where served within that timeframe. In terms of quality of service, isn't that all that matters? I can see that throughput is an important measurement when you're trying to plan infrastructure, and concurrency is an important concept when you want to optimize it. but if I were to guarantee you <100ms latency for the next six months, you could drop your pen and take a vacation, right?

neeleshs 10y ago
While 99% latency provides a good measure of QoS, it does not tell me the scale at which it is operating. I may have a service that renders some static content, say a hundred times a day. My 99th percentile latency is < 50ms. Now my observed throughput is literally non-existent. If 99% latency was < 100ms at 10k requests/s throughput, now we are talking!

sp527 10y ago

Parallelism and concurrency aren't the same thing. Normally I wouldn't nitpick something like that, but it seemed important to clarify that in the context of talking about very esoteric distributed system considerations.

nkurz 10y ago
In some cases the definitions are different, but aren't they usually used to describe the same concept at different levels of granularity. Which do you think is correct terminology in this context? And which is appropriate for the example in Little's paper of how long an average bottle remains in the wine cellar? And why do you feel it's esoteric? I was trying to argue that the knowing the relationship between latency, throughput, and concurrency/parallelism is essential for proper design of just about any situation where performance is a concern. Rather than esoteric, I see it as fundamental --- although I suppose that depends on what one is trying to do.
- sp527 10y ago
  Sorry I didn't offer additional clarity. Parallelism generally implies a task that can be split into smaller chunks, which are then processed more or less simultaneously. Concurrency, by contrast, refers to a situation in which multiple executions of the same task can occur concurrently. I think it's important to get that right when talking about dist. systems simply to avoid potential confusion.