4 ms·

With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.

by Rapzid 7y ago

sokoloff 7y ago
The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.
- ses1984 7y ago
  It has nothing to do with your experience being 99% good enough and more to do with the size of the revenue opportunity that could be lost messing up 1% of requests.
- kevan 7y ago
  As an individual you don't care as long as you aren't in that 1%, but as a customer-obsessed service owner I absolutely care if a bad experience is happening 100k times per day. If you're operating at internet scale you need to look at both percentiles and absolute numbers to assess customer impact. Also keep in mind how percentiles compound when you have more than one service involved in serving a customer request. For example, let's say it takes 5 internal requests to serve an external customer request and each of those services measures latency SLAs at the 99th percentile. The customer request may only finish inside the SLA 95% of the time (99%^5)
- scottlamb 7y ago
  > The latency to human user is what matters. Agreed, which is why the most surprising part of this article for me was this phrase: "In our main data center serving non-China traffic". The transit latency to the one main datacenter in the (non-China) world is a lot more significant than the server time here. (Over 50 ms just to cross the US; compare with their server-side latency numbers of 95%ile < 5 ms, 99%ile < 50 ms.) If they're serious about latency, their deployment is holding them back much more than their choice of programming language or algorithm. Or maybe the client is not the user's phone, but some other service running within the same datacenter?
sethammons 7y ago
I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.
- lclarkmichalek 7y ago
  Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.
  - sethammons 7y ago
    And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.
  - hobofan 7y ago
    > and almost certainly had timeouts, the "max" would also likely be at the timeout You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.
    - spookthesunset 7y ago
      Sure but now your "100th percentile" would be nothing but the timeout. Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else. .... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.

With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.

by Rapzid 7y ago

With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.

The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.

ses1984 7y ago
It has nothing to do with your experience being 99% good enough and more to do with the size of the revenue opportunity that could be lost messing up 1% of requests.
kevan 7y ago
As an individual you don't care as long as you aren't in that 1%, but as a customer-obsessed service owner I absolutely care if a bad experience is happening 100k times per day. If you're operating at internet scale you need to look at both percentiles and absolute numbers to assess customer impact. Also keep in mind how percentiles compound when you have more than one service involved in serving a customer request. For example, let's say it takes 5 internal requests to serve an external customer request and each of those services measures latency SLAs at the 99th percentile. The customer request may only finish inside the SLA 95% of the time (99%^5)
scottlamb 7y ago
> The latency to human user is what matters. Agreed, which is why the most surprising part of this article for me was this phrase: "In our main data center serving non-China traffic". The transit latency to the one main datacenter in the (non-China) world is a lot more significant than the server time here. (Over 50 ms just to cross the US; compare with their server-side latency numbers of 95%ile < 5 ms, 99%ile < 50 ms.) If they're serious about latency, their deployment is holding them back much more than their choice of programming language or algorithm. Or maybe the client is not the user's phone, but some other service running within the same datacenter?

sethammons 7y ago

I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.

lclarkmichalek 7y ago
Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.
- sethammons 7y ago
  And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.
- hobofan 7y ago
  > and almost certainly had timeouts, the "max" would also likely be at the timeout You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.
  - spookthesunset 7y ago
    Sure but now your "100th percentile" would be nothing but the timeout. Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else. .... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.