3 ms·

Sorry dumb question, but does this (averaging the averages) only not work because you can't guarantee the latency is normally distributed?

kqr 6y ago
Latencies are patently not normally distributed. I could offer you 100:1 odds on that bet. But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.
- nonameiguess 6y ago
  They're Poisson distributed, by construction. Queuing service time is actually the go to example when first introducing the Poisson distribution in an intro to statistics class. No need to make a bet. Anything that can't be negative isn't normally distributed. The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
  - mjb 6y ago
    Service latencies are typically not Poisson distributed, partially because their queuing systems are non-trivial, but mostly because there's typically a lot more going on than just queues. For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf https://drkp.net/papers/latency-socc14.pdf
    - nonameiguess 6y ago
      Thank you, that is quite interesting! As you can see, even though I have an applied math degree, I still can't just analytically infer a distribution. You need empirical results. This reminds me of the Jellyfish network topology for data centers (https://www.researchgate.net/publication/51943691_Jellyfish_Networking_Data_Centers_Randomly https://www.researchgate.net/publication/51943691_Jellyfish_...), where they found completely random route selection to be faster and more scalable than any other existing solution. I suspect, naively, that latency purely from network appliance queueing probably gives a mixture of Poissons in a straight line bus topology. But add in arbitrary routing and the much more complicated schedulers of the servers themselves, and you get no well-behaved distribution at all. It's especially interesting to compare the SLAs given by data center providers that tend to always use 99th or even 99.9th percentile worst case performance, with application metrics using arithmetic averages.
  - kqr 6y ago
    > Anything that can't be negative isn't normally distributed. Human height?
hnaccy 6y ago
The averages are weighted equally even though your traffic pattern may not be.
simiones 6y ago
The basic problem is something like this: - Minute 1: 10 requests, 5 with 10ms latency, 5 with 500ms latency; avg latency 255ms - Minute 2: 1000 requests, 900 with 10ms latency, 100 with 500ms latency; avg latency 59ms Average of averages: 157ms Average latency: 97ms You can make them arbitrarily different.
- squeaky-clean 6y ago
  But you can make this work if you also just store the number of samples per bucket, right? > (255ms + 59ms) / 2 = 157ms But > ((10 * 255ms) + (1000 * 59ms)) / 1010 = 60.94ms Which is the same answer I get as averaging all the samples without bucketing. (I can't get 97ms though)
  - klodolph 6y ago
    The example is an illustration of “if you collect the wrong data, you can’t calculate the statistics you want.” So if your response is, “can’t you just collect the correct data in the first place?” Well, that’s the hard part.
    - deleted 6y ago
      [deleted]
nerdponx 6y ago
Average is "sum(x)/length(x)". If all you have is 50 averages, there is no way to combine them to get the average of the underlying data, because you don't have the original "length(x)".

kqr 6y ago

Latencies are patently not normally distributed. I could offer you 100:1 odds on that bet. But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.

nonameiguess 6y ago
They're Poisson distributed, by construction. Queuing service time is actually the go to example when first introducing the Poisson distribution in an intro to statistics class. No need to make a bet. Anything that can't be negative isn't normally distributed. The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
- mjb 6y ago
  Service latencies are typically not Poisson distributed, partially because their queuing systems are non-trivial, but mostly because there's typically a lot more going on than just queues. For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf https://drkp.net/papers/latency-socc14.pdf
  - nonameiguess 6y ago
    Thank you, that is quite interesting! As you can see, even though I have an applied math degree, I still can't just analytically infer a distribution. You need empirical results. This reminds me of the Jellyfish network topology for data centers (https://www.researchgate.net/publication/51943691_Jellyfish_Networking_Data_Centers_Randomly https://www.researchgate.net/publication/51943691_Jellyfish_...), where they found completely random route selection to be faster and more scalable than any other existing solution. I suspect, naively, that latency purely from network appliance queueing probably gives a mixture of Poissons in a straight line bus topology. But add in arbitrary routing and the much more complicated schedulers of the servers themselves, and you get no well-behaved distribution at all. It's especially interesting to compare the SLAs given by data center providers that tend to always use 99th or even 99.9th percentile worst case performance, with application metrics using arithmetic averages.
- kqr 6y ago
  > Anything that can't be negative isn't normally distributed. Human height?

hnaccy 6y ago

The averages are weighted equally even though your traffic pattern may not be.

simiones 6y ago

The basic problem is something like this: - Minute 1: 10 requests, 5 with 10ms latency, 5 with 500ms latency; avg latency 255ms - Minute 2: 1000 requests, 900 with 10ms latency, 100 with 500ms latency; avg latency 59ms Average of averages: 157ms Average latency: 97ms You can make them arbitrarily different.

squeaky-clean 6y ago
But you can make this work if you also just store the number of samples per bucket, right? > (255ms + 59ms) / 2 = 157ms But > ((10 * 255ms) + (1000 * 59ms)) / 1010 = 60.94ms Which is the same answer I get as averaging all the samples without bucketing. (I can't get 97ms though)
- klodolph 6y ago
  The example is an illustration of “if you collect the wrong data, you can’t calculate the statistics you want.” So if your response is, “can’t you just collect the correct data in the first place?” Well, that’s the hard part.
  - deleted 6y ago
    [deleted]

nerdponx 6y ago

Average is "sum(x)/length(x)". If all you have is 50 averages, there is no way to combine them to get the average of the underlying data, because you don't have the original "length(x)".