5 ms·

The article is useful because it outlines many different ways in which to monitor the performance of a system, many of which are better than just looking at the

The article is useful because it outlines many different ways in which to monitor the performance of a system, many of which are better than just looking at the mean and P99. However, the main thesis that "an average of a percentile is meaningless" is just plain wrong. If the distribution is fixed, then averaging different P99 measurements will give you the best possible estimate of what P99 is in the population (as opposed to your sample.) If the distribution is moving (you're making performance improvements or your user base is growing) then a moving average of a percentile will move with it.

swk82 11y ago
You are right when looking at the true P99 of the whole distribution. Then the P99 will always have the same value. The problem arises when estimating a distribution from incomplete data. E.g. estimating the P99 from 200 data points each will give you a large variation. In this case averaging will be wrong. This gets worse if you experience bunching. E.g. a server will have a high ping on several packets when the load is high; after that it will have a low ping for thousands of packets when the load is low. Here, it just doesn't work. Instead, you can get a better result by adding the (estimated) histograms and resampling the quantiles from the combined distribution function. Here I've written how that works with a live demonstration: http://www.siegfried-kettlitz.de/blog/posts/2015/11/28/linlog_plot_quantiles/ http://www.siegfried-kettlitz.de/blog/posts/2015/11/28/linlo...
- stdbrouw 11y ago
  Even P99 estimates averaged from many tiny samples are at most inaccurate, they won't be biased. The problem you talk about is one of granularity: you don't want to know how bad things are in general, you want to know how bad things are when they are actually bad. That's perfectly legitimate, but it's a different metric altogether that just happens to also be based off of the 99th percentile. It depends on your definition of "the distribution". This probably sounds like nitpicking, but people should know what they're talking about when they call out others for being wrong.
  - swk82 11y ago
    > Even P99 estimates averaged from many tiny samples are at most inaccurate, they won't be biased. Are you sure about this?
    - stdbrouw 11y ago
      Sure, simulate it if you don't believe me. x <- rnorm(1000, mean=50, sd=10) quantile(x, 0.99) mean(sapply(1:20, function(i) { i <- i * 50 quantile(x[i-49:i], 0.99) }))
      - swk82 11y ago
        Matlab / GNU/Octave code: rand('seed',123); outer_rounds = 100; inner_rounds = 100; items = 500; q99_mean_vec = []; q99_complete_vec = []; for ior = 1:outer_rounds; q99_vec = []; x = rand( 1, rounds*items ); for iir = 1:inner_rounds q99_vec(iir) = quantile( x( (iir-1)*items+(1:items) )', 0.99 ); end q99_complete_vec(ior) = quantile( x', 0.99 ); q99_mean_vec(ior) = mean( q99_vec' ); end hist( [ q99_complete_vec; q99_mean_vec ]', [0.988:0.0001:0.991] ); xlim( [0.988, 0.991] ); legend( 'from complete data', 'from means' ); This looks biased to me even though the distribution is uniform. I did not expect that. Maybe the quantile function is broken. EDIT: I've installed R and run your code. You use a very low number of samples in the mean, so I changed it a bit. 1. I use the uniform distribution, so we know the true value of the 99th percentile is 0.99. 2. I've increased the number of samples: x <- runif(100000) quantile(x, 0.99) mean(sapply(1:2000, function(i) { i <- i * 50 quantile(x[i-49:i], 0.99) })) When run multiple times, I've found that the mean is always lower than the quantile over the full sample.
        stdbrouw 11y ago
        Actually, doh, you're right, my bad, turns out sample quantiles are only asymptotically unbiased, and a better estimator would take some sort of weighted average of P99 and P99.5 with the weights depending on the sample size. (You still don't need the full population and you don't have to merge histograms, though.)

swk82 11y ago

You are right when looking at the true P99 of the whole distribution. Then the P99 will always have the same value. The problem arises when estimating a distribution from incomplete data. E.g. estimating the P99 from 200 data points each will give you a large variation. In this case averaging will be wrong. This gets worse if you experience bunching. E.g. a server will have a high ping on several packets when the load is high; after that it will have a low ping for thousands of packets when the load is low. Here, it just doesn't work. Instead, you can get a better result by adding the (estimated) histograms and resampling the quantiles from the combined distribution function. Here I've written how that works with a live demonstration: http://www.siegfried-kettlitz.de/blog/posts/2015/11/28/linlog_plot_quantiles/ http://www.siegfried-kettlitz.de/blog/posts/2015/11/28/linlo...

stdbrouw 11y ago
Even P99 estimates averaged from many tiny samples are at most inaccurate, they won't be biased. The problem you talk about is one of granularity: you don't want to know how bad things are in general, you want to know how bad things are when they are actually bad. That's perfectly legitimate, but it's a different metric altogether that just happens to also be based off of the 99th percentile. It depends on your definition of "the distribution". This probably sounds like nitpicking, but people should know what they're talking about when they call out others for being wrong.
- swk82 11y ago
  > Even P99 estimates averaged from many tiny samples are at most inaccurate, they won't be biased. Are you sure about this?
  - stdbrouw 11y ago
    Sure, simulate it if you don't believe me. x <- rnorm(1000, mean=50, sd=10) quantile(x, 0.99) mean(sapply(1:20, function(i) { i <- i * 50 quantile(x[i-49:i], 0.99) }))
    - swk82 11y ago
      Matlab / GNU/Octave code: rand('seed',123); outer_rounds = 100; inner_rounds = 100; items = 500; q99_mean_vec = []; q99_complete_vec = []; for ior = 1:outer_rounds; q99_vec = []; x = rand( 1, rounds*items ); for iir = 1:inner_rounds q99_vec(iir) = quantile( x( (iir-1)*items+(1:items) )', 0.99 ); end q99_complete_vec(ior) = quantile( x', 0.99 ); q99_mean_vec(ior) = mean( q99_vec' ); end hist( [ q99_complete_vec; q99_mean_vec ]', [0.988:0.0001:0.991] ); xlim( [0.988, 0.991] ); legend( 'from complete data', 'from means' ); This looks biased to me even though the distribution is uniform. I did not expect that. Maybe the quantile function is broken. EDIT: I've installed R and run your code. You use a very low number of samples in the mean, so I changed it a bit. 1. I use the uniform distribution, so we know the true value of the 99th percentile is 0.99. 2. I've increased the number of samples: x <- runif(100000) quantile(x, 0.99) mean(sapply(1:2000, function(i) { i <- i * 50 quantile(x[i-49:i], 0.99) })) When run multiple times, I've found that the mean is always lower than the quantile over the full sample.
      - stdbrouw 11y ago
        Actually, doh, you're right, my bad, turns out sample quantiles are only asymptotically unbiased, and a better estimator would take some sort of weighted average of P99 and P99.5 with the weights depending on the sample size. (You still don't need the full population and you don't have to merge histograms, though.)