3 ms·
More knowledge is always better, but percentiles are a little misleading as well - the 99% at 867 ms latency makes you have a moment of panic, but when you see
by SketchySeaBeast 8y ago
More knowledge is always better, but percentiles are a little misleading as well - the 99% at 867 ms latency makes you have a moment of panic, but when you see that 95% is 60 ms, then you really realize how few of your visitors are experiencing the slow response. Might it be a problem? Possibly, and I has brought awareness to that potential, but it also has the possibility to blow it out of proportion if you don't look at the rest of the data.
Edit: I'm not saying Averages are better, but that Percentiles can be misleading as well.
- Pfhreak 8y agoHaving multiple percentiles is key, but I can't think of a time when I've ever found average to be useful.
- deleted 8y ago[deleted]
- dragontamer 8y agoThe average (or more specifically: arithmetic mean) has a number of key properties that allows for advanced analysis. If you have two pieces of data: say the average roll of a D6 (6-sided dice) + average roll of D20 (20-sided dice), you wouldn't be able to do anything with percentiles. The 90% percentile roll of D20 is 18. The 90% percentile roll of D6 is 5(ish). But the sum of these numbers tells us nothing (18 + 5 == 23, which tells us... nothing). In contrast, the mean / average roll of D20 is 10.5, while the average roll of D6 is 3.5. The average roll of D20 + average of D6 is 14. You can add and subtract averages just fine, combining separate pieces of data to make a larger conclusion. You cannot do the same with percentages. ---------- There's nothing "misleading" about averages. Its just that most people are awful at statistics. The real lesson is... learn statistics, so that you can learn to use these tools correctly. Averages, and standard deviation / variance, are excellent for combining data. Its a blurry picture, but its still mathematically correct. Percentiles / quartiles / graphs are more precise and allow for deeper conclusions... but its not always possible to create a percentile graph, especially if you cannot directly measure some attribute. "Indirect measurement", by way of arithmetic, averages, and standard deviation, is quite useful. ---------- EDIT: While I'm talking statistics, don't forget about the three kinds of averages: Arithmetic Mean is the most common, but is often meaningless. You have to also understand geometric mean, and harmonic mean. Multiplicative problems should use a geometric mean. Harmonic Mean is used when comparing speeds (roughly). Going 60MPH for 10 miles, and then going 120MPH for 10 miles is averaged using Harmonic Mean. Benchmarks should typically use harmonic mean: the arithmetic mean is meaningless in the scope of benchmarks. Etc. etc. But this is all statistics, that for some reason is rarely taught well in the US High School level. IMO, its an issue with our school system... we really should be teaching more statistics, especially given how common the analysis of data is in today's world.
- coldtea 8y ago>There's nothing "misleading" about averages Well, the fact that the average of my wealth and a Bill Gates' wealth is dozens of billions of dollars shows why averages are misleading -- and frequently used to give a false image with statistics. Misleading doesn't necessarily mean mathematically or factually wrong. Something can be totally true, and still give the false impression, and averages do that. And saying "people should learn statistics" wont change this. Even knowing statistics, the average doesn't tell me much.
- dragontamer 8y ago> Well, the fact that the average of my wealth and a Bill Gates' wealth is dozens of billions of dollars shows why averages are misleading -- and frequently used to give a false image with statistics. I'm not sure I follow your example. But let me explain my point of view first. Lets say we have a game, where we flip a coin. Heads, you win your amount of money (lets say $100,000). Tails, you win Bill Gate's money (lets say $50 Billion). If we play the game 50 times, how much money will you make on the average? Well, that's just 50 times the expected value of the game. Even if we have grossly separate results for heads vs tails, the mathematical properties of the arithmetic mean / common average remains the same. ---------- "Proper use" of averages depends entirely upon the use of the number afterwards. How you're interpreting the data and why. That's solidly within the realm of statistics: understanding exactly what "Average" means, and using the math correctly. I fully agree with you that very few people out there seem to understand the definition of "Average". In fact, I personally prefer the more precise term "arithmetic mean", specifically so that I don't mislead those who say "Average" (also, "average" is ambiguous in the statistical world: mean, median mode? If mean, then arithmetic, geometric, or harmonic??). But that's more of a writing issue as opposed to an interpretation / mathematical issue. I've used ALL average calculations (median, mode, arithmetic mean, geometric mean, and harmonic mean) at some point in my life. They're useful, and I'm not even a statistician by trade. (Actually, I use those calculations mostly in my video-game analysis...)
- ComputerGuru 8y agoYou have a knack for explaining statistics in an approachable way.
- chewbacha 8y agoPercentages without volume are still useless though. 99% percentile of 1 billion requests could still mean it's impacting a huge number of users. moral: No one number is a panacea
- pmart123 8y agoPercentiles can be misleading if the data follows a Poisson distribution, especially if the lambda coefficient is closer to 1. Latencies I would imagine, would typically be Poisson due to arrival time.
- the8472 8y ago> the 99% at 867 ms latency makes you have a moment of panic, but when you see that 95% is 60 ms, then you really realize how few of your visitors are experiencing the slow response. And then you realize that visitors are hitting your application with hundreds of requests per page load and the 99th percentile suddenly becomes your average. And then you realize that you didn't plot the windowed maximum and have some crazy hangs every now and then that block entire page loads for a whole minute.
- Xorlev 8y ago> but that Percentiles can be misleading as well. I'm not sure I agree. If they're computed wrong, sure, but this is what your system is actually doing. And honestly, the tail has a way of dictating your system's performance as a whole. > the 99% at 867 ms latency makes you have a moment of panic, but when you see that 95% is 60 ms It's easy to write off 1 in 100 users, but the reality is a little more dim. If your P(slow request) is normally distributed (it isn't always -- some requests are more expensive, some data is on worse disks, etc.), then you can compute the (extremely rough) probability a user will run into a slow request in a session: P(slow request for user) = 1 - (0.99)^N N = number of requests. For example, lets say a user visits 15 pages in a session with that call in each. They have a ~13.9% chance of running into that 99th percentile. :( Now if you're fanning out lookups (as one often does), you could easy have 50 lookups for a single request. Now you're at 39.5%! What happens in 1% of requests can become extremely important and essentially dictate your user's experience. The Tail at Scale [1] talks a lot about this. I'd recommend it as reading. [1] https://blog.acolyer.org/2015/01/15/the-tail-at-scale/ https://blog.acolyer.org/2015/01/15/the-tail-at-scale/