3 ms·
The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic co
by jameshart 7y ago
The issue is that percentiles are a very crude tool. This isn’t just a multitenancy problem, it can manifest in any multiuser system. If 0.5% of your traffic comes from New Zealand, a DNS issue affecting your CDN routing that causes all NZ traffic to time out won’t affect your 99th percentile loadtime graph at all. Essentially, percentiles are useless for discovering problems that have a strong effect on a small portion of your traffic.
Scatter plots and histograms are much better at telling you when the distribution of a stat has gone bimodal with a small but consistent group in the outlier group. Percentiles only make sense for telling you, when you already know what shape the distribution curve should be, how flat that curve is right now. They don’t tell you when the shape has changed.
- bosie 7y agoWhat scatter plot and/or histogram do you have in mind to dig up the small but consistent group in the outlier group?
- jameshart 7y agoAh, yeah - figuring out what they have in common is the trick :) Good APMs and trace tools will let you zero in on traces by characteristics - so if you notice there’s a bump in requests which have a 2 second load time, you can select them all and analyze how they are distributed - whether they are mostly one browser, one location, or one user even. But you need a solid strategy for tagging and logging traces.
- jgtrosh 7y agoYou could statistically test whether your data matches the expected distribution with automated tests.