4 ms·
Generally I want 90th, 99th, and max on the same graph/plot. Otherwise there's less sense of the overall distribution. 90th percentile is the fast path. You
by benlivengood 6y ago
Generally I want 90th, 99th, and max on the same graph/plot. Otherwise there's less sense of the overall distribution.
90th percentile is the fast path. You notice overall effects of growth, load, regressions, etc. 99th percentile (or sometimes 99.9th) is often the actual SLA/SLO so it should be on the plot. Max, as you point out, is necessary to actually see the long tail.
The other thing that's important is to make sure the timeseries and plotting software isn't averaging the max value; it's easy to miss spikes of latency because someone thought the graphs looked nicer with avg(1 min) vs max(1 min) or a tsdb is automatically rewindowing/aggregating with the wrong function. To a lesser extent even percentile buckets can be deceiving if they're too big. 99th percentile with 5-min buckets can miss significant but brief (~3 second) degradations.
- solipsism 6y agoMax, as you point out, is necessary to actually see the long tail. What a strange way to put it. No one wants to see the long tail. The long tail is the means, not the end. To a lesser extent even percentile buckets can be deceiving if they're too big. 99th percentile with 5-min buckets can miss significant but brief (~3 second) degradations. Now that's the actual reason to care about max. Otherwise you have values (and often negative user experiences) getting lost in the aggregation.