4 ms·
Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just
by imtringued 7y ago
Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but not that good for analyzing them.
- lonelappde 7y agoThe main objection is that it's hard to get a 3D graph of the distribution over time. But it's still worth drawing some snapshots of the distribution.
- meatmanek 7y agoHeat maps (showing histogram bin count as color/intensity instead of bar height) and scatterplots (potentially of a subset of data points) are both relatively straightforward ways to visualize distributions over time.
- TeMPOraL 7y ago> Of course graphing percentiles over time is much easier because they just represent a single value. Ridgeline plots (joyplots) are severely underutilized.
- kasey_junk 7y agoI generally love joyplots but haven’t seen a great use of them showing latency distribution over time. Do you know of any examples?
- mncharity 7y agoR https://cran.r-project.org/web/packages/ggridges/vignettes/introduction.html https://cran.r-project.org/web/packages/ggridges/vignettes/i... ; d3 https://observablehq.com/@d3/ridgeline-plot https://observablehq.com/@d3/ridgeline-plot ; py https://github.com/sbebo/joypy https://github.com/sbebo/joypy .
- hinkley 7y agoThere are also violin plots, which are like a cross between a box plot and a ridgeline plot. https://en.wikipedia.org/wiki/Violin_plot https://en.wikipedia.org/wiki/Violin_plot
- pdpi 7y agoHistograms, too, bin values and thus show only a few values instead of the whole distribution. In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.
- djspoons 7y agoWell, yeah, there's not much difference between a histogram and hundreds-of-percentiles. But there's a huge difference between a histogram and just a single percentile (whether it's p50, p95, p99, or whatever). If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really important. Most of the time there are discrete behaviors/factors that are driving performance: timeouts, cache hits/misses, one overloaded host, a canary. The shape of the histogram will help you see those in a way that a single percentile can't.
- djspoons 7y agoActually one of my coworkers gave a talk at Facebook Performance summit on this: https://www.youtube.com/watch?v=EG7Zhd6gLiw https://www.youtube.com/watch?v=EG7Zhd6gLiw (Disclaimer: brief product pitch in the first minute as part of the speaker intro)
- OliverJones 7y agoYes, absolutely, show the distribution whenever you can. But consider production alerting purposes. For example, consider database user-lookup time. Use the 95th percentile, or even the maximum value to compare with an alert threshold. In many (most?) real-world cases system trouble shows up as a few lookup-time outliers. If you only look at averages, or even medians, you miss those outliers.
- kqr 7y agoHeatmaps sorta-kinda solve the problem of plotting histograms over time, except of course it's harder to detect differences in brightness than size.
- GGfpc 7y agoI tend to use cumulative distribution functions when comparing distributions. Histograms also tend to be quite easy to manipulate