4 ms·

true, percentiles are important for telling you how your system is doing and how typical users experience it. However, I contend that you should also measure ma

true, percentiles are important for telling you how your system is doing and how typical users experience it. However, I contend that you should also measure max latency, just that is is less important for understanding typical usage. But seeing your max go crazy can signal that there is something else in the stack that needs work. Silly example that I hit: max was consistently 2hr on something that p99'd at like 1min and p50'd at 2 seconds. Dug in. The 2hr was a default tcp timeout being tripped due to some connections not being properly returned to the pool. Fix the pool logic and max times came down into the few minute range. This also lowered p99.

corytheboyd 6y ago
I agree with this, don't exclude extremes because a blog post or comment said it was bad, they can still be useful sometimes. For general purpose web development response times, I like to refer to p99 and p95 for general healthiness. P50 doesn't do much more than make me feel better. Max has its uses, it can be a good timeout gauge as the commenter above me pointed out. If the RATE at which max values occur changes, you may have something worth investigating.
philwelch 6y ago
Good points. To my point though, you mention that you noticed a "consistent" max of 2hr. Meaning the consistency was the interesting part. You probably consider this issue to be more important than some other hypothetical issue that caused, say, literally a single request to have a 2hr latency on Tuesday. In other words, if you monitored over a period that was long enough to establish a statistically significant p99.9 or even p99.99 and then measured those percentiles, you still would have not only caught this issue, but also distinguished it from the other hypothetical issue. Monitoring your maximum and worrying about consistently high maximums might be a more effective mechanism, but in principle, you still care more about top percentiles.
- eru 6y ago
  In sum: top percentiles are more important than the max; but monitoring the max can still be useful. Just like monitoring the arithmetic average ain't completely useless either. It's just far from the most important metric.

corytheboyd 6y ago

I agree with this, don't exclude extremes because a blog post or comment said it was bad, they can still be useful sometimes. For general purpose web development response times, I like to refer to p99 and p95 for general healthiness. P50 doesn't do much more than make me feel better. Max has its uses, it can be a good timeout gauge as the commenter above me pointed out. If the RATE at which max values occur changes, you may have something worth investigating.

philwelch 6y ago

Good points. To my point though, you mention that you noticed a "consistent" max of 2hr. Meaning the consistency was the interesting part. You probably consider this issue to be more important than some other hypothetical issue that caused, say, literally a single request to have a 2hr latency on Tuesday. In other words, if you monitored over a period that was long enough to establish a statistically significant p99.9 or even p99.99 and then measured those percentiles, you still would have not only caught this issue, but also distinguished it from the other hypothetical issue. Monitoring your maximum and worrying about consistently high maximums might be a more effective mechanism, but in principle, you still care more about top percentiles.

eru 6y ago
In sum: top percentiles are more important than the max; but monitoring the max can still be useful. Just like monitoring the arithmetic average ain't completely useless either. It's just far from the most important metric.