3 ms·
Can someone explain this? > In practice, teams continue to ignore this problem and instead rely on aggregate time-series that are essentially impossible to int
by karmakaze 7y ago
Can someone explain this?
> In practice, teams continue to ignore this problem and instead rely on aggregate time-series that are essentially impossible to interpret correctly, such as “maximum of 99th percentile latency by server.”
I get that we can't know P99 across all servers. Let's say we're talking about latency and the max of P99's will be an upper bound on the P99 across all servers. Why isn't this true or useful?
- viraptor 7y agoAn example: your server is handling requests to two endpoints. Endpoint A responds in ~100ms, B in ~200ms. You start caching some responses to A higher up. Now your P99 across all servers is higher even though you improved the performance. (Potentially setting off alarms on previous thresholds) While it doesn't solve the alerts, for viewing the data like that heatmaps are amazing. But you need the raw numbers (or predefined bands for aggregation, but that limits flexibility)
- falsedan 7y agoBut your p99 for endpoint B hasn’t changed, why would you alert based on service-wide stats? It’s easy to tag/dimension metrics by urls
- viraptor 7y agoChange the level of caching and you get the same result. Let's say you have the same endpoint, but start caching user X but not others, because reasons.
- falsedan 7y agoStill struggling, if requests from X were slow, caching their responses would only drop or maintain p99. If those requests were fast, p99 might increase & alert... but if you were caring about performance at a user-level, you can imagine you're dimensioning on users too, and you'd see the p99s for each user hadn't changed, and would shrug and bump the alert threshold up. If it were me, I'd probably start by caching the slow requests (that are above the p99) first & adjusting the alert thresholds to match the new performance.
- karmakaze 7y agoWhat you're describing isn't about stats it's about what's being measured. The P99 of both endpoints taken together really does go up if A is called less frequently. Also not a good example as different endpoints wouldn't share a metric.
- Fiadliel 7y agoHow about this: you have a load balancer that sends much less traffic to servers that are running slowly (but still a small amount of traffic, so that it recognises when the machine recovers). One backend server has a hardware problem; requests that usually take milliseconds now take seconds; the 99th percentile on that machine is much higher than anywhere else. But it's actually serving a tiny tiny fraction of traffic. Is the value useful in indicating the performance of the service as a whole?
- karmakaze 7y agoYes in this case max of P99 latencies across all servers is useful. It will be high if any machine's P99 is high indicating that something is not normal. If it's a known ongoing condition, I would make two metrics, one that includes the trouble server in the aggregate and one that doesn't and alert on that to find out new unknown problems.
- Fiadliel 7y agoOkay, I see what you mean. It is a statistic, and so far as it is what it is describes, it's true. And it has some use. The problem is that what people are usually asking for, is the 99th percentile of response time for user requests, and they get the above statistic, not because they want it, but because they are limited by their monitoring systems; they accept it as a proxy for what they want, when it is actually something completely different, and may have completely different values. It would be more useful to calculate the distribution for overall response time, and also calculate the distribution for each individual machine; you could then look for anomalies (in your statistics) that could identify which machines are performing badly.