3 ms·
An example: your server is handling requests to two endpoints. Endpoint A responds in ~100ms, B in ~200ms. You start caching some responses to A higher up. Now
by viraptor 7y ago
An example: your server is handling requests to two endpoints. Endpoint A responds in ~100ms, B in ~200ms. You start caching some responses to A higher up. Now your P99 across all servers is higher even though you improved the performance. (Potentially setting off alarms on previous thresholds)
While it doesn't solve the alerts, for viewing the data like that heatmaps are amazing. But you need the raw numbers (or predefined bands for aggregation, but that limits flexibility)
- falsedan 7y agoBut your p99 for endpoint B hasn’t changed, why would you alert based on service-wide stats? It’s easy to tag/dimension metrics by urls
- viraptor 7y agoChange the level of caching and you get the same result. Let's say you have the same endpoint, but start caching user X but not others, because reasons.
- falsedan 7y agoStill struggling, if requests from X were slow, caching their responses would only drop or maintain p99. If those requests were fast, p99 might increase & alert... but if you were caring about performance at a user-level, you can imagine you're dimensioning on users too, and you'd see the p99s for each user hadn't changed, and would shrug and bump the alert threshold up. If it were me, I'd probably start by caching the slow requests (that are above the p99) first & adjusting the alert thresholds to match the new performance.
- karmakaze 7y agoWhat you're describing isn't about stats it's about what's being measured. The P99 of both endpoints taken together really does go up if A is called less frequently. Also not a good example as different endpoints wouldn't share a metric.