3 ms·
Jim Roskind had a really nice talk [https://www.youtube.com/watch?v=_uaaCiyJCFA https://www.youtube.com/watch?v=_uaaCiyJCFA] at AWS Reinvent 2022 which highligh
by dadkins 3mo ago
Jim Roskind had a really nice talk [https://www.youtube.com/watch?v=_uaaCiyJCFA https://www.youtube.com/watch?v=_uaaCiyJCFA] at AWS Reinvent 2022 which highlighted the fence post problem. Basically, AWS was targeting P50 and P90 latencies in many services, and sure enough there were big clumps of latencies right under those target fence posts as engineers were basically gaming the metrics and nerfing the >P90 latencies to make it happen.
- fc417fc802 3mo agoI don't think that counts as gaming the metric. It's an example of a metric that even when targeted in good faith results in consequences management didn't intend.
- TeMPOraL 3mo agoGood faith starts with understanding that "90" in P90 is an arbitrary number, and that the exact sampling threshold is not important to why anyone cares about that metric.
- khuey 3mo agoThe 99 in his TM99 is also an arbitrary number. The more interesting part of the talk is how using what he calls "fenceposts" naturally results in bunching up against the fenceposts because even small regressions that cross the fenceposts are blocked but regressions of any size that don't cross a fencepost go unnoticed.
- fc417fc802 3mo agoIt's just a plain bad metric. Useful due to brevity though. A decent metric for testing against might consist of a target mean, the sigma specifying a gaussian about that mean, a minimum sample size, and a hard upper cutoff.
- keynha 3mo ago[dead]