Metrics can be useful when they are thoughtfully chosen for the context in which they are used. The more abstract or synthetic a benchmark is, the less context

Metrics can be useful when they are thoughtfully chosen for the context in which they are used. The more abstract or synthetic a benchmark is, the less context there is to inform such a choice. Said another way, metrics characterizing synthetic benchmarks inherit/expose one of the key problems of synthetic benchmarks - a lack of context. In practical settings, the way to characterize performance will be more clear from context, often giving one (-ish) clear metric for each dimension you care about. For example, if an endpoint has a hard timeout of 100 ms, then it's pretty interesting to look at the percentage of real world requests that are faster than 100 ms. If the same solution is also used in a setting where throughput is more important than latency, then an additional metric is probably needed for that use case. Multiple metrics are needed for multiple use cases to capture trade-offs being made between those use cases.