How "recent" are your percentiles? I have found that calculating percentiles is a pretty CPU heavy task. And you if you have a giant Grafana querying every 30s

How "recent" are your percentiles? I have found that calculating percentiles is a pretty CPU heavy task. And you if you have a giant Grafana querying every 30s it can stress out your prometheus/graphite whatever. But if you take small data size, like 95th percentile of latencies in the last 2minutes, it's not really a very accurate representation either. And ofcourse there is another problem of correctly storing all your latencies accurately which becomes pretty hard if you are using something like prometheus.