3 ms·

NewRelic graphs are another highly misleading example. They average the response time across all endpoints and make it very hard to understand the 95th percent

NewRelic graphs are another highly misleading example. They average the response time across all endpoints and make it very hard to understand the 95th percentile for a specific endpoint. They also don't let you exclude static assets from the graphs, so the numbers are fairly unhelpful when trying to understand performance bottlenecks of a dynamic application.

bbrazil 11y ago
In Prometheus we solved this by having Histograms where we exported cumulative buckets of latencies, which can be aggregated and then the quantiles calculated from them. http://prometheus.io/docs/practices/histograms/#quantiles http://prometheus.io/docs/practices/histograms/#quantiles
deleted 11y ago
[deleted]
Lightbody 11y ago
Disclaimer: I'm part of New Relic's Product Management team. I think you might have a misunderstanding of how our stuff works. While we originally only captured aggregates, the last ~2 years or so we've been capturing and reporting on every transaction/request taking place. As such, when you do a 95th percentile chart in our product, we're not "averaging the percentile" like many monitoring tools do. We are literally looking at every single record during that time. We also allow filtering by transaction, which means you can check out just the percentile for "CheckoutController" or "AddToCartController" -- definitely not just the aggregate application. And if you're a customer and want to verify this yourself, just pop over to New Relic Insights (insights.newrelic.com) and run a query like this to really see the power that comes from not pre-aggregating anything: SELECT count(*), percentile(duration, 50, 90, 87, 95) FROM Transaction SINCE 1 day ago FACET BY name That will return all transactions in the last day, grouped by transaction name (ex: "CheckoutController"), along with their respective count and median, 90th, and 95th percentile. I threw in the 87th percentile just to show that you can do that too if that's your kind of thing.
- grandalf 11y ago
  > every transaction/request taking place. It's definitely gotten better, however I think the default view is still just averaging all requests, which isn't very helpful. For a blocking server (like most Rails apps use) the key insight has to do with which controller actions need to be optimized to prevent slow user-facing aggregate response times. I don't currently have it in use on any of my apps, but next time I'll give that query a try.
  - Lightbody 11y ago
    Yes, the first graph is for the entire app. But we (try to) make it easy to quickly drop down to a view that shows you the most time consuming controllers and then let you quickly see performance for just a single one. Feel free to drop me an email at lightbody@newrelic.com if you decide to check it out again and have more feedback. Take care!
    - grandalf 11y ago
      I appreciate the response here, and appreciate the offer.
  - joshwa 11y ago
    This is a problem with other APM tools, too... someone inevitably wants to know in one number how the system is responding, and the best you can do naively is to average all of them. Defining critical transactions, and measuring them against their SLAs is really the only valid way of summarizing total application performance (x% of traffic is meeting SLA, y% is 1 stddev above, z% 2 stddev, etc).

bbrazil 11y ago

In Prometheus we solved this by having Histograms where we exported cumulative buckets of latencies, which can be aggregated and then the quantiles calculated from them. http://prometheus.io/docs/practices/histograms/#quantiles http://prometheus.io/docs/practices/histograms/#quantiles

deleted 11y ago

[deleted]

Lightbody 11y ago

Disclaimer: I'm part of New Relic's Product Management team. I think you might have a misunderstanding of how our stuff works. While we originally only captured aggregates, the last ~2 years or so we've been capturing and reporting on every transaction/request taking place. As such, when you do a 95th percentile chart in our product, we're not "averaging the percentile" like many monitoring tools do. We are literally looking at every single record during that time. We also allow filtering by transaction, which means you can check out just the percentile for "CheckoutController" or "AddToCartController" -- definitely not just the aggregate application. And if you're a customer and want to verify this yourself, just pop over to New Relic Insights (insights.newrelic.com) and run a query like this to really see the power that comes from not pre-aggregating anything: SELECT count(*), percentile(duration, 50, 90, 87, 95) FROM Transaction SINCE 1 day ago FACET BY name That will return all transactions in the last day, grouped by transaction name (ex: "CheckoutController"), along with their respective count and median, 90th, and 95th percentile. I threw in the 87th percentile just to show that you can do that too if that's your kind of thing.

grandalf 11y ago
> every transaction/request taking place. It's definitely gotten better, however I think the default view is still just averaging all requests, which isn't very helpful. For a blocking server (like most Rails apps use) the key insight has to do with which controller actions need to be optimized to prevent slow user-facing aggregate response times. I don't currently have it in use on any of my apps, but next time I'll give that query a try.
- Lightbody 11y ago
  Yes, the first graph is for the entire app. But we (try to) make it easy to quickly drop down to a view that shows you the most time consuming controllers and then let you quickly see performance for just a single one. Feel free to drop me an email at lightbody@newrelic.com if you decide to check it out again and have more feedback. Take care!
  - grandalf 11y ago
    I appreciate the response here, and appreciate the offer.
- joshwa 11y ago
  This is a problem with other APM tools, too... someone inevitably wants to know in one number how the system is responding, and the best you can do naively is to average all of them. Defining critical transactions, and measuring them against their SLAs is really the only valid way of summarizing total application performance (x% of traffic is meeting SLA, y% is 1 stddev above, z% 2 stddev, etc).