4 ms·

This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single

This means each new query over a certain size becomes a federation problem, so the friction for trying new things becomes very high above the scale of a single instance. Monitoring as a service has a lot of advantages.

preseinger 4y ago
Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.
- gttalbot 4y ago
  Monarch can support both ad-hoc and periodic, standing queries of arbitrarily large size, and has the means to spread the computation out over many intermediate mixer and leaf nodes. It does query push-down so that the "expensive" parts of aggregations, joins, etc., can be done in massively parallel fashion at the leaf level. It scales so well that many aggregations are set up and computed for every service across the whole company (CPU, memory usage, error rates, etc.). For basic monitoring you can run a new service in production and go and look at a basic dashboard for it without doing anything else to set up monitoring.
- gttalbot 4y ago
  Also, very large ad-hoc queries are supported, with really good user isolation, so that (in general) users don't harm each other.
- xyzzy_plugh 4y ago
  That's a choice Prometheus has made, not an invariant. Many scalable systems support arbitrarily large queries as they themselves scale. Prometheus pretty much prevents creating arbitrarily large datasets to begin with, so the point is kind of moot. Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.
  - hiptobecubic 4y ago
    To be fair, monarch has scaling limits that require you to be aware of how aggregation is done as well. It's amazing what it can do, but depending on how much data you have you might need to design your schema/collection with monarch's architecture in mind.

preseinger 4y ago

Well you obviously don't issue metrics queries over arbitrarily large datasets, right? The Prometheus architecture reflects this invariant. You constrain queries against both time and domain boundaries.

gttalbot 4y ago
Monarch can support both ad-hoc and periodic, standing queries of arbitrarily large size, and has the means to spread the computation out over many intermediate mixer and leaf nodes. It does query push-down so that the "expensive" parts of aggregations, joins, etc., can be done in massively parallel fashion at the leaf level. It scales so well that many aggregations are set up and computed for every service across the whole company (CPU, memory usage, error rates, etc.). For basic monitoring you can run a new service in production and go and look at a basic dashboard for it without doing anything else to set up monitoring.
gttalbot 4y ago
Also, very large ad-hoc queries are supported, with really good user isolation, so that (in general) users don't harm each other.
xyzzy_plugh 4y ago
That's a choice Prometheus has made, not an invariant. Many scalable systems support arbitrarily large queries as they themselves scale. Prometheus pretty much prevents creating arbitrarily large datasets to begin with, so the point is kind of moot. Requiring query authors to understand the arrangement of their aggregation layer seems like a reasonable idea but is in fact quite ridiculous.
- hiptobecubic 4y ago
  To be fair, monarch has scaling limits that require you to be aware of how aggregation is done as well. It's amazing what it can do, but depending on how much data you have you might need to design your schema/collection with monarch's architecture in mind.