4 ms·

Looking at this architecture, I wonder if it would make sense to replace Prometheus itself with something simpler? Normally Prometheus pulls metrics, evaluates

Looking at this architecture, I wonder if it would make sense to replace Prometheus itself with something simpler? Normally Prometheus pulls metrics, evaluates rules against them, stores them, and answers queries on them. It looks like in Cortex the latter three functions are handled by other services, and Prometheus is just pulling and forwarding metrics. Also, the article says Cortex "completes" Prometheus. While it does fill many of the gaps, there is still a big one: First-class support for pushing metrics. This seems especially important if you want to offer Prometheus as SaaS. Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all), and asking customers to open their firewalls so that your Prometheus can discover and scrape their infrastructure is probably a non-starter.

GauntletWizard 8y ago
Prometheus is winning because push-based metrics are the wrong approach. Integration with service discovery to tell if your metrics are operating or not is, in my experience, the #2 or #3 preventer of outages.
- sciurus 8y ago
  If I had to speculate why Prometheus is doing well, I'd put the simplicity of deployment and ongoing operations first, and probably the great integration with Kubernetes second. I think that if it had gone with a push-based model it would have been just as successful. Push vs Pull is unrelated to integrating service discovery. Right now I push metrics to Datadog. It can tell if my metrics are operating or not, because it's aware of the state of the Datadog agent and of what the EC2 API reports.
- comboy 8y ago
  I love Prometheus but in some cases push-based metrics have they advantages. E.g. I'm using push based system in my home automation, where apart from metrics (stats of temp and humidity) it also works as a state machine (door open, switch off) so that I can build logic on top of that by listening to changes. I think having realtime data helps in evaluating some server problems too. Obviously everything depends on use case and not every solution scales well. Re having downtime indicator that's really a few lines of code, if no event received in the last X seconds then the host is down or busy enough to need attention. With websocket connection you can react to such downtime pretty much immediately instead of waiting for the next poll. I know there are workarounds but I really wish Prometheus would support some push based system out of the box. It also helps with tracking simple scripts that only run occasionally etc. And yes, I know, they provide workarounds, the docs are really good. It seems to me that having a realtime system with push based metrics is cheap enough that we should have it already. Imagine ideal monitoring system of the future. Does it poll and refresh data every so often or are you able to see everything as it's happening?
- deleted 8y ago
  [deleted]
NikolaeVarius 8y ago
Supporting push metrics is not a Gap. It is literally the design goal of Prometheus to have a pull based metrics model.
- netingle 8y ago
  Whatsmore, Cortex is push based...
deforciant 8y ago
I am not sure why are you saying that the customers don't want to run their own instances as they are very lightweight. It makes architecture really flexible as one Prom instance can scrape whole environment. New applications just need to expose /metrics endpoint and set some annotations on their deployment manifests. The main problems with running Prometheus yourself are just storage capacity and query speed. I played with Cortex for a bit, built my own gateway to match & assign user IDs for incoming Prometheus push requests but from operational perspective running Cortex can be very expensive (if you want it just as a side project) :) GCP Bigtable seems like the best way to store & query data but even the dev instance is prohibitively expensive (last time I checked it was ~1k USD)
- sciurus 8y ago
  At that point you're just treating Prometheus as an agent, like telegraf or collectd, running in the customers infrastructure to collect metrics from it and publish them to Cortex. That gets to my first point that Prometheus does a lot more than that, so maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.
  - slgeorge 8y ago
    maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. The way the user runs it means that they don't store any metrics locally. The Prometheus agent on the cluster does scraping and then sends the data immediately up to the central SaaS instance (Cortex). what if your potential customer runs on a PaaS like Heroku The range of exporters is very extensive (https://prometheus.io/docs/instrumenting/exporters/ https://prometheus.io/docs/instrumenting/exporters/). And, as K8s has become so important lots of other people are publishing Prometheus metrics. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you. Cortex provides a scalable backend storage for Prometheus metrics, it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.
hangonhn 8y ago
There are reasons for push-vs-pull but one huge advantage of pull vs push is that you're less likely to get overwhelmed by useless metrics. You have to make a very deliberate decision to take the metrics you want. It also makes sense that the observer decides what to observe. Also, if you really, really want push, there's always the push gateway in the Prometheus tool kit. Also, you can use Telegraf from InfluxDB with Prometheus.
- postwait 8y ago
  I think this misrepresents the pull vs. push issue. I wrote something up two years about this that was pretty well received. https://www.circonus.com/2016/06/pully-mcpushface/ https://www.circonus.com/2016/06/pully-mcpushface/
  - hangonhn 8y ago
    Thanks for that amazing insight! Super helpful. I like how you framed your thinking.
slgeorge 8y ago
This seems especially important if you want to offer Prometheus as SaaS. Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all), and asking customers to open their firewalls so that your Prometheus can discover and scrape their infrastructure is probably a non-starter. At Weaveworks we provide Prometheus as a SaaS using Cortex, which is how the project was started. For our service we have an agent that installs a Prometheus agent (and some other ones) on the customers K8s cluster. The upsides of running the agent on their side is that they benefit from Prometheus' automatic service discovery. They can also use standard Prometheus configuration and scrapers to get the full capabilities for their environment. There are also situations where the user wants to continue to run their own Prometheus, but down sample and send a limited number of metrics for long-term storage in the Weave Cloud SaaS. This can be done with one line in a standard Prometheus. It's true that running the agent takes resources, I think the benefits are worth it. For most people the key question is getting the full value from their metrics data to enable better development and operations. [NOTE]: Full Disclosure I work at Weaveworks where we provide Prometheus-as-a-Service (https://www.weave.works/features/prometheus-monitoring/ https://www.weave.works/features/prometheus-monitoring/)
netingle 8y ago
I recently opened a PR to add support for pushing metrics to Cortex from Telegraf, so you don't need Prometheus as the agent: https://github.com/influxdata/telegraf/pull/5003 https://github.com/influxdata/telegraf/pull/5003 NB I'm the main author of Cortex...

GauntletWizard 8y ago

Prometheus is winning because push-based metrics are the wrong approach. Integration with service discovery to tell if your metrics are operating or not is, in my experience, the #2 or #3 preventer of outages.

sciurus 8y ago
If I had to speculate why Prometheus is doing well, I'd put the simplicity of deployment and ongoing operations first, and probably the great integration with Kubernetes second. I think that if it had gone with a push-based model it would have been just as successful. Push vs Pull is unrelated to integrating service discovery. Right now I push metrics to Datadog. It can tell if my metrics are operating or not, because it's aware of the state of the Datadog agent and of what the EC2 API reports.
comboy 8y ago
I love Prometheus but in some cases push-based metrics have they advantages. E.g. I'm using push based system in my home automation, where apart from metrics (stats of temp and humidity) it also works as a state machine (door open, switch off) so that I can build logic on top of that by listening to changes. I think having realtime data helps in evaluating some server problems too. Obviously everything depends on use case and not every solution scales well. Re having downtime indicator that's really a few lines of code, if no event received in the last X seconds then the host is down or busy enough to need attention. With websocket connection you can react to such downtime pretty much immediately instead of waiting for the next poll. I know there are workarounds but I really wish Prometheus would support some push based system out of the box. It also helps with tracking simple scripts that only run occasionally etc. And yes, I know, they provide workarounds, the docs are really good. It seems to me that having a realtime system with push based metrics is cheap enough that we should have it already. Imagine ideal monitoring system of the future. Does it poll and refresh data every so often or are you able to see everything as it's happening?
deleted 8y ago
[deleted]

NikolaeVarius 8y ago

Supporting push metrics is not a Gap. It is literally the design goal of Prometheus to have a pull based metrics model.

netingle 8y ago
Whatsmore, Cortex is push based...

deforciant 8y ago

I am not sure why are you saying that the customers don't want to run their own instances as they are very lightweight. It makes architecture really flexible as one Prom instance can scrape whole environment. New applications just need to expose /metrics endpoint and set some annotations on their deployment manifests. The main problems with running Prometheus yourself are just storage capacity and query speed. I played with Cortex for a bit, built my own gateway to match & assign user IDs for incoming Prometheus push requests but from operational perspective running Cortex can be very expensive (if you want it just as a side project) :) GCP Bigtable seems like the best way to store & query data but even the dev instance is prohibitively expensive (last time I checked it was ~1k USD)

sciurus 8y ago
At that point you're just treating Prometheus as an agent, like telegraf or collectd, running in the customers infrastructure to collect metrics from it and publish them to Cortex. That gets to my first point that Prometheus does a lot more than that, so maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. It's flexible for some architectures, but not others. If your potential customer runs their own servers or something like k8s that they can deploy Prometheus to, sure. But what if your potential customer runs on a PaaS like Heroku? Or a FaaS like lambda? Or what if what they primarily care about getting metrics from are jobs on some managed data processing service, like cloud dataflow? Etc. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you.
- slgeorge 8y ago
  maybe it would be beneficial to replace it or strip it down to ensure your customer never has to worry about storage performance, for example. The way the user runs it means that they don't store any metrics locally. The Prometheus agent on the cluster does scraping and then sends the data immediately up to the central SaaS instance (Cortex). what if your potential customer runs on a PaaS like Heroku The range of exporters is very extensive (https://prometheus.io/docs/instrumenting/exporters/ https://prometheus.io/docs/instrumenting/exporters/). And, as K8s has become so important lots of other people are publishing Prometheus metrics. In the end, to offer Prometheus/Cortex as a SaaS and be competitive with existing players like Datadog, SignalFX, etc I'd expect you'd have to run an API that your customers can code against to push data to you. Cortex provides a scalable backend storage for Prometheus metrics, it's designed as one component that someone running at large scale or a SaaS vendors needs. Your comparison isn't apples to apples. In Weaveworks case we've build integrations into our product that help connect and pull data from other systems. Overall I think Prometheus metrics have a strong future, and there will be lots of SaaS options for sending and consuming the data.

hangonhn 8y ago

There are reasons for push-vs-pull but one huge advantage of pull vs push is that you're less likely to get overwhelmed by useless metrics. You have to make a very deliberate decision to take the metrics you want. It also makes sense that the observer decides what to observe. Also, if you really, really want push, there's always the push gateway in the Prometheus tool kit. Also, you can use Telegraf from InfluxDB with Prometheus.

postwait 8y ago
I think this misrepresents the pull vs. push issue. I wrote something up two years about this that was pretty well received. https://www.circonus.com/2016/06/pully-mcpushface/ https://www.circonus.com/2016/06/pully-mcpushface/
- hangonhn 8y ago
  Thanks for that amazing insight! Super helpful. I like how you framed your thinking.

slgeorge 8y ago

This seems especially important if you want to offer Prometheus as SaaS. Your customers won't want to run their own Prometheus installs (that's what they pay you for, after all), and asking customers to open their firewalls so that your Prometheus can discover and scrape their infrastructure is probably a non-starter. At Weaveworks we provide Prometheus as a SaaS using Cortex, which is how the project was started. For our service we have an agent that installs a Prometheus agent (and some other ones) on the customers K8s cluster. The upsides of running the agent on their side is that they benefit from Prometheus' automatic service discovery. They can also use standard Prometheus configuration and scrapers to get the full capabilities for their environment. There are also situations where the user wants to continue to run their own Prometheus, but down sample and send a limited number of metrics for long-term storage in the Weave Cloud SaaS. This can be done with one line in a standard Prometheus. It's true that running the agent takes resources, I think the benefits are worth it. For most people the key question is getting the full value from their metrics data to enable better development and operations. [NOTE]: Full Disclosure I work at Weaveworks where we provide Prometheus-as-a-Service (https://www.weave.works/features/prometheus-monitoring/ https://www.weave.works/features/prometheus-monitoring/)

netingle 8y ago

I recently opened a PR to add support for pushing metrics to Cortex from Telegraf, so you don't need Prometheus as the agent: https://github.com/influxdata/telegraf/pull/5003 https://github.com/influxdata/telegraf/pull/5003 NB I'm the main author of Cortex...