6 ms·

I don't get the hype around Prometheus. What makes its pull-based mechanism superior to push-based ones like statsd? And using exporters sounds clunky - inste

by lima 9y ago

I don't get the hype around Prometheus. What makes its pull-based mechanism superior to push-based ones like statsd? And using exporters sounds clunky - instead of directly querying a metric and sending it to your metrics collector, you have an intermediate component which exposes them for collection.

tzumby 9y ago
I might be wrong but one reasoning I saw mentioned was that pulling would give you a better picture into failures. If one of the services you are monintoring goes down it won't be able to push anything.
- lima 9y ago
  But where's the difference between not being able to pull vs. not receiving metrics for a period of time?
  - sz4kerto 9y ago
    > But where's the difference between not being able to pull vs. not receiving metrics for a period of time? Let's say I have a service that's only used once per day, at a random point in the day. I need to know if the service is available all the time, even if it's not used. With pull, I'll get back an 'OK' e.g. every minute. You could say that we should implement heartbeats, but again, then this service needs to be configurable to pick up changes in the monitoring infrastructure, then I need to have a separate check for the presence of heartbeats and the values of the samples..
    - zaroth 9y ago
      Monitoring if the process is running, if the input queue is free, if it will take a new job, etc. seems to me the responsibility for all that should be with whatever supervisor has the ability to reset/restart the service. Then it can report stats on uptime and number of restarts required. What if the service spins up on demand when it's needed, does its job, and then exits. You want to track how often it starts, how long it runs, how long the job was in the queue, how much CPU/RAM/IO was consumed, etc. With push that's all part of the cleanup code in the process. With pull it seems like you could completely miss that the process was even ever run?
      - bbrazil 9y ago
        That's a short lived batch job, for which a different approach is needed - basically push is the sane option there as pull won't cut it. Prometheus has the Pushgateway for this. The push vs pull debate is relevant for longer running daemons, not batch jobs.
  - bostik 9y ago
    Well, there's one thing going for pull over push - and generally I am strongly in favour of push. With push, you need to know where to send the data. So you depend on a fixed configuration. With pull, you can essentially do a periodic nmap sweep to discover and update the monitoring data sources. That's one less thing to go wrong.
    - the_evacuator 9y ago
      Another nice thing about pull is you only pull from the authoritative source. If you have a daemon that has gone rogue your load balancer or naming system can throw it out of the named group/job/whatever even if it can’t shut it off. With push you might have zombies pushing duplicate data for what should be a unique target.
sz4kerto 9y ago
There are services that might be used rarely, but they're still critical. Therefore you either implement a heartbeat-like approach where the monitored system indicates regularly that it's alive, or use pull. Using exporters might sound clunky, but in many cases you need to implement some kind of pull-based system anyway, because sometimes the monitored system does not support the monitoring infrastructure directly. Example: most database systems don't have Graphite or Prometheus support, they just expose their stats. So you'll end up writing or using a component that pulls these stats regularly, then pushes them. Also, when you use pull then you only need to configure the monitoring system, not the monitored services separately (e.g. if you relocate the monitoring system to a different host, then you don't need to update every single monitored service to talk to the new host). I found that people who're new to larger-scale monitoring favor push, because that's somehow more intuitive; but pull really works very well, it's not clunky at all.
- user5994461 9y ago
  You are conflating metrics collection and metrics storage. The metrics collection needs to be done by a local agent installed on the system, for the reason you gave, that's the only place the data is available. The metrics storage is somewhere else. Prometheus does the storage. For the collection, you still have to install collectd, statsd or similar on your hosts. Sure, prometheus could do a HTTP check remotely, but that won't get cover much of anything.
  - atombender 9y ago
    Prometheus does collection, though. The Prometheus pattern with central things like a database (e.g. monitoring Postgres) is to let the exporter (the thing that acts as an intermediary to expose metrics for Prometheus to fetch) run anywhere it wants. It absolutely does not need to be a "local agent". (In fact, if you're using something like Kubernetes, the only thing that needs local access to a node is the exporter that exposes node-specific metrics. Everything else can chat over the network.) The benefit here is that if you have 10 Postgres databases, you can still run just a single exporter and have it extract data from all of them. Or you can run one exporter per database; there's conceptually little difference. On Kubernetes, we usually run an exporter as a sidecar container, which means it can talk to Postgres or whatever on localhost and just live alongside the process that it's exporting metrics for, and we rely on Prometheus' automatic discovery to make Prometheus pull from it. Start a new Postgres instance and its metrics are almost immediately fed into Prometheus. You don't need collectd etc. with Prometheus. There are exporters around for just about anything.
atombender 9y ago
Prometheus does support push. It's just that it's considered such an antipattern that it's been moved into a separate module (the Push Gateway) that you need to run separately. Pulling has a few technical benefits, though. For one, only the puller needs to know what's being monitored; the thing being monitored can therefore be exceedingly simple, dumb and passive. Statsd is similarly simple in that it's just local UDP broadcast, of course, which leads to the next point: Another benefit is that it allows better fine-grained control over when metrics gathering is done, and what. Since Prometheus best practices dictate that metrics should be computed at pull time, it means you can fine-tune collection intervals to specific metrics, and this can be done centrally. And since you only pull from what you have, it means there can't be a rogue agent somewhere that's spewing out data (i.e. what a sibling comment calls "authorative sources"). But to understand why pull is a better model, you have to understand Google's/Prometheus's "observer/reactor" mindset towards large-scale computing; it's just easier to scale up with this model. Consider an application that implements some kind of REST API. You want metrics for things like the total number of requests served, which you'll sample now and then. You add an endpoint /metrics running on port 9100. Then you tell Prometheus to scrape (pull from) http://example.com:9100/metrics http://example.com:9100/metrics. So far so good. The beauty of the model arises when you involve a dynamic orchestration like Kubernetes. Now we're running the app on Kubernetes, which means the app can run on many nodes, across many clusters, at the same time; it will have a lot of different IPs (one IP per instance) that are completely dynamic. Instead of adding a rule to scrape a specific URL, you tell Prometheus to ask Kubernetes for all services and then use that information to figure out the endpoint. This dynamic discovery means that as you take apps up and down, Prometheus will automatically update its list of endpoints and scrape them. Equally importing, Prometheus goes to the source of the data at any given time. The services are already scaled up; there's no corresponding metrics collection to scale up, other than in the internal machinery of Prometheus' scraping system. In other words, Prometheus is observing the cluster and reacting to changes in it to reconfigure it self. This isn't exactly new, but it's core to Google's/Prometheus's way of thinking about applications and services, which has subseqently coloured the whole Kubernetes culture. Instead of configuring the chess pieces, you let the board inspect the chess pieces and configure itself. You want the individual, lower-level apps to be as mundane as possible, let the behavioural signals flow upstream, and let the higher-level pieces make decisions. This dovetails nicely with the observational data model you need for monitoring, anyway: First you collect the data, then you check the data, then you report anomalies within the data. For example, if you're measuring some number that can go critically high, you don't make the application issue a warning if it goes above a threshold; rather, you collect the data from the application as a raw number, then perform calculations (e.g. max over the last N mins, sum over the last N mins, total count, etc.) that you compare against the threshold. In practice, implementing a metrics endpoint is exceedingly simple, and you get used to "just writing another exporter". I've written a lot of exporters, and while this initially struck me as heavyweight and clunky, my mindset is now that an HTTP listener is actually more lightweight than an "imperative" pusher script.
SuperQue 9y ago
It's not about push or pull or exporters, it's about where you aggregate your events. As soon as you are working with a non-trivial number of events, you need to think more carefully about how deal with them. With standard statsd, you're sending a message per event, this might be fine for one or two servers, with a trickle of traffic. But when you've got 100 servers handling 100s to 1000s of requests per second, we're talking about 10-100k/sec. This is a large amount of load to be directly sending to your network. With statsd, you could buffer, but now you're losing the benefit of the live event stream. With Prometheus, we just say keep it all in memory, localy, and as thread-local as possible. This way the cost to update an event is very, very tiny. Much smaller than even a log line. This allows every debug level log line to be recorded as an event counter. For example, the Prometheus Go library only requires ~15 nanoseconds of CPU time to update a single counter. This cheapness allows for sprinkling metrics everywhere in your code with little worry that it'll be a performance problem. On the topic of exporters, stand-alone exporters are only necessary where the existing code doesn't directly support Prometheus's metrics format. The good news here is that we're working with several other metrics systems in order to create a common format for polling and pushing metrics. Yes, insert XKCD joke here. :-) Once we have a better common metrics format, think SNMP but not crazy, "exporters" will be unnecessary.

by lima 9y ago

tzumby 9y ago

I might be wrong but one reasoning I saw mentioned was that pulling would give you a better picture into failures. If one of the services you are monintoring goes down it won't be able to push anything.

lima 9y ago
But where's the difference between not being able to pull vs. not receiving metrics for a period of time?
- sz4kerto 9y ago
  > But where's the difference between not being able to pull vs. not receiving metrics for a period of time? Let's say I have a service that's only used once per day, at a random point in the day. I need to know if the service is available all the time, even if it's not used. With pull, I'll get back an 'OK' e.g. every minute. You could say that we should implement heartbeats, but again, then this service needs to be configurable to pick up changes in the monitoring infrastructure, then I need to have a separate check for the presence of heartbeats and the values of the samples..
  - zaroth 9y ago
    Monitoring if the process is running, if the input queue is free, if it will take a new job, etc. seems to me the responsibility for all that should be with whatever supervisor has the ability to reset/restart the service. Then it can report stats on uptime and number of restarts required. What if the service spins up on demand when it's needed, does its job, and then exits. You want to track how often it starts, how long it runs, how long the job was in the queue, how much CPU/RAM/IO was consumed, etc. With push that's all part of the cleanup code in the process. With pull it seems like you could completely miss that the process was even ever run?
    - bbrazil 9y ago
      That's a short lived batch job, for which a different approach is needed - basically push is the sane option there as pull won't cut it. Prometheus has the Pushgateway for this. The push vs pull debate is relevant for longer running daemons, not batch jobs.
- bostik 9y ago
  Well, there's one thing going for pull over push - and generally I am strongly in favour of push. With push, you need to know where to send the data. So you depend on a fixed configuration. With pull, you can essentially do a periodic nmap sweep to discover and update the monitoring data sources. That's one less thing to go wrong.
  - the_evacuator 9y ago
    Another nice thing about pull is you only pull from the authoritative source. If you have a daemon that has gone rogue your load balancer or naming system can throw it out of the named group/job/whatever even if it can’t shut it off. With push you might have zombies pushing duplicate data for what should be a unique target.

sz4kerto 9y ago

There are services that might be used rarely, but they're still critical. Therefore you either implement a heartbeat-like approach where the monitored system indicates regularly that it's alive, or use pull. Using exporters might sound clunky, but in many cases you need to implement some kind of pull-based system anyway, because sometimes the monitored system does not support the monitoring infrastructure directly. Example: most database systems don't have Graphite or Prometheus support, they just expose their stats. So you'll end up writing or using a component that pulls these stats regularly, then pushes them. Also, when you use pull then you only need to configure the monitoring system, not the monitored services separately (e.g. if you relocate the monitoring system to a different host, then you don't need to update every single monitored service to talk to the new host). I found that people who're new to larger-scale monitoring favor push, because that's somehow more intuitive; but pull really works very well, it's not clunky at all.

user5994461 9y ago
You are conflating metrics collection and metrics storage. The metrics collection needs to be done by a local agent installed on the system, for the reason you gave, that's the only place the data is available. The metrics storage is somewhere else. Prometheus does the storage. For the collection, you still have to install collectd, statsd or similar on your hosts. Sure, prometheus could do a HTTP check remotely, but that won't get cover much of anything.
- atombender 9y ago
  Prometheus does collection, though. The Prometheus pattern with central things like a database (e.g. monitoring Postgres) is to let the exporter (the thing that acts as an intermediary to expose metrics for Prometheus to fetch) run anywhere it wants. It absolutely does not need to be a "local agent". (In fact, if you're using something like Kubernetes, the only thing that needs local access to a node is the exporter that exposes node-specific metrics. Everything else can chat over the network.) The benefit here is that if you have 10 Postgres databases, you can still run just a single exporter and have it extract data from all of them. Or you can run one exporter per database; there's conceptually little difference. On Kubernetes, we usually run an exporter as a sidecar container, which means it can talk to Postgres or whatever on localhost and just live alongside the process that it's exporting metrics for, and we rely on Prometheus' automatic discovery to make Prometheus pull from it. Start a new Postgres instance and its metrics are almost immediately fed into Prometheus. You don't need collectd etc. with Prometheus. There are exporters around for just about anything.

atombender 9y ago

Prometheus does support push. It's just that it's considered such an antipattern that it's been moved into a separate module (the Push Gateway) that you need to run separately. Pulling has a few technical benefits, though. For one, only the puller needs to know what's being monitored; the thing being monitored can therefore be exceedingly simple, dumb and passive. Statsd is similarly simple in that it's just local UDP broadcast, of course, which leads to the next point: Another benefit is that it allows better fine-grained control over when metrics gathering is done, and what. Since Prometheus best practices dictate that metrics should be computed at pull time, it means you can fine-tune collection intervals to specific metrics, and this can be done centrally. And since you only pull from what you have, it means there can't be a rogue agent somewhere that's spewing out data (i.e. what a sibling comment calls "authorative sources"). But to understand why pull is a better model, you have to understand Google's/Prometheus's "observer/reactor" mindset towards large-scale computing; it's just easier to scale up with this model. Consider an application that implements some kind of REST API. You want metrics for things like the total number of requests served, which you'll sample now and then. You add an endpoint /metrics running on port 9100. Then you tell Prometheus to scrape (pull from) http://example.com:9100/metrics http://example.com:9100/metrics. So far so good. The beauty of the model arises when you involve a dynamic orchestration like Kubernetes. Now we're running the app on Kubernetes, which means the app can run on many nodes, across many clusters, at the same time; it will have a lot of different IPs (one IP per instance) that are completely dynamic. Instead of adding a rule to scrape a specific URL, you tell Prometheus to ask Kubernetes for all services and then use that information to figure out the endpoint. This dynamic discovery means that as you take apps up and down, Prometheus will automatically update its list of endpoints and scrape them. Equally importing, Prometheus goes to the source of the data at any given time. The services are already scaled up; there's no corresponding metrics collection to scale up, other than in the internal machinery of Prometheus' scraping system. In other words, Prometheus is observing the cluster and reacting to changes in it to reconfigure it self. This isn't exactly new, but it's core to Google's/Prometheus's way of thinking about applications and services, which has subseqently coloured the whole Kubernetes culture. Instead of configuring the chess pieces, you let the board inspect the chess pieces and configure itself. You want the individual, lower-level apps to be as mundane as possible, let the behavioural signals flow upstream, and let the higher-level pieces make decisions. This dovetails nicely with the observational data model you need for monitoring, anyway: First you collect the data, then you check the data, then you report anomalies within the data. For example, if you're measuring some number that can go critically high, you don't make the application issue a warning if it goes above a threshold; rather, you collect the data from the application as a raw number, then perform calculations (e.g. max over the last N mins, sum over the last N mins, total count, etc.) that you compare against the threshold. In practice, implementing a metrics endpoint is exceedingly simple, and you get used to "just writing another exporter". I've written a lot of exporters, and while this initially struck me as heavyweight and clunky, my mindset is now that an HTTP listener is actually more lightweight than an "imperative" pusher script.

SuperQue 9y ago

It's not about push or pull or exporters, it's about where you aggregate your events. As soon as you are working with a non-trivial number of events, you need to think more carefully about how deal with them. With standard statsd, you're sending a message per event, this might be fine for one or two servers, with a trickle of traffic. But when you've got 100 servers handling 100s to 1000s of requests per second, we're talking about 10-100k/sec. This is a large amount of load to be directly sending to your network. With statsd, you could buffer, but now you're losing the benefit of the live event stream. With Prometheus, we just say keep it all in memory, localy, and as thread-local as possible. This way the cost to update an event is very, very tiny. Much smaller than even a log line. This allows every debug level log line to be recorded as an event counter. For example, the Prometheus Go library only requires ~15 nanoseconds of CPU time to update a single counter. This cheapness allows for sprinkling metrics everywhere in your code with little worry that it'll be a performance problem. On the topic of exporters, stand-alone exporters are only necessary where the existing code doesn't directly support Prometheus's metrics format. The good news here is that we're working with several other metrics systems in order to create a common format for polling and pushing metrics. Yes, insert XKCD joke here. :-) Once we have a better common metrics format, think SNMP but not crazy, "exporters" will be unnecessary.