5 ms·

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't

We've been looking for something like this, unfortunately the "pull" model won't work for us. We really need a push model so that our statistics server doesn't need access to every single producer. I see the pushgateway, but it seems deliberately not a centralized storage. I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.

bbrazil 12y ago
(One of the Prometheus authors here) My understanding is that InfluxDB will work for your use case, as it's all push. > I see the pushgateway, but it seems deliberately not a centralized storage. Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs. May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.
- clarkevans 12y ago
  Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed". A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.
  - bbrazil 12y ago
    That'd be a challenge alright. One thing to consider is having each owner run their own Prometheus so that they can take advantage of all it's features, though it's a harder sell than "just push over here". http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring looks at other bits of the push vs. pull question.
  - hartror 12y ago
    This is a common problem that you describe.
Rapzid 12y ago
Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!
- clarkevans 12y ago
  What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.
  - Rapzid 12y ago
    I push all my logs into logstash->elasticsearch. For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense. For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.
  - bbrazil 12y ago
    https://github.com/rcrowley/go-metrics https://github.com/rcrowley/go-metrics is one I've came across while researching potential Prometheus integrations, it seems to support a number of backends including Graphite and InfluxDB. One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server. For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.
jrv 12y ago
(initial Prometheus author here) I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19 https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/.
paraboul 12y ago
what about druid.io?
JanezStupar 12y ago
Why do you need push replication? From my experience with distributed systems push replication will get you in trouble, very soon. Edit: I was too quick to post this question. An obvious scenario is where the client is behind a firewall. Never mind me. I am an idiot.
kordless 12y ago
Is the use case mostly about permissions and networking setup? I could see why a pull model wouldn't work in those cases.

bbrazil 12y ago

(One of the Prometheus authors here) My understanding is that InfluxDB will work for your use case, as it's all push. > I see the pushgateway, but it seems deliberately not a centralized storage. Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs. May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.

clarkevans 12y ago
Each node in our network has it's own access control permissions and organizational owner. It's easy enough to provide a centralized service with an end-point to push statistics from each node. In this case, organizational participation is "distributed". A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.
- bbrazil 12y ago
  That'd be a challenge alright. One thing to consider is having each owner run their own Prometheus so that they can take advantage of all it's features, though it's a harder sell than "just push over here". http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring looks at other bits of the push vs. pull question.
- hartror 12y ago
  This is a common problem that you describe.

Rapzid 12y ago

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

clarkevans 12y ago
What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.
- Rapzid 12y ago
  I push all my logs into logstash->elasticsearch. For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense. For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.
- bbrazil 12y ago
  https://github.com/rcrowley/go-metrics https://github.com/rcrowley/go-metrics is one I've came across while researching potential Prometheus integrations, it seems to support a number of backends including Graphite and InfluxDB. One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server. For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.

jrv 12y ago

(initial Prometheus author here) I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19 https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/.

paraboul 12y ago

what about druid.io?

JanezStupar 12y ago

Why do you need push replication? From my experience with distributed systems push replication will get you in trouble, very soon. Edit: I was too quick to post this question. An obvious scenario is where the client is behind a firewall. Never mind me. I am an idiot.

kordless 12y ago

Is the use case mostly about permissions and networking setup? I could see why a pull model wouldn't work in those cases.