4 ms·

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you

by Rapzid 12y ago

Same here; need push for integration. It appears that Prometheus favours pull to a fault. To me it makes sense to have a push/message infrastructure that you can then write scrapers for to your hearts content. InfluxDB has push, but I read that it uses 12X the storage due to storing metadata with each metric; Yikes!

clarkevans 12y ago
What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.
- Rapzid 12y ago
  I push all my logs into logstash->elasticsearch. For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense. For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.
- bbrazil 12y ago
  https://github.com/rcrowley/go-metrics https://github.com/rcrowley/go-metrics is one I've came across while researching potential Prometheus integrations, it seems to support a number of backends including Graphite and InfluxDB. One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server. For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.
bbrazil 12y ago
We favour pull, but we don't force it. The collectd exporter is an integration with a push-based system for example. My full thoughts are up at http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring The short version is that I consider pull slightly better, but not majorly so. Push is more difficult to work with as you scale up, but it is doable.
jrv 12y ago
(Prometheus author here) Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting: https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9BuiqRNS3oA62i8fJbwwQ8/edit https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...
- rdsubhas 12y ago
  Hi, this definitely looks very cool, but how about cases where we have a bunch of instances running behind a load balancer, and each servers its own metrics? We can't pull them, because hitting the load balancer would randomly choose only one instance. Instances are scaled up based on load. So we can't specify the target instances in Prometheus because it keeps on changing. We'd like to try this out, but any ideas what to do for the above?
  - bbrazil 12y ago
    What you want to do is separately scrape each instance. We're working on service discovery support[1] so that you can dynamically change what hosts/ports Prometheus scrapes. Currently you can use DNS for service discovery, or change the config file and restart prometheus. [1]http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/
- pauldix 12y ago
  InfluxDB CEO here. The next version (0.9.0) has support for tags and efficiently encodes the measurement name + tagset as a single 4 byte uint. Previously, you would have had to encode metadata in the series name. Otherwise if you used string columns you'd see a massive waste in disk space since they were repeated on every measurement.
pauldix 12y ago
InfluxDB CEO here. With the current version of InfluxDB (0.8.8), metadata should be encoded in the name of the series, which means you don't repeat it. In the next version of InfluxDB (0.9.0), you can encode metadata as tags and it gets converted to a single id. With either of those schemes you should see much better numbers on storage.
- jrv 12y ago
  Hey Paul, this is exciting news! So perhaps InfluxDB could work well as a Prometheus long-term storage backend after all. It would be so much nicer than having to run OpenTSDB based on Hadoop/HBase. It might be a while until I find the time, but looking forward to giving it another try!

by Rapzid 12y ago

clarkevans 12y ago

What do you use for push-oriented time-series metrics? We are thinking of using LogStash (it has a go client). Unfortunately, as I understand, it's not time-series oriented.

Rapzid 12y ago
I push all my logs into logstash->elasticsearch. For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense. For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.
bbrazil 12y ago
https://github.com/rcrowley/go-metrics https://github.com/rcrowley/go-metrics is one I've came across while researching potential Prometheus integrations, it seems to support a number of backends including Graphite and InfluxDB. One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server. For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.

bbrazil 12y ago

We favour pull, but we don't force it. The collectd exporter is an integration with a push-based system for example. My full thoughts are up at http://www.boxever.com/push-vs-pull-for-monitoring http://www.boxever.com/push-vs-pull-for-monitoring The short version is that I consider pull slightly better, but not majorly so. Push is more difficult to work with as you scale up, but it is doable.

jrv 12y ago

(Prometheus author here) Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting: https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9BuiqRNS3oA62i8fJbwwQ8/edit https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...

rdsubhas 12y ago
Hi, this definitely looks very cool, but how about cases where we have a bunch of instances running behind a load balancer, and each servers its own metrics? We can't pull them, because hitting the load balancer would randomly choose only one instance. Instances are scaled up based on load. So we can't specify the target instances in Prometheus because it keeps on changing. We'd like to try this out, but any ideas what to do for the above?
- bbrazil 12y ago
  What you want to do is separately scrape each instance. We're working on service discovery support[1] so that you can dynamically change what hosts/ports Prometheus scrapes. Currently you can use DNS for service discovery, or change the config file and restart prometheus. [1]http://prometheus.io/docs/introduction/roadmap/ http://prometheus.io/docs/introduction/roadmap/
pauldix 12y ago
InfluxDB CEO here. The next version (0.9.0) has support for tags and efficiently encodes the measurement name + tagset as a single 4 byte uint. Previously, you would have had to encode metadata in the series name. Otherwise if you used string columns you'd see a massive waste in disk space since they were repeated on every measurement.

pauldix 12y ago

InfluxDB CEO here. With the current version of InfluxDB (0.8.8), metadata should be encoded in the name of the series, which means you don't repeat it. In the next version of InfluxDB (0.9.0), you can encode metadata as tags and it gets converted to a single id. With either of those schemes you should see much better numbers on storage.

jrv 12y ago
Hey Paul, this is exciting news! So perhaps InfluxDB could work well as a Prometheus long-term storage backend after all. It would be so much nicer than having to run OpenTSDB based on Hadoop/HBase. It might be a while until I find the time, but looking forward to giving it another try!