4 ms·

> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditio

by bbrazil 12y ago

> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue. A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case. > Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.

bhuga 12y ago
> I think you're conflating instrumentation and alerting. A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though. > Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). Interesting. That architecture or configuration of this exporter isn't documented yet, but if just magically sent numbers upstream from text files that I keep up to date, that might be worth a migration by itself. Nagios SSH checks are ridiculous to manage. Thanks for indulging my conversation in any case. I'll put this on my radar and watch where it goes.
- bbrazil 12y ago
  > I'm hesitant to cram into a time series database data that I consider boolean. A boolean can be treated as a 1 or a 0. Sometimes you can even go richer than that, such as with a count of affected processes from your example - which you could convert back to a 1/0 if you wanted in the expression language. > That architecture or configuration of this exporter isn't documented yet The node_exporter as a whole still needs docs. That module is pretty new, and it's not exactly obvious what it does or why it exists. Labels/dimensions are supported too, it's the same text format that Prometheus uses elsewhere ( http://prometheus.io/docs/instrumenting/exposition_formats/ http://prometheus.io/docs/instrumenting/exposition_formats/) > if just magically sent numbers upstream from text files that I keep up to date Pretty much, you'll also get the rest of the module of the node exporter (cpu, ram, disk, network) and associated consoles which http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh... describes how to setup.

by bbrazil 12y ago

bhuga 12y ago

> I think you're conflating instrumentation and alerting. A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though. > Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). Interesting. That architecture or configuration of this exporter isn't documented yet, but if just magically sent numbers upstream from text files that I keep up to date, that might be worth a migration by itself. Nagios SSH checks are ridiculous to manage. Thanks for indulging my conversation in any case. I'll put this on my radar and watch where it goes.

bbrazil 12y ago
> I'm hesitant to cram into a time series database data that I consider boolean. A boolean can be treated as a 1 or a 0. Sometimes you can even go richer than that, such as with a count of affected processes from your example - which you could convert back to a 1/0 if you wanted in the expression language. > That architecture or configuration of this exporter isn't documented yet The node_exporter as a whole still needs docs. That module is pretty new, and it's not exactly obvious what it does or why it exists. Labels/dimensions are supported too, it's the same text format that Prometheus uses elsewhere ( http://prometheus.io/docs/instrumenting/exposition_formats/ http://prometheus.io/docs/instrumenting/exposition_formats/) > if just magically sent numbers upstream from text files that I keep up to date Pretty much, you'll also get the rest of the module of the node exporter (cpu, ram, disk, network) and associated consoles which http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh... describes how to setup.