3 ms·

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-fea

by bhuga 12y ago

I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places. For example, in response to the latest security hullabaloo, we recently added a Nagios check for libc being too old, and for running processes that are linked to a libc older than the one on disk. This isn't a time series check; it's just an alert condition. Polling for that alert condition isn't what Prometheus is going to be good at, so cramming it into that alert pipeline is awkward. If we used something like Prometheus, we'd have rules in two places already. Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There's still some logic about what you're measuring in Prometheus, but not about what thresholds are alerts, or when, or for how long, or who cares. Threshold rules could all live in one place. This isn't a knock on Prometheus, which of course supports a workflow along those lines, and seems interesting. Nor is it a knock on the prometheus alert system. I just wonder why software in the monitoring/notification/alerting space always ends up muddy. I believe the overlapping boundaries between projects are a symptom, not a cause. I keep asking myself if there's some fundamental complexity that I'm not yet appreciating.

bbrazil 12y ago
> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue. A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case. > Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.
- bhuga 12y ago
  > I think you're conflating instrumentation and alerting. A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though. > Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). Interesting. That architecture or configuration of this exporter isn't documented yet, but if just magically sent numbers upstream from text files that I keep up to date, that might be worth a migration by itself. Nagios SSH checks are ridiculous to manage. Thanks for indulging my conversation in any case. I'll put this on my radar and watch where it goes.
  - bbrazil 12y ago
    > I'm hesitant to cram into a time series database data that I consider boolean. A boolean can be treated as a 1 or a 0. Sometimes you can even go richer than that, such as with a count of affected processes from your example - which you could convert back to a 1/0 if you wanted in the expression language. > That architecture or configuration of this exporter isn't documented yet The node_exporter as a whole still needs docs. That module is pretty new, and it's not exactly obvious what it does or why it exists. Labels/dimensions are supported too, it's the same text format that Prometheus uses elsewhere ( http://prometheus.io/docs/instrumenting/exposition_formats/ http://prometheus.io/docs/instrumenting/exposition_formats/) > if just magically sent numbers upstream from text files that I keep up to date Pretty much, you'll also get the rest of the module of the node exporter (cpu, ram, disk, network) and associated consoles which http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh... describes how to setup.

by bhuga 12y ago

bbrazil 12y ago

> This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue. A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case. > Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.

bhuga 12y ago
> I think you're conflating instrumentation and alerting. A bit, because I'm hesitant to cram into a time series database data that I consider boolean. I'd move past this for consistency if the checks were easy to create, though. > Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). Interesting. That architecture or configuration of this exporter isn't documented yet, but if just magically sent numbers upstream from text files that I keep up to date, that might be worth a migration by itself. Nagios SSH checks are ridiculous to manage. Thanks for indulging my conversation in any case. I'll put this on my radar and watch where it goes.
- bbrazil 12y ago
  > I'm hesitant to cram into a time series database data that I consider boolean. A boolean can be treated as a 1 or a 0. Sometimes you can even go richer than that, such as with a count of affected processes from your example - which you could convert back to a 1/0 if you wanted in the expression language. > That architecture or configuration of this exporter isn't documented yet The node_exporter as a whole still needs docs. That module is pretty new, and it's not exactly obvious what it does or why it exists. Labels/dimensions are supported too, it's the same text format that Prometheus uses elsewhere ( http://prometheus.io/docs/instrumenting/exposition_formats/ http://prometheus.io/docs/instrumenting/exposition_formats/) > if just magically sent numbers upstream from text files that I keep up to date Pretty much, you'll also get the rest of the module of the node exporter (cpu, ram, disk, network) and associated consoles which http://www.boxever.com/monitoring-your-machines-with-prometheus http://www.boxever.com/monitoring-your-machines-with-prometh... describes how to setup.