5 ms·

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted

by bhuga 12y ago

It's great to see new entrants into the monitoring and graphing space. These are problems that every company has, and yet there's no solution as widely accepted for monitoring, notifications or graphing as nginx is for a web server. Not that I'd do a better job, but every time I further configure our monitoring system, I get that feeling that we're missing something as an industry. It's a space with lots of tools that feel too big or too small; only graphite feels like it's doing one job fairly well. Alerting is the worst of it. Nagios and all the other alerting solutions I've played with feel just a bit off. They're either doing too much or carve out a role at boundaries that aren't quite right. This results in other systems wanting to do alerting, making it tough to compare tools. As an example, Prometheus has an alert manager under development: https://github.com/prometheus/alertmanager https://github.com/prometheus/alertmanager. Why isn't doing a great job at graphing enough of a goal? Is it a problem with the alerting tools, or is it a problem with boundaries between alerting, graphing, and notifications?

bbrazil 12y ago
> This results in other systems wanting to do alerting, making it tough to compare tools. I'd think you've hit the nail on the head, every monitoring system has to do a bit of everything such as alerting, machine monitoring, and graphing; rather than focusing on just doing one thing and doing it well. Prometheus comes with a powerful data model and query language, few existing systems support the same (e.g. Atlas and Riemann have the notion of a query language and labels) so we have to do a bit of everything to produce a coherent system. I think a separate common alert manager would be good if we could combine, currently in practice that role isn't being done as you rely on de-duplication in a tool such as Pagerduty with support for silencing alerts not being unified.
jrv 12y ago
Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.
- bhuga 12y ago
  I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places. For example, in response to the latest security hullabaloo, we recently added a Nagios check for libc being too old, and for running processes that are linked to a libc older than the one on disk. This isn't a time series check; it's just an alert condition. Polling for that alert condition isn't what Prometheus is going to be good at, so cramming it into that alert pipeline is awkward. If we used something like Prometheus, we'd have rules in two places already. Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There's still some logic about what you're measuring in Prometheus, but not about what thresholds are alerts, or when, or for how long, or who cares. Threshold rules could all live in one place. This isn't a knock on Prometheus, which of course supports a workflow along those lines, and seems interesting. Nor is it a knock on the prometheus alert system. I just wonder why software in the monitoring/notification/alerting space always ends up muddy. I believe the overlapping boundaries between projects are a symptom, not a cause. I keep asking myself if there's some fundamental complexity that I'm not yet appreciating.
  - bbrazil 12y ago
    > This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue. A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case. > Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.
KyleBrandt 12y ago
Shameless plug. But Bosun's focus is largely on Alerting (http://bosun.org http://bosun.org). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results. It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo. Another nice thing about Bosun is you can test your alerts against time series history to see when they would have triggered so you can largely tune them before you commit them to production.
- jrv 12y ago
  Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time. Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is there any local storage component? I guess you could run OpenTSDB colocated on a single node...
  - KyleBrandt 12y ago
    OpenTSDB is our main backend. But we can also query graphite and logstash. However the graphing page doesn't work with Graphite.
    - jrv 12y ago
      Ah, ok. By the way, in case you make it to GopherCon this year, it would be interesting to exchange ideas! One of the things I'm happy about is that finally systems with multi-dimensional time series (based on OpenTSDB or not) are becoming more common...
      - KyleBrandt 12y ago
        I was there last year but not sure if I'm going this year yet. Matt is going though (http://mattjibson.com/ http://mattjibson.com/). I'd love to chat at some point though! I need to take a closer look at stealing ideas from your tool :-) We are both leveraging Go templates (Bosun uses it for Alert notifications, but I've thought about using it to create dashboards as well).

by bhuga 12y ago

bbrazil 12y ago

> This results in other systems wanting to do alerting, making it tough to compare tools. I'd think you've hit the nail on the head, every monitoring system has to do a bit of everything such as alerting, machine monitoring, and graphing; rather than focusing on just doing one thing and doing it well. Prometheus comes with a powerful data model and query language, few existing systems support the same (e.g. Atlas and Riemann have the notion of a query language and labels) so we have to do a bit of everything to produce a coherent system. I think a separate common alert manager would be good if we could combine, currently in practice that role isn't being done as you rely on de-duplication in a tool such as Pagerduty with support for silencing alerts not being unified.

jrv 12y ago

Brian already hinted at this, but the main reason why we do this is to have coherency of the data model across the whole chain: client app -> Prometheus server -> alerting rules -> alert handling This allows e.g. initial dimensional labels from the clients to cleanly propagate all the way into alerts, and allows you to silence, aggregate, or route alerts by any label combination. If OpenTSDB-style dimensional metrics would have been more of a standard already, maybe that would be different.

bhuga 12y ago
I haven't used Prometheus yet, but I'm not sure I agree that it makes sense to send dimensional time series all the way down the alert chain. It allows full-featured alerts to be used, yes. But you're not going to be able to put all of your alert logic into the prometheus chain, so you're going to end up having alert rules in at least two places. For example, in response to the latest security hullabaloo, we recently added a Nagios check for libc being too old, and for running processes that are linked to a libc older than the one on disk. This isn't a time series check; it's just an alert condition. Polling for that alert condition isn't what Prometheus is going to be good at, so cramming it into that alert pipeline is awkward. If we used something like Prometheus, we'd have rules in two places already. Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There's still some logic about what you're measuring in Prometheus, but not about what thresholds are alerts, or when, or for how long, or who cares. Threshold rules could all live in one place. This isn't a knock on Prometheus, which of course supports a workflow along those lines, and seems interesting. Nor is it a knock on the prometheus alert system. I just wonder why software in the monitoring/notification/alerting space always ends up muddy. I believe the overlapping boundaries between projects are a symptom, not a cause. I keep asking myself if there's some fundamental complexity that I'm not yet appreciating.
- bbrazil 12y ago
  > This isn't a time series check; it's just an alert condition. I think you're conflating instrumentation and alerting. Many systems only offer alert conditions on very low-level data that's tightly-bound to the instrumentation (e.g. a single check on a single host), Prometheus is more powerful as the instrumentation and alerting are separate, with dimensions adding further power. Prometheus has many instrumentation options. For your example I'd suggest a cronjob that outputs a file to the textfile node_exporter module (echo old_libc_process $COUNT > /configured/path/oldlibc.prom). You can then setup an alert for when the value is greater than 0 on any host (ALERT OldLibcAlert IF old_libc_process > 0 WITH ...). The advantage of having all the dimensions is that you can analyse that over time, and graph how many hosts had a problem to trend how well you're dealing with the libc issue. A big win for having dimensions for alerting all the way up the stack is in silencing, notification and throttling of related alerts. You can slice and dice and have things as granular as makes sense for your use case. > Polling prometheus for saved queries and simply alerting on a threshold might result in a simpler system. The rules that could be expressed in an alert pipeline could be expressed as different queries, and all rules about thresholds, alerting or notifications could be done further down the pipeline. There is an alert bridge to nagios if you want to send your alerts out that way. You'll lose some of the rich data that the dimensions provide though.

KyleBrandt 12y ago

Shameless plug. But Bosun's focus is largely on Alerting (http://bosun.org http://bosun.org). We have an expression language built in that allows for complex rules. It leverages OpenTSDBs multi dimensional facet to make alert instantiations, but you can also change the scope by transposing your results. It also now supports Logstash and Graphite as backends as well. The Graphite support is thanks to work at Vimeo. Another nice thing about Bosun is you can test your alerts against time series history to see when they would have triggered so you can largely tune them before you commit them to production.

jrv 12y ago
Great. Yeah, adding unit tests for alerts is something that we still need to add the capability for. But at least you can always already manually graph your alert expressions and see how they would have evaluated over time. Interesting work on Bosun by the way! Seems like there is quite some overlap with Prometheus, but I yet have to study it in depth. Is my impression correct that OpenTSDB is a requirement, or is there any local storage component? I guess you could run OpenTSDB colocated on a single node...
- KyleBrandt 12y ago
  OpenTSDB is our main backend. But we can also query graphite and logstash. However the graphing page doesn't work with Graphite.
  - jrv 12y ago
    Ah, ok. By the way, in case you make it to GopherCon this year, it would be interesting to exchange ideas! One of the things I'm happy about is that finally systems with multi-dimensional time series (based on OpenTSDB or not) are becoming more common...
    - KyleBrandt 12y ago
      I was there last year but not sure if I'm going this year yet. Matt is going though (http://mattjibson.com/ http://mattjibson.com/). I'd love to chat at some point though! I need to take a closer look at stealing ideas from your tool :-) We are both leveraging Go templates (Bosun uses it for Alert notifications, but I've thought about using it to create dashboards as well).