3 ms·

I like Riemann and actually read The Art of Monitoring (https://artofmonitoring.com/ https://artofmonitoring.com/) which was a great book. There are two big do

I like Riemann and actually read The Art of Monitoring (https://artofmonitoring.com/ https://artofmonitoring.com/) which was a great book. There are two big downsides for me on this one. First, you MUST build your monitoring solution from scratch and you MUST learn Clojure (which can be hard to get a whole team to agree to). Second, there's no alerting dashboard, which makes it difficult to see the overall state of the clusters you're dealing with. The only way you know there's a problem if you get an email. Maybe there's a better way to handle that but I wasn't able to find one.

jamtur01 10y ago
You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9). I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console. Glad you enjoyed the book!
bbrazil 10y ago
> you MUST learn Clojure (which can be hard to get a whole team to agree to) For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it. I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.
- mspaulding06 10y ago
  It's a major issue for me because of the time investment required to learn the language (functional programming isn't exactly second nature to most programmers) and the need to design the monitoring system from scratch. Instead of an out-of-the-box system you're getting more of a framework which allows you to build one. I actually like Clojure and agree that it is readable as well as fun to learn. But I think with something like Prometheus you do more simple configuration than monitoring system design.

jamtur01 10y ago

You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9). I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console. Glad you enjoyed the book!

bbrazil 10y ago

> you MUST learn Clojure (which can be hard to get a whole team to agree to) For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it. I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.

mspaulding06 10y ago
It's a major issue for me because of the time investment required to learn the language (functional programming isn't exactly second nature to most programmers) and the need to design the monitoring system from scratch. Instead of an out-of-the-box system you're getting more of a framework which allows you to build one. I actually like Clojure and agree that it is readable as well as fun to learn. But I think with something like Prometheus you do more simple configuration than monitoring system design.