5 ms·

Prometheus.io which is a modern fresh monitoring system that I would checkout if replacing a legacy system. Also take a look at Riemann which is system monitor

by acd 10y ago

Prometheus.io which is a modern fresh monitoring system that I would checkout if replacing a legacy system. Also take a look at Riemann which is system monitoring written in Clojure. Riemann should be good for monitoring latency of the system. If it helps here is Slidedeck from Spotify how they do their monitoring https://www.netways.de/fileadmin/images/Events_Trainings/Events/OSMC/2015/Slides_2015/Monitoring_at_Spotify_When_things_go_ping_in_the_night-Martin_Parm.pdf https://www.netways.de/fileadmin/images/Events_Trainings/Eve...

illicium 10y ago
Riemann is a generic event processor. You can use it to generate alerts or aggregate metrics, but you still need something like collectd or telegraf to collect system/app stats from each machine and send it to Riemann.
- jpitz 10y ago
  Riemann comes with collection clients as well
tex0 10y ago
I can only second Prometheus. It's modeled after Google Borgmon (and possibly Monarch) and excels in any flexible workload. Really great piece of engineering.
solipsism 10y ago
It should be noted that Riemann is different from Nagios and Prometheus in that it's push-based. The other two are primarily pull-based.
- mspaulding06 10y ago
  Yep, I think that is Riemann's greatest advantage. I don't like the fact that with Prometheus I have to set up some sort of service discovery mechanism. Right now we're using Foreman to manage our clusters, so most likely I would have to use the file service discovery type and query the Foreman API to produce a YAML or JSON file to provide a list of static hosts.
  - jrv 10y ago
    If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?
    - mspaulding06 10y ago
      Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.
      - bbrazil 10y ago
        How does this tell you that something is missing if it never reported, or that something is no longer meant to exist?
        mspaulding06 10y ago
        It's true, you can't tell if something is missing if it never reported. I guess it depends on your requirements. In a situation where your workloads are ephemeral you may not care as much if all of your services have reported as long as most have. In the case that something should no longer exist you could write that functionality into your Riemann configuration. As an example, I could write a "dead service" stream processor that is used to inform Riemann that a service should no longer exist. When a "dead service" event is injected into the event stream for a particular service, Riemann will note the dead service and ignore future events from it.
mspaulding06 10y ago
I like Riemann and actually read The Art of Monitoring (https://artofmonitoring.com/ https://artofmonitoring.com/) which was a great book. There are two big downsides for me on this one. First, you MUST build your monitoring solution from scratch and you MUST learn Clojure (which can be hard to get a whole team to agree to). Second, there's no alerting dashboard, which makes it difficult to see the overall state of the clusters you're dealing with. The only way you know there's a problem if you get an email. Maybe there's a better way to handle that but I wasn't able to find one.
- jamtur01 10y ago
  You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9). I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console. Glad you enjoyed the book!
- bbrazil 10y ago
  > you MUST learn Clojure (which can be hard to get a whole team to agree to) For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it. I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.
  - mspaulding06 10y ago
    It's a major issue for me because of the time investment required to learn the language (functional programming isn't exactly second nature to most programmers) and the need to design the monitoring system from scratch. Instead of an out-of-the-box system you're getting more of a framework which allows you to build one. I actually like Clojure and agree that it is readable as well as fun to learn. But I think with something like Prometheus you do more simple configuration than monitoring system design.

by acd 10y ago

illicium 10y ago

Riemann is a generic event processor. You can use it to generate alerts or aggregate metrics, but you still need something like collectd or telegraf to collect system/app stats from each machine and send it to Riemann.

jpitz 10y ago
Riemann comes with collection clients as well

tex0 10y ago

I can only second Prometheus. It's modeled after Google Borgmon (and possibly Monarch) and excels in any flexible workload. Really great piece of engineering.

solipsism 10y ago

It should be noted that Riemann is different from Nagios and Prometheus in that it's push-based. The other two are primarily pull-based.

mspaulding06 10y ago
Yep, I think that is Riemann's greatest advantage. I don't like the fact that with Prometheus I have to set up some sort of service discovery mechanism. Right now we're using Foreman to manage our clusters, so most likely I would have to use the file service discovery type and query the Foreman API to produce a YAML or JSON file to provide a list of static hosts.
- jrv 10y ago
  If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?
  - mspaulding06 10y ago
    Riemann injects expire events into the event stream when it doesn't hear from a service for a while. Each event has a TTL that it uses to figure this out. Then of course you can alert on the expired events or take other actions.
    - bbrazil 10y ago
      How does this tell you that something is missing if it never reported, or that something is no longer meant to exist?
      - mspaulding06 10y ago
        It's true, you can't tell if something is missing if it never reported. I guess it depends on your requirements. In a situation where your workloads are ephemeral you may not care as much if all of your services have reported as long as most have. In the case that something should no longer exist you could write that functionality into your Riemann configuration. As an example, I could write a "dead service" stream processor that is used to inform Riemann that a service should no longer exist. When a "dead service" event is injected into the event stream for a particular service, Riemann will note the dead service and ignore future events from it.

mspaulding06 10y ago

I like Riemann and actually read The Art of Monitoring (https://artofmonitoring.com/ https://artofmonitoring.com/) which was a great book. There are two big downsides for me on this one. First, you MUST build your monitoring solution from scratch and you MUST learn Clojure (which can be hard to get a whole team to agree to). Second, there's no alerting dashboard, which makes it difficult to see the overall state of the clusters you're dealing with. The only way you know there's a problem if you get an email. Maybe there's a better way to handle that but I wasn't able to find one.

jamtur01 10y ago
You can alert via a number of mechanisms: email, PagerDuty, Slack et al, etc (I talk about most of those in Chapter 9). I have never been very keen on alerting dashboards, I find they are rarely actually reviewed and flash red for days or weeks. :) So I only covered metrics/graphing as a console rather than a status console. If you want to add such a console it'd be easy to output Riemann events via an API to such a console. Glad you enjoyed the book!
bbrazil 10y ago
> you MUST learn Clojure (which can be hard to get a whole team to agree to) For any sufficiently advanced monitoring system, you're going to have to learn some form of DSL/language to take advantage of it. I wouldn't consider this to be a major issue, more part of the irreducible complexity of the problem. The Riemann examples I've seen all seemed pretty readable, and routing alerts is not world away from what you'd be doing in say a Prometheus Alertmanager config; just with S-Expressions against YAML.
- mspaulding06 10y ago
  It's a major issue for me because of the time investment required to learn the language (functional programming isn't exactly second nature to most programmers) and the need to design the monitoring system from scratch. Instead of an out-of-the-box system you're getting more of a framework which allows you to build one. I actually like Clojure and agree that it is readable as well as fun to learn. But I think with something like Prometheus you do more simple configuration than monitoring system design.