6 ms·

Why Big Monitoring Software Sucks

65 points by craigkerstiens 14y ago

tpsreport 14y ago
Nice article, whose central message ("don't bundle your tools") is widely applicable to domains other than monitoring. Small, specialized tools that can be combined together is the very essence of the Unix philosophy.
- dasil003 14y ago
  The problem is particularly acute in monitoring because you tend to need to monitor many disparate systems and subsystems in a very fine-grained manner. It needs to be reliable and not impact performance no matter where it is run which usually has to be everywhere. The Unix philosophy is definitely useful on a wider basis, but monitoring could be the poster child.
sigre 14y ago
So, what would a "small, sharp" monitoring tool look like that's compelling to customers? Just an agent, and the customer supplies their own notification and trend-reporting services?
- obfuscurity_ 14y ago
  I think Pingdom applies to some extent. In particular because it allows you to get the raw data out which can then by imported into trending software or notification services elsewhere.
- sciurus 14y ago
  For example, http://www.control-alt-del.org/2012/03/28/collectd-esper-amqp-opentsdbgraphite-oh-my/ http://www.control-alt-del.org/2012/03/28/collectd-esper-amq...
ghshephard 14y ago
To some degree, the best "monitoring software" doesn't even monitor at all - it simply provides a framework for people to add their monitors to. To take one example I'm familiar with (though I'm sure this could be applied to the rest of the great monitoring systems) - Nagios is, at it's heart, a State-Tracking/Notification/Scheduling Engine. The fact that you can add commands like "ping" or "http test" or "Database Schema Verification" to the tasks that you are scheduling and tracking the state on, is almost incidental to the core of what Nagios does.
- spudlyo 14y ago
  I agree completely, and I think Nagios does a good job at state tracking, notification, and scheduling. The interface is a bit clunky, but with the nuvola makeover it's completely serviceable. Pair it with something like ganglia or cacti for visualization and you're most of the way there.
  - LaSombra 14y ago
    Take a look at Opsview. It runs on top of Nagios 3 and makes you not edit those pesky configuration files and integrates with MRTG. Pretty neat.
josephruscio 14y ago
Could not agree more. This is the approach we've strived to take while building https://metrics.librato.com https://metrics.librato.com ... API access for everything, integration with popular OSS for metric collection (e.g. statsd, collectd), loosely coupled integration with complementary tools (e.g. Papertrail, Pagerduty), etc.
- obfuscurity_ 14y ago
  @josephruscio - I was >this< close to going back and giving a quick mention to Librato Metrics. You guys definitely "get" what I'm talking about. I like that you provide well-defined interfaces for easily getting data into and out of your application. You focus on trending and let other (better?) software handle the other stuff.
antoncohen 14y ago
Sounds like Sensu (https://github.com/sensu/sensu https://github.com/sensu/sensu) fits the requirements. It's a small monitoring framework that works in conjunction with Chef or Puppet, PagerDuty, Librato, Graphite, and others.
- sciurus 14y ago
  Sensu came up several times in the recent monitoring discussion on devops-toolchain [0]. I'm going to try a setup with sensu, graphite, and collectd soon- like the one Sean Escriva described at ChefConf [1]. [0] https://groups.google.com/group/devops-toolchain/browse_thread/thread/e5dac7a03dfb4fbf https://groups.google.com/group/devops-toolchain/browse_thre... [1] http://www.youtube.com/watch?v=BXxtdE-Paco http://www.youtube.com/watch?v=BXxtdE-Paco
shortlived 14y ago
Yeah but does it scale? Edit: I should clarify, I really liked the article and I like the unix philosophy but I'm truly interested to know what type of networks can be monitored with this approach. I work in the enterprise monitoring space and a lot of the solutions are truly horrible, but on the other hand, I'm not sure this approach would work either when you have 500K managed nodes and dealing with multiple campuses across 3 continents...
- obfuscurity_ 14y ago
  Yes, it certainly /can/ scale. In fact, a properly modularized monitoring solution /should/ be more capable of scaling if it was designed with this in mind. This is certainly the approach we've taken with our monitoring and trending components at Heroku. We don't have anywhere near 500K nodes, but the principles scale.
beedogs 14y ago
I swear I've seen HP OpenView processes using more CPU than Oracle. I have no idea why companies throw away so much money on such terrible software.
- spudlyo 14y ago
  Monitoring systems that have to execute thousands of active checks every polling interval consume a bit of CPU. I've certainly seen high CPU usage and run queue depth on a busy Nagios server.
  - beedogs 14y ago
    On a central server, I can understand. These processes were running on clients, though, and consuming as much memory and CPU as they could find.
suprgeek 14y ago
It is easy to beat up on big complicated monitoring software - call it Enterprise Level and then proceed to find flaws. There is a reason it is complicated - because it tries to solve Tough problems: Can these other offerings give me: 1) High-availability with minimal downtime (seconds) 2) I18N readiness & L10n language packs 3) Scale on the order of 10000 or 100K 4) Guaranteed delivery of Alerts and Metrics 5) Easy Deployment and configuration - where the first step is NOT download and deploy Redis and configure it. Make no mistake I love open source offerings but having worked in this domain for a while, Enterprise software becomes complicated to solve complicated problems with minimal intervention by users - not every IT shop has super-duper DevOps ninjas who Crunch Machine Learning Pardigms for breakfast.
- Morg 14y ago
  1) any replicated solution does that below a second, I've personally built stuff like that for fun 2) WTF ARE YOU INSANE ? monitoring isn't about displaying foreign characters, it's done in integers and english strings, like all programming. You will use user-related mess only in the user-related part 3) Every database does that with ease, if you talk about scaling from 100k monitored nodes, that only requires an aggregation layer, and that's still easy 4) That's a commercial "guaranteed", noone can do that with standard networks and machines 5) Indeed, but you'll need consultants anyway so shifting part of the costs to man days instead of licensing is more or less equal. Now in reality, most products that are known actually suck (like how mysql is a "standard" although it's a total fail) and enterprisy solutions only have a little software packaging to make them look smooth - and in the case of monitoring, I haven't seen a sensible solution yet, with a complete round robin system, logical rather than string events, distributed 1st level pre-processing, etc. I'll build one if I need one - right now I'm busy on something else.
- gouranga 14y ago
  Excuse the cynical POV here, but it's from a number of years of dealing with such things. The problems I find with enterprise software as a rule are it solves the above but fails at: 1) Realistic scaling. Just about every piece of enterprise grade software I've seen falls off a cliff when it hits a certain load point. This is ALWAYS enough to sell it to you. When it goes wrong... 2) When it goes wrong, it's a nightmare of maintenance contracts, verification, finger pointing and telephone calls that last hours. 3) It rarely works as advertised. I mean literally 1% works as advertised. 4) Trials and realistic evaluations without crazy constraints are not usually possible (this is changing slowly). 5) Installation is a breeze but when it comes to backup/restore and upgrade, it's a pain in the butt. Time is money. The difference between "enterprise level software" and "OSS" platforms are the following: * OSS time is spent up front. * Enterprise time is spent later on and costs more up front. I'd rather have one Ninja on the team and lose the enterprise software. Ninjas scale better as well.

Nice article, whose central message ("don't bundle your tools") is widely applicable to domains other than monitoring. Small, specialized tools that can be combined together is the very essence of the Unix philosophy.

dasil003 14y ago
The problem is particularly acute in monitoring because you tend to need to monitor many disparate systems and subsystems in a very fine-grained manner. It needs to be reliable and not impact performance no matter where it is run which usually has to be everywhere. The Unix philosophy is definitely useful on a wider basis, but monitoring could be the poster child.

sigre 14y ago

So, what would a "small, sharp" monitoring tool look like that's compelling to customers? Just an agent, and the customer supplies their own notification and trend-reporting services?

obfuscurity_ 14y ago
I think Pingdom applies to some extent. In particular because it allows you to get the raw data out which can then by imported into trending software or notification services elsewhere.
sciurus 14y ago
For example, http://www.control-alt-del.org/2012/03/28/collectd-esper-amqp-opentsdbgraphite-oh-my/ http://www.control-alt-del.org/2012/03/28/collectd-esper-amq...

ghshephard 14y ago

To some degree, the best "monitoring software" doesn't even monitor at all - it simply provides a framework for people to add their monitors to. To take one example I'm familiar with (though I'm sure this could be applied to the rest of the great monitoring systems) - Nagios is, at it's heart, a State-Tracking/Notification/Scheduling Engine. The fact that you can add commands like "ping" or "http test" or "Database Schema Verification" to the tasks that you are scheduling and tracking the state on, is almost incidental to the core of what Nagios does.

spudlyo 14y ago
I agree completely, and I think Nagios does a good job at state tracking, notification, and scheduling. The interface is a bit clunky, but with the nuvola makeover it's completely serviceable. Pair it with something like ganglia or cacti for visualization and you're most of the way there.
- LaSombra 14y ago
  Take a look at Opsview. It runs on top of Nagios 3 and makes you not edit those pesky configuration files and integrates with MRTG. Pretty neat.

josephruscio 14y ago

Could not agree more. This is the approach we've strived to take while building https://metrics.librato.com https://metrics.librato.com ... API access for everything, integration with popular OSS for metric collection (e.g. statsd, collectd), loosely coupled integration with complementary tools (e.g. Papertrail, Pagerduty), etc.

obfuscurity_ 14y ago
@josephruscio - I was >this< close to going back and giving a quick mention to Librato Metrics. You guys definitely "get" what I'm talking about. I like that you provide well-defined interfaces for easily getting data into and out of your application. You focus on trending and let other (better?) software handle the other stuff.

antoncohen 14y ago

Sounds like Sensu (https://github.com/sensu/sensu https://github.com/sensu/sensu) fits the requirements. It's a small monitoring framework that works in conjunction with Chef or Puppet, PagerDuty, Librato, Graphite, and others.

sciurus 14y ago
Sensu came up several times in the recent monitoring discussion on devops-toolchain [0]. I'm going to try a setup with sensu, graphite, and collectd soon- like the one Sean Escriva described at ChefConf [1]. [0] https://groups.google.com/group/devops-toolchain/browse_thread/thread/e5dac7a03dfb4fbf https://groups.google.com/group/devops-toolchain/browse_thre... [1] http://www.youtube.com/watch?v=BXxtdE-Paco http://www.youtube.com/watch?v=BXxtdE-Paco

shortlived 14y ago

Yeah but does it scale? Edit: I should clarify, I really liked the article and I like the unix philosophy but I'm truly interested to know what type of networks can be monitored with this approach. I work in the enterprise monitoring space and a lot of the solutions are truly horrible, but on the other hand, I'm not sure this approach would work either when you have 500K managed nodes and dealing with multiple campuses across 3 continents...

obfuscurity_ 14y ago
Yes, it certainly /can/ scale. In fact, a properly modularized monitoring solution /should/ be more capable of scaling if it was designed with this in mind. This is certainly the approach we've taken with our monitoring and trending components at Heroku. We don't have anywhere near 500K nodes, but the principles scale.

beedogs 14y ago

I swear I've seen HP OpenView processes using more CPU than Oracle. I have no idea why companies throw away so much money on such terrible software.

spudlyo 14y ago
Monitoring systems that have to execute thousands of active checks every polling interval consume a bit of CPU. I've certainly seen high CPU usage and run queue depth on a busy Nagios server.
- beedogs 14y ago
  On a central server, I can understand. These processes were running on clients, though, and consuming as much memory and CPU as they could find.

suprgeek 14y ago

It is easy to beat up on big complicated monitoring software - call it Enterprise Level and then proceed to find flaws. There is a reason it is complicated - because it tries to solve Tough problems: Can these other offerings give me: 1) High-availability with minimal downtime (seconds) 2) I18N readiness & L10n language packs 3) Scale on the order of 10000 or 100K 4) Guaranteed delivery of Alerts and Metrics 5) Easy Deployment and configuration - where the first step is NOT download and deploy Redis and configure it. Make no mistake I love open source offerings but having worked in this domain for a while, Enterprise software becomes complicated to solve complicated problems with minimal intervention by users - not every IT shop has super-duper DevOps ninjas who Crunch Machine Learning Pardigms for breakfast.

Morg 14y ago
1) any replicated solution does that below a second, I've personally built stuff like that for fun 2) WTF ARE YOU INSANE ? monitoring isn't about displaying foreign characters, it's done in integers and english strings, like all programming. You will use user-related mess only in the user-related part 3) Every database does that with ease, if you talk about scaling from 100k monitored nodes, that only requires an aggregation layer, and that's still easy 4) That's a commercial "guaranteed", noone can do that with standard networks and machines 5) Indeed, but you'll need consultants anyway so shifting part of the costs to man days instead of licensing is more or less equal. Now in reality, most products that are known actually suck (like how mysql is a "standard" although it's a total fail) and enterprisy solutions only have a little software packaging to make them look smooth - and in the case of monitoring, I haven't seen a sensible solution yet, with a complete round robin system, logical rather than string events, distributed 1st level pre-processing, etc. I'll build one if I need one - right now I'm busy on something else.
gouranga 14y ago
Excuse the cynical POV here, but it's from a number of years of dealing with such things. The problems I find with enterprise software as a rule are it solves the above but fails at: 1) Realistic scaling. Just about every piece of enterprise grade software I've seen falls off a cliff when it hits a certain load point. This is ALWAYS enough to sell it to you. When it goes wrong... 2) When it goes wrong, it's a nightmare of maintenance contracts, verification, finger pointing and telephone calls that last hours. 3) It rarely works as advertised. I mean literally 1% works as advertised. 4) Trials and realistic evaluations without crazy constraints are not usually possible (this is changing slowly). 5) Installation is a breeze but when it comes to backup/restore and upgrade, it's a pain in the butt. Time is money. The difference between "enterprise level software" and "OSS" platforms are the following: * OSS time is spent up front. * Enterprise time is spent later on and costs more up front. I'd rather have one Ninja on the team and lose the enterprise software. Ninjas scale better as well.