3 ms·

I've spent 3 years making Nagios 'work'. I've had to add all the things Nagios didn't have, like inventory management, graphing, combining graphs, trend detect

by kokey 13y ago

I've spent 3 years making Nagios 'work'. I've had to add all the things Nagios didn't have, like inventory management, graphing, combining graphs, trend detection, service discovery, etc. I also had to make it scale. This also involved writing plugins in C to conserve memory since plugs are run through forking processes. The 'community' plugins are mostly really terrible. Scaling it also running things outside of the Nagios scheduler with another script and feeding the results back to Nagios. This allowed me to collect data for 120k parameters in 5 minutes from a single monitoring server. Anyway, this is just a long way of saying writing your own thing is probably not that bad an idea compared to having to learn Nagios and then trying to hack it into the shape of an enterprise monitoring system.

falcolas 13y ago
Sorry, inventory management? Graphing? Service discovery? Those don't strike me as fitting into the monitoring role. It seems to me like you spent 3 years bloating Nagios into a multi-role service that it was never originally designed to fulfill, as opposed to setting up 3-4 different services which specialize in what you're looking for (which took us significantly less than 3 years). Monitoring is looking at the services and making sure they're running within tolerances. Graphing is not useful for monitoring, its useful for postmortems and planning. Inventory management is not useful for monitoring, it's useful for building out servers. Service discovery has nothing to do with monitoring, though it's certainly be useful for automatically populating your monitoring configuration. However, trend detection, I agree; nagios needs more of that. I should definitely get alerts when my disk space usage suddenly starts growing 4x its normal amount, or when my CPU usage spikes to 3x its normal usage.
- deathcakes 13y ago
  > Graphing is not useful for monitoring, its useful for postmortems and planning. I'm sorry, I have to completely disagree with this point. Graphing is a fantastic way of visualising information to show trends, such as disk space growth, or memory usage, which can and have, in our company's case, led to diagnosis of problems that hadn't yet manifested in a complete crash. I realise you add in a comment saying that Nagios should be doing these things internally, which it also should, but what about situations you haven't planned for? Or intermittent events that don't look like anything individually, but when graphed over 6 months turn out to reveal the subtle failure of an aircon unit in a server room?
  - falcolas 13y ago
    My apologies if I wasn't clear. My intention was to say, with regards to graphing: Graphing should be part of your overall tooling for monitoring boxes, but it should not be part of process monitoring. It should be a separate component (which should in turn be monitored by Nagios), for use in post-mortems and future planning. It should not be built into Nagios, since the graph itself is completely irrelevent to whether Nagios should alert about a problem. I also think that graphing is different from heueristic rate monitoring; graphs are for humans to spot trends/correlations. Alerting on rate changes is a binary alert/don't alert. I.E. Cacti/Graphite should be creating graphs about services for human consumption, Nagios should be watching for problems with services.
- mlieberman85 13y ago
  If Nagios was a lightweight monitoring solution I would agree with you. Nagios however is a large monolithic system that is hard to integrate with third party metrics solutions, inventory management, etc. Though something like inventory management isn't part of monitoring it is still part of your whole system and if your monitoring solution can't easily integrate with your inventory management solution it gets frustrating. I also tend to think service discovery or at least a flexible API that allows other systems to create new checks is a huge part of monitoring. Nagios' configuration files are a nightmare to deal with especially in a configuration managed environment.
  - falcolas 13y ago
    I think you're looking at it backwards. Nagios shouldn't integrate with your inventory management; inventory management should be configuring Nagios. Chef, Puppet and Ansible all have modules for idempotently configuring services in Nagios, making it simple to have your nagios configuration be part of your inventory management solution. Ideally, this also resolves your configuration management problem as well. If it does not, there are tools which can probe a running Nagios state file, and provide you feedback on what's being monitored, and its state. The python library nagparser is one that I've used in the past to probe Nagios status for a status aggregation tool.
jamespo 13y ago
Running scripts outside the nagios scheduler is straightforward with passive checks. Also nagios 4 has changed the way forking checks work - http://labs.nagios.com/2013/09/20/nagios-core-4-now-available/ http://labs.nagios.com/2013/09/20/nagios-core-4-now-availabl... - I appreciate this was too late for you.

by kokey 13y ago

falcolas 13y ago

Sorry, inventory management? Graphing? Service discovery? Those don't strike me as fitting into the monitoring role. It seems to me like you spent 3 years bloating Nagios into a multi-role service that it was never originally designed to fulfill, as opposed to setting up 3-4 different services which specialize in what you're looking for (which took us significantly less than 3 years). Monitoring is looking at the services and making sure they're running within tolerances. Graphing is not useful for monitoring, its useful for postmortems and planning. Inventory management is not useful for monitoring, it's useful for building out servers. Service discovery has nothing to do with monitoring, though it's certainly be useful for automatically populating your monitoring configuration. However, trend detection, I agree; nagios needs more of that. I should definitely get alerts when my disk space usage suddenly starts growing 4x its normal amount, or when my CPU usage spikes to 3x its normal usage.

deathcakes 13y ago
> Graphing is not useful for monitoring, its useful for postmortems and planning. I'm sorry, I have to completely disagree with this point. Graphing is a fantastic way of visualising information to show trends, such as disk space growth, or memory usage, which can and have, in our company's case, led to diagnosis of problems that hadn't yet manifested in a complete crash. I realise you add in a comment saying that Nagios should be doing these things internally, which it also should, but what about situations you haven't planned for? Or intermittent events that don't look like anything individually, but when graphed over 6 months turn out to reveal the subtle failure of an aircon unit in a server room?
- falcolas 13y ago
  My apologies if I wasn't clear. My intention was to say, with regards to graphing: Graphing should be part of your overall tooling for monitoring boxes, but it should not be part of process monitoring. It should be a separate component (which should in turn be monitored by Nagios), for use in post-mortems and future planning. It should not be built into Nagios, since the graph itself is completely irrelevent to whether Nagios should alert about a problem. I also think that graphing is different from heueristic rate monitoring; graphs are for humans to spot trends/correlations. Alerting on rate changes is a binary alert/don't alert. I.E. Cacti/Graphite should be creating graphs about services for human consumption, Nagios should be watching for problems with services.
mlieberman85 13y ago
If Nagios was a lightweight monitoring solution I would agree with you. Nagios however is a large monolithic system that is hard to integrate with third party metrics solutions, inventory management, etc. Though something like inventory management isn't part of monitoring it is still part of your whole system and if your monitoring solution can't easily integrate with your inventory management solution it gets frustrating. I also tend to think service discovery or at least a flexible API that allows other systems to create new checks is a huge part of monitoring. Nagios' configuration files are a nightmare to deal with especially in a configuration managed environment.
- falcolas 13y ago
  I think you're looking at it backwards. Nagios shouldn't integrate with your inventory management; inventory management should be configuring Nagios. Chef, Puppet and Ansible all have modules for idempotently configuring services in Nagios, making it simple to have your nagios configuration be part of your inventory management solution. Ideally, this also resolves your configuration management problem as well. If it does not, there are tools which can probe a running Nagios state file, and provide you feedback on what's being monitored, and its state. The python library nagparser is one that I've used in the past to probe Nagios status for a status aggregation tool.

jamespo 13y ago

Running scripts outside the nagios scheduler is straightforward with passive checks. Also nagios 4 has changed the way forking checks work - http://labs.nagios.com/2013/09/20/nagios-core-4-now-available/ http://labs.nagios.com/2013/09/20/nagios-core-4-now-availabl... - I appreciate this was too late for you.