3 ms·

Had you tried Nagios and Puppet/Chef when you started developing these? The logging tool is the only one I'm not sure about, if you're parsing 1200 events a se

Had you tried Nagios and Puppet/Chef when you started developing these? The logging tool is the only one I'm not sure about, if you're parsing 1200 events a second. I'm not familiar enough with Orion to understand why it was insufficient. Would you say these in-house tools are primarily about integrating / presenting the data, or are there custom parts doing more heavy lifting too?

KyleBrandt 13y ago
We use Puppet a lot and are generally quite happy. We have recently started leveraging Hiera which is addressing our main issue we had with Puppet: Reusability problems in the code since Modules (code) and our data were not easily separated. A lot of these tools are about integrating / presenting but that isn't entirely the case. Status does its own polling of things like Redis and SQL. Realog parses the data, structures it in Redis etc, and the patching dashboard can kick of updates.
- dvanduzer 13y ago
  Status handles the scheduling of the polling, or the actual polling, or both? One thing Nagios handles well, although it isn't exactly well polished, is distribution, scheduling, and aggregation of the polling. Also, anecdotally, "nobody" seems to know how easy it is to do simple-intermediate monitoring of MS-SQL in Nagios. I happen to be using this, at the moment: https://github.com/scot0357/check_mssql_collection https://github.com/scot0357/check_mssql_collection edit: And thank you for the concise explanation of Hiera. I had entirely ignored it as just another add-on as Puppet "goes enterprise"
stevenjmurawski 13y ago
We tried Nagios a while back, before going to Orion. Nagios has some great structural ideas, but fell down in other areas (don't know all of them, that was before I came on board). We use Puppet to manage our Linux infrastructure and are testing Desired State Configuration for our Windows systems. Puppet and DSC solve a different problem than monitoring. Our tools span the gamut of aggregating and presenting data, to doing "heavy lifting" of things like installing patches, managing our load balancers, and removing bad query plans from our database servers. Our in-house tooling fills in gaps where existing tooling wasn't responsive enough or made it difficult to deal with certain edge cases. General purpose tools like Orion satisfy 80% of our use cases, but we are fanatical about performance and functionality so we want to fill in that additional 20%. In fact, the majority of our Orion monitors are custom script monitors, which we have to create ourselves as it is. If what we do can work for others (I've worked in several environments with Orion and had the same issues in each environment), that's a bonus. We've found we are building a significant number of these projects and that's why we are looking for a dedicated developer for our team.
- dvanduzer 13y ago
  I'll have to look at the overlap between Puppet's Windows tools and DSC. These are definitely distinct from monitoring, but monitoring configuration should be populated from the same configuration store as Puppet. (Sometimes Puppet is the configuration store.) I've never evaluated Orion in particular, but it's slightly puzzling if you're creating a lot of custom monitoring scripts. In the low-touch Nagios deployments I've seen, this is often because someone didn't understand good places to use macros, and centralize more of the parameters.
  - jacques_chester 13y ago
    The problem is due to history. Configuration systems have historically looked at config as bits-on-a-disk. Supervisor systems look at bits-in-memory. And they emerged and evolved independently. So there's pain points and impedance mismatches regardless of which one you start with. What's needed is a system that sees that configuration and supervision are the same problem: you have a directed graph of what a system can look like, plus a compare-and-repair mechanism to drag the system to that state frequently. I've personally looked at Chef, Puppet and Cfengine, none of which really does all of them the way I would like. http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...

KyleBrandt 13y ago

We use Puppet a lot and are generally quite happy. We have recently started leveraging Hiera which is addressing our main issue we had with Puppet: Reusability problems in the code since Modules (code) and our data were not easily separated. A lot of these tools are about integrating / presenting but that isn't entirely the case. Status does its own polling of things like Redis and SQL. Realog parses the data, structures it in Redis etc, and the patching dashboard can kick of updates.

dvanduzer 13y ago
Status handles the scheduling of the polling, or the actual polling, or both? One thing Nagios handles well, although it isn't exactly well polished, is distribution, scheduling, and aggregation of the polling. Also, anecdotally, "nobody" seems to know how easy it is to do simple-intermediate monitoring of MS-SQL in Nagios. I happen to be using this, at the moment: https://github.com/scot0357/check_mssql_collection https://github.com/scot0357/check_mssql_collection edit: And thank you for the concise explanation of Hiera. I had entirely ignored it as just another add-on as Puppet "goes enterprise"

stevenjmurawski 13y ago

We tried Nagios a while back, before going to Orion. Nagios has some great structural ideas, but fell down in other areas (don't know all of them, that was before I came on board). We use Puppet to manage our Linux infrastructure and are testing Desired State Configuration for our Windows systems. Puppet and DSC solve a different problem than monitoring. Our tools span the gamut of aggregating and presenting data, to doing "heavy lifting" of things like installing patches, managing our load balancers, and removing bad query plans from our database servers. Our in-house tooling fills in gaps where existing tooling wasn't responsive enough or made it difficult to deal with certain edge cases. General purpose tools like Orion satisfy 80% of our use cases, but we are fanatical about performance and functionality so we want to fill in that additional 20%. In fact, the majority of our Orion monitors are custom script monitors, which we have to create ourselves as it is. If what we do can work for others (I've worked in several environments with Orion and had the same issues in each environment), that's a bonus. We've found we are building a significant number of these projects and that's why we are looking for a dedicated developer for our team.

dvanduzer 13y ago
I'll have to look at the overlap between Puppet's Windows tools and DSC. These are definitely distinct from monitoring, but monitoring configuration should be populated from the same configuration store as Puppet. (Sometimes Puppet is the configuration store.) I've never evaluated Orion in particular, but it's slightly puzzling if you're creating a lot of custom monitoring scripts. In the low-touch Nagios deployments I've seen, this is often because someone didn't understand good places to use macros, and centralize more of the parameters.
- jacques_chester 13y ago
  The problem is due to history. Configuration systems have historically looked at config as bits-on-a-disk. Supervisor systems look at bits-in-memory. And they emerged and evolved independently. So there's pain points and impedance mismatches regardless of which one you start with. What's needed is a system that sees that configuration and supervision are the same problem: you have a directed graph of what a system can look like, plus a compare-and-repair mechanism to drag the system to that state frequently. I've personally looked at Chef, Puppet and Cfengine, none of which really does all of them the way I would like. http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reigning-in-chaos/ http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign...