3 ms·

I should probably couch this in the caveat that I am using Packer and Terraform to control the infrastructure itself :) So machines are effectively configured

by duggan 12y ago

I should probably couch this in the caveat that I am using Packer and Terraform to control the infrastructure itself :) So machines are effectively configured "offline", tested, and replaced. It's not enough though (usual things like databases and development tweaks like config changes are problematic) so I'm back into the world of config management to see how it can help. Matt makes some good points, and we broadly agree on controlling infrastructure, I just think that most of the CM systems are inadequate abstractions - they don't go far enough, and it leaves them largely as convenience wrappers around shell, but ironically enough, usually without the debuggability of shell. I've worked with Chef for years, and recently have started seeing what I can do with Ansible. For me, the major thing in Chef's favour was the combination of attribute driven configuration and integration with things like Test Kitchen and ServerSpec. Chef requries buy in to the ruby ecosystem though, and that's just not something I want to expose non-ruby developers to (it's just too brittle), which is why I'm exploring Ansible. What I'm finding, though, is that for any reasonably interesting tasks, I'm still forced to execute "command" strings, which is not an improvement over shell, and gives people the false impression that it's idempotent (or even just safer). Also most of these systems are trying to upsell me a centralized management system and UI, handing over the keys to my infrastructure, which is just not something I'm willing to do. Edit: I'd also add that cross-OS portability of CM (if that's what you're referring to), in my experience, a bit of a red herring.

skuhn 12y ago
I think the biggest issue with current CM systems is that they aren't made by people with any experience actually administering large scale infrastructures. The focus to me seems all wrong. Ansible has a blue million modules, but so many of them are incomplete or downright useless. The cron module can handle cron.d files, but it still tries to manage individual lines in the file and tracks state with comments. It can easily get confused and add multiple entries for the same cron job. Puppet had the same problem. The solution is obvious: just make it barf a pre-made file into /etc/cron.d. But why does no CM actually do this? Over the last year I have spent a lot of time using Ansible, and I've eventually given up on the cron, sysctl, service and get_url modules and resorted to running commands or copying files. This isn't very clean and it can be brittle, but the alternative of fixing the modules is off the table. The debugging or logging support in most modules is literally non-existent. There also seems to be a pretty serious scaling issue with the way Ansible processes its inventory, even when using the (absolutely essential) _meta optimization that provides the entire inventory in one command. This is at only a few hundred hosts and tens of thousands of variables. Contrast this with Puppet: while there are a great many things I hate about it, the dependency system and ability to define your own types inside of the DSL was pretty powerful. Too bad that you eventually had to get down to writing Ruby modules to do certain things. Also too bad that they built it as a client-server system, allow you to export resources between hosts, and took forever to make the execution order deterministic. The final nail in the coffin for CMs and me is that Ansible (along with Chef, Puppet, Salt) try to do so much but have no concept of absolute state enforcement. The host should always either be in state A or B; the CM system should be able to move in either direction between A and B; and it should never be left in a state inbetween A and B. With Ansible it's quite possible to get permanently locked into a middle state, since it may die before handlers are run (which are then never triggered in subsequent runs). Building these tools is admittedly harder than it looks, but the fundamental building blocks of a CM system aren't that tough. So many recipes boil down to literally a three step process: install package, copy config file (templating is super nice), start service. Stop trying to play nice with local modifications, the whole point is that I want you to enforce state absolutely -- local modifications are the enemy! I wish there was a CM that focused on getting this stuff absolutely right and then made it easy to build on top of that.
- helloiamaperson 12y ago
  Take a look at bosh: http://drnicwilliams.com/2012/05/15/bosh-what-how-when/ http://drnicwilliams.com/2012/05/15/bosh-what-how-when/ . It has its own problems, but it does exactly what you want, and is used for large scale deployments.

by duggan 12y ago

skuhn 12y ago

I think the biggest issue with current CM systems is that they aren't made by people with any experience actually administering large scale infrastructures. The focus to me seems all wrong. Ansible has a blue million modules, but so many of them are incomplete or downright useless. The cron module can handle cron.d files, but it still tries to manage individual lines in the file and tracks state with comments. It can easily get confused and add multiple entries for the same cron job. Puppet had the same problem. The solution is obvious: just make it barf a pre-made file into /etc/cron.d. But why does no CM actually do this? Over the last year I have spent a lot of time using Ansible, and I've eventually given up on the cron, sysctl, service and get_url modules and resorted to running commands or copying files. This isn't very clean and it can be brittle, but the alternative of fixing the modules is off the table. The debugging or logging support in most modules is literally non-existent. There also seems to be a pretty serious scaling issue with the way Ansible processes its inventory, even when using the (absolutely essential) _meta optimization that provides the entire inventory in one command. This is at only a few hundred hosts and tens of thousands of variables. Contrast this with Puppet: while there are a great many things I hate about it, the dependency system and ability to define your own types inside of the DSL was pretty powerful. Too bad that you eventually had to get down to writing Ruby modules to do certain things. Also too bad that they built it as a client-server system, allow you to export resources between hosts, and took forever to make the execution order deterministic. The final nail in the coffin for CMs and me is that Ansible (along with Chef, Puppet, Salt) try to do so much but have no concept of absolute state enforcement. The host should always either be in state A or B; the CM system should be able to move in either direction between A and B; and it should never be left in a state inbetween A and B. With Ansible it's quite possible to get permanently locked into a middle state, since it may die before handlers are run (which are then never triggered in subsequent runs). Building these tools is admittedly harder than it looks, but the fundamental building blocks of a CM system aren't that tough. So many recipes boil down to literally a three step process: install package, copy config file (templating is super nice), start service. Stop trying to play nice with local modifications, the whole point is that I want you to enforce state absolutely -- local modifications are the enemy! I wish there was a CM that focused on getting this stuff absolutely right and then made it easy to build on top of that.

helloiamaperson 12y ago
Take a look at bosh: http://drnicwilliams.com/2012/05/15/bosh-what-how-when/ http://drnicwilliams.com/2012/05/15/bosh-what-how-when/ . It has its own problems, but it does exactly what you want, and is used for large scale deployments.