9 ms·

I'm not a web developer but I have a side-project that runs on a cobbled together EC2 instance. The server state is in theory documented in a set of of shell sc

by SandB0x 13y ago

I'm not a web developer but I have a side-project that runs on a cobbled together EC2 instance. The server state is in theory documented in a set of of shell scripts and virtualenv requirements files. I know that I should be doing this in a more robust way but whenever I try and read up on configuration management tools like Puppet and Chef, they're all described in comparative terms - Puppet does X better than Vagrant which does Y better than Chef, etc. I quickly lose patience and get back to digging myself into a deeper technical hole. Is there a non-recursive explanation of what these tools are able to do and where someone like me should start? Edit: Thanks for the helpful responses!

danudey 13y ago
These tools allow you to specify a state and then specify which servers should have that state. A 'state', loosely speaking, is a collection of definitions of 'X should be Y'. For example: Package 'libapache2-mod-php' should be installed. File '/etc/apache2/sites-available/customersite' should be the contents of this file we have on the master server. File '/etc/apache2/sites-enabled/customersite' should be a symlink to above. Directory '/var/www/customersite' should be the contents of this git repository. Command 'apache2ctl graceful' should be run if any of the above change. With these five rules, you can turn a default install of Ubuntu into a webserver providing client files in a matter of seconds. Once you have those rules written, you then apply them. You can wildcard, so that hosts matching 'www*.yourhostingco.com' get these rules. Now, when you want to add a new webserver to your cluster, you install a new Ubuntu instance, you register it with the Salt master server, and then trigger a state update, and you're done. No SCP'ing files, no manual git checkouts, no copy-paste-edit configuration management. Then, once you've got all that, you can get into templating. You can do templating both on the contents of files (e.g. for memcached config, insert the machine's internal IP address instead of having one config per server) as well as the rules themselves to avoid having a ton of boilerplate (for module in 'list','of','python','modules', install the module via pip with these rules). One of my favourite benefits of something like Salt is that if you use Salt to do all of your configuration, then just back up your salt config to github or wherever, then you always have a record of how you did things. You don't get the incomplete documentation or missing steps that happen with most approaches. As a sysadmin managing a cluster of systems, it can be life-changing.
- nwmcsween 13y ago
  But the problem that all of the configuration managers have and also why they all suck is that the state you specify has no guarantee of being the end state. A much better alternative would be to manage state in chunks and test those chunks, e.g an http-server transaction would require responding on port 80 if not fail or roll it back. Rolling back could be using namespaces, jails, etc.
  - johtso 13y ago
    You might be interested in Salt's requisite statements: http://docs.saltstack.com/ref/states/ordering.html#requisite-statements http://docs.saltstack.com/ref/states/ordering.html#requisite... In combination with service.running you can require that, for example, package x is installed, and its service is running, before service y is started.
  - showerst 13y ago
    Ansible allows you to do this, it has failure modes that let you take actions, including rolling back.
  - Goladus 13y ago
    I would not say that is a reason why all configuration managers suck. The feature you'd like to see would represent a significant increase in overhead, complexity, and potential drawbacks that not everyone needs yet. Meanwhile, a disciplined system of designing and documenting tests and rollback procedures before committing changes will accomplish this without the need for additional software. My guess is that if you are not disciplined enough to design tests and rollback procedures on your own, then you are probably going to just switch off or work around those features if they are built into the software.
mitchellh 13y ago
The other responses are awesome, but I just want to nitpick one thing (as the creator of Vagrant): Vagrant isn't a configuration management tool, so it can't really be compared with Chef/Puppet. Vagrant actually works with Chef, Puppet, CFEngine, Ansible, Salt, and more.
- tieTYT 13y ago
  You told us what it "isn't" but you didn't tell us what it "is". Yes, I could just google, but I don't care enough. I think you missed out on a little bit of marketing here.
  - nbouscal 13y ago
    I don't think Mitchell really needs to worry much at this point about marketing Vagrant. Everyone in the DevOps space is already plenty well familiar with it (mostly because it's incredibly awesome). Also, TFA explains what it is: “Vagrant is an established project that wraps around a number of existing virtualization providers to allow for extremely quick provisioning of disposable, consistent environments.”
    - SandB0x 13y ago
      TFA, or any least the quote you have selected, does not explain what it is. It states (in quite ambiguous terms) what the combination of Vagrant and "existing virtualization providers" can achieve but not what Vagrant's role is. From http://www.vagrantup.com http://www.vagrantup.com: "Create a single file for your project to describe the type of machine you want, the software that needs to be installed, and the way you want to access the machine. Store this file with your project code" That, and http://docs.vagrantup.com/v2/why-vagrant/ http://docs.vagrantup.com/v2/why-vagrant/, sounds quite similar to a configuration management tool to me (I believe you that it's not, I'm just saying I wouldn't have understood that). Seems geared to launching instances/VMs too perhaps. Maybe it's my lack of domain knowledge and maybe I have unfair expectations but I think that the sales pitch should be accessible to a developer who's not solely in the "DevOps space". Accessibility is good because I'm sure it is incredibly awesome and may well be useful to a broader set of people who aren't yet in the loop.
deleted 13y ago
[deleted]
- danudey 13y ago
  This is a pretty reductionist explanation, and it misses a lot of the benefits of state management. You make it sound like you just write a bunch of lines of shell script and you can sub in some variables, which is not at all their major strength. The benefit of Salt, which it has over Ansible, is that it doesn't have to use SSH (which, as mentioned in the article, doesn't scale well to large deploys), and it can do a great deal more things than just 'run shell scripts from a file'. If someone were to read only your comment they wouldn't see the benefit of these systems at all, which is unfortunate.
  - tr4656 13y ago
    Ansible with Fireball wouldn't use SSH but rather 0mq.
crdoconnor 13y ago
They basically let you write, using a declarative language, what state the server should be in. The software figures out how to set that up for you. When you want to change the state, you edit the file - say, to add a package. At any point you can set up an entirely new server from scratch using that same code. You can't do that easily with bash scripts. (you could if they worked like virtualenv requirements files but that only works for python's virtualenv...)
AlexanderDhoore 13y ago
I just now (past few days) started learning about this stuff. I've been doing the "Learning Puppet" tutorials [1]. The most important thing about it is that you describe the state of the server. And puppet applies it for you. It's idempotent (you might know this term from REST). You can run puppet multiple times and the end result would be the same. Normally (without config management) you would write scripts that do the configuration. If you ran them twice, things could go wrong, or they would overwrite the stuff from last time. Scripts are descriptions of what steps to take. Puppet manifests are descriptions of what state you want the server to be in. [1] http://docs.puppetlabs.com/learning/index.html http://docs.puppetlabs.com/learning/index.html
- fmstephe 13y ago
  Ok, the declarative and idempotency sounds great. But consider this one use case. I want to stop a service, upgrade it to a new version, restart that service. The service is an application which I wrote that runs in apache tomcat. 1: Stop Service 2: Upgrade 3: Start Service Right now I have just made three non-declarative statements about my tomcat server. I am describing a process that runs through a series of steps to reach a new conclusion. So puppet has these beautiful (sounding) properties. But declaratively describing the state of my servers isn't a real problem I have. I need to deal with the way that my servers change. I need to upgrade services, including starting and stopping. Puppet absolutely fails at this. Any attempt to describe a process is antagonistic to puppet at a fundamental level. Whenever people seek a solution to these problems they always get fed that line. That puppet is declarative and idempotent, which is fine. But we should be _very_ clear that puppet does a spectacularly bad job of solving a very ordinary problem that I have, and which I would expect almost everyone else working with large numbers of servers to have too.
  - MarcParadise 13y ago
    Your actual objective seems to be: 1. The current version of MyApplication must be deployed and running. If you're interested in trying out Chef - which seems like a good fit from your comment - read on. For tomcat you could take a look at this community-provided cookbook, which contains resources that make it simple: https://github.com/opscode-cookbooks/application_java https://github.com/opscode-cookbooks/application_java If you explicitly need to stop/start tomcat beforehand (I believe this resource does a restart after deploying your app) you can do that with the "service" resource built into Chef - though doing so may mean that you're restarting every time you converge the node, which is probably not what you're looking for.
the1 13y ago
if you're using EC2, why not just create an AMI and forget about configuration tools?
- bfrog 13y ago
  Having done this a dozen or more times... can you package your ami for your developers to use? Can you remake your image exactly the same way again as it was when you first imaged it should it be lost in the cloud (100% reliability isn't something AWS provides). Provisioning tools let you create it, and incrementally update your image in a way that lets you redo it from scratch at any time. That alone is the reason why I like the idea, not necessarily the resulting applications that have been created so far for the task. Shell scripts could do the same and have for years if your only interested in from scratch setups.
  - pestaa 13y ago
    > can you package your ami for your developers to use SuSE Studio lets you do exactly that. (Though exporting to AMIs is just a feature, not the main objective.)
  - gyepi 13y ago
    This is not a difficult problem. I recently had to stand up a cluster of EC2 instances for a job that required a cluster of them and used these three steps: 1. Write a script to configure an instance and run it when the instance starts. 2. Clone the instance to an image 3. Run instances based on the image. It's quite straightforward to do. See http://github.com/gyepisam/fcc-textify http://github.com/gyepisam/fcc-textify for more details.
- ef4 13y ago
  That doesn't really solve the problem, it just moves it. If AMIs are your deployment method of choice, you still need to build a repeatable process for applying updates and changes to your previous base AMI, testing that it still works under all your realistic configurations, and then gracefully deploying it to your production infrastructure. Which is all doable, but not trivial. And the processes you build will be very Amazon-centric.
nodesocket 13y ago
Founder of https://commando.io https://commando.io here. We are a web based interface for doing remote executions via SSH. Our approach is a beautiful web interface with no external dependencies (agents). Salt and Ansible are amazing, but they still have a bit of a learning curve and setup "costs". Our target market also tends to be less technically proficient, and thus not willing to dive into more robust solutions. Think cPanel for operations. SandBOx, if you'd like to try out Commando.io, let me know, and I'll get you a private beta invite.
- halayli 13y ago
  My Ansible learning curve was 5 minutes.
  - StavrosK 13y ago
    I'm betting it would have been 2 if there were better examples. The concept is trivial to get and the reference is great, but a few more fleshed-out examples would be nice. After having used it for a while, I adore it. One minute and one command to go from a new project to a full deployment, what more can you ask.
- dsl 13y ago
  A giant database of SSH keys from people who can't setup basic operational tools? Jesus, and I thought the CDCs "big vault full of things that will kill you" was a terrible idea...
coolsunglasses 13y ago
Fabric can make for a nice midway point between a pile of scripts and using Chef/Puppet.
saintx 13y ago
Reading through many of the responses, they're almost all phrased in terms of "X does Y better than Z" comparisons, directly ignoring your amusing and insightful comment. At this point, the only thing that could save ordinary users from the melee of competing Open Source configuration management frameworks is an illustration by Randall Munroe. (My hacker senses are tingling. If you listen very carefully, you can just barely hear a new eponymous law being born.)
- deleted 13y ago
  [deleted]
StavrosK 13y ago
Here's a small, concrete example I wrote to showcase ansible: http://www.stavros.io/posts/example-provisioning-and-deployment-ansible/ http://www.stavros.io/posts/example-provisioning-and-deploym...
grandalf 13y ago
The tools are overly complicated and offer little value for installations with fewer than hundreds of servers. Checkout fabric.py and cuisine.py, both are small, simple, elegant approaches. Combined with a native package manager (like apt), fabric+cuisine is nearly as powerful as puppet or chef.

by SandB0x 13y ago

danudey 13y ago

These tools allow you to specify a state and then specify which servers should have that state. A 'state', loosely speaking, is a collection of definitions of 'X should be Y'. For example: Package 'libapache2-mod-php' should be installed. File '/etc/apache2/sites-available/customersite' should be the contents of this file we have on the master server. File '/etc/apache2/sites-enabled/customersite' should be a symlink to above. Directory '/var/www/customersite' should be the contents of this git repository. Command 'apache2ctl graceful' should be run if any of the above change. With these five rules, you can turn a default install of Ubuntu into a webserver providing client files in a matter of seconds. Once you have those rules written, you then apply them. You can wildcard, so that hosts matching 'www*.yourhostingco.com' get these rules. Now, when you want to add a new webserver to your cluster, you install a new Ubuntu instance, you register it with the Salt master server, and then trigger a state update, and you're done. No SCP'ing files, no manual git checkouts, no copy-paste-edit configuration management. Then, once you've got all that, you can get into templating. You can do templating both on the contents of files (e.g. for memcached config, insert the machine's internal IP address instead of having one config per server) as well as the rules themselves to avoid having a ton of boilerplate (for module in 'list','of','python','modules', install the module via pip with these rules). One of my favourite benefits of something like Salt is that if you use Salt to do all of your configuration, then just back up your salt config to github or wherever, then you always have a record of how you did things. You don't get the incomplete documentation or missing steps that happen with most approaches. As a sysadmin managing a cluster of systems, it can be life-changing.

nwmcsween 13y ago
But the problem that all of the configuration managers have and also why they all suck is that the state you specify has no guarantee of being the end state. A much better alternative would be to manage state in chunks and test those chunks, e.g an http-server transaction would require responding on port 80 if not fail or roll it back. Rolling back could be using namespaces, jails, etc.
- johtso 13y ago
  You might be interested in Salt's requisite statements: http://docs.saltstack.com/ref/states/ordering.html#requisite-statements http://docs.saltstack.com/ref/states/ordering.html#requisite... In combination with service.running you can require that, for example, package x is installed, and its service is running, before service y is started.
- showerst 13y ago
  Ansible allows you to do this, it has failure modes that let you take actions, including rolling back.
- Goladus 13y ago
  I would not say that is a reason why all configuration managers suck. The feature you'd like to see would represent a significant increase in overhead, complexity, and potential drawbacks that not everyone needs yet. Meanwhile, a disciplined system of designing and documenting tests and rollback procedures before committing changes will accomplish this without the need for additional software. My guess is that if you are not disciplined enough to design tests and rollback procedures on your own, then you are probably going to just switch off or work around those features if they are built into the software.

mitchellh 13y ago

The other responses are awesome, but I just want to nitpick one thing (as the creator of Vagrant): Vagrant isn't a configuration management tool, so it can't really be compared with Chef/Puppet. Vagrant actually works with Chef, Puppet, CFEngine, Ansible, Salt, and more.

tieTYT 13y ago
You told us what it "isn't" but you didn't tell us what it "is". Yes, I could just google, but I don't care enough. I think you missed out on a little bit of marketing here.
- nbouscal 13y ago
  I don't think Mitchell really needs to worry much at this point about marketing Vagrant. Everyone in the DevOps space is already plenty well familiar with it (mostly because it's incredibly awesome). Also, TFA explains what it is: “Vagrant is an established project that wraps around a number of existing virtualization providers to allow for extremely quick provisioning of disposable, consistent environments.”
  - SandB0x 13y ago
    TFA, or any least the quote you have selected, does not explain what it is. It states (in quite ambiguous terms) what the combination of Vagrant and "existing virtualization providers" can achieve but not what Vagrant's role is. From http://www.vagrantup.com http://www.vagrantup.com: "Create a single file for your project to describe the type of machine you want, the software that needs to be installed, and the way you want to access the machine. Store this file with your project code" That, and http://docs.vagrantup.com/v2/why-vagrant/ http://docs.vagrantup.com/v2/why-vagrant/, sounds quite similar to a configuration management tool to me (I believe you that it's not, I'm just saying I wouldn't have understood that). Seems geared to launching instances/VMs too perhaps. Maybe it's my lack of domain knowledge and maybe I have unfair expectations but I think that the sales pitch should be accessible to a developer who's not solely in the "DevOps space". Accessibility is good because I'm sure it is incredibly awesome and may well be useful to a broader set of people who aren't yet in the loop.

deleted 13y ago

[deleted]

danudey 13y ago
This is a pretty reductionist explanation, and it misses a lot of the benefits of state management. You make it sound like you just write a bunch of lines of shell script and you can sub in some variables, which is not at all their major strength. The benefit of Salt, which it has over Ansible, is that it doesn't have to use SSH (which, as mentioned in the article, doesn't scale well to large deploys), and it can do a great deal more things than just 'run shell scripts from a file'. If someone were to read only your comment they wouldn't see the benefit of these systems at all, which is unfortunate.
- tr4656 13y ago
  Ansible with Fireball wouldn't use SSH but rather 0mq.

crdoconnor 13y ago

They basically let you write, using a declarative language, what state the server should be in. The software figures out how to set that up for you. When you want to change the state, you edit the file - say, to add a package. At any point you can set up an entirely new server from scratch using that same code. You can't do that easily with bash scripts. (you could if they worked like virtualenv requirements files but that only works for python's virtualenv...)

AlexanderDhoore 13y ago

I just now (past few days) started learning about this stuff. I've been doing the "Learning Puppet" tutorials [1]. The most important thing about it is that you describe the state of the server. And puppet applies it for you. It's idempotent (you might know this term from REST). You can run puppet multiple times and the end result would be the same. Normally (without config management) you would write scripts that do the configuration. If you ran them twice, things could go wrong, or they would overwrite the stuff from last time. Scripts are descriptions of what steps to take. Puppet manifests are descriptions of what state you want the server to be in. [1] http://docs.puppetlabs.com/learning/index.html http://docs.puppetlabs.com/learning/index.html

fmstephe 13y ago
Ok, the declarative and idempotency sounds great. But consider this one use case. I want to stop a service, upgrade it to a new version, restart that service. The service is an application which I wrote that runs in apache tomcat. 1: Stop Service 2: Upgrade 3: Start Service Right now I have just made three non-declarative statements about my tomcat server. I am describing a process that runs through a series of steps to reach a new conclusion. So puppet has these beautiful (sounding) properties. But declaratively describing the state of my servers isn't a real problem I have. I need to deal with the way that my servers change. I need to upgrade services, including starting and stopping. Puppet absolutely fails at this. Any attempt to describe a process is antagonistic to puppet at a fundamental level. Whenever people seek a solution to these problems they always get fed that line. That puppet is declarative and idempotent, which is fine. But we should be _very_ clear that puppet does a spectacularly bad job of solving a very ordinary problem that I have, and which I would expect almost everyone else working with large numbers of servers to have too.
- MarcParadise 13y ago
  Your actual objective seems to be: 1. The current version of MyApplication must be deployed and running. If you're interested in trying out Chef - which seems like a good fit from your comment - read on. For tomcat you could take a look at this community-provided cookbook, which contains resources that make it simple: https://github.com/opscode-cookbooks/application_java https://github.com/opscode-cookbooks/application_java If you explicitly need to stop/start tomcat beforehand (I believe this resource does a restart after deploying your app) you can do that with the "service" resource built into Chef - though doing so may mean that you're restarting every time you converge the node, which is probably not what you're looking for.

the1 13y ago

if you're using EC2, why not just create an AMI and forget about configuration tools?

bfrog 13y ago
Having done this a dozen or more times... can you package your ami for your developers to use? Can you remake your image exactly the same way again as it was when you first imaged it should it be lost in the cloud (100% reliability isn't something AWS provides). Provisioning tools let you create it, and incrementally update your image in a way that lets you redo it from scratch at any time. That alone is the reason why I like the idea, not necessarily the resulting applications that have been created so far for the task. Shell scripts could do the same and have for years if your only interested in from scratch setups.
- pestaa 13y ago
  > can you package your ami for your developers to use SuSE Studio lets you do exactly that. (Though exporting to AMIs is just a feature, not the main objective.)
- gyepi 13y ago
  This is not a difficult problem. I recently had to stand up a cluster of EC2 instances for a job that required a cluster of them and used these three steps: 1. Write a script to configure an instance and run it when the instance starts. 2. Clone the instance to an image 3. Run instances based on the image. It's quite straightforward to do. See http://github.com/gyepisam/fcc-textify http://github.com/gyepisam/fcc-textify for more details.
ef4 13y ago
That doesn't really solve the problem, it just moves it. If AMIs are your deployment method of choice, you still need to build a repeatable process for applying updates and changes to your previous base AMI, testing that it still works under all your realistic configurations, and then gracefully deploying it to your production infrastructure. Which is all doable, but not trivial. And the processes you build will be very Amazon-centric.

nodesocket 13y ago

Founder of https://commando.io https://commando.io here. We are a web based interface for doing remote executions via SSH. Our approach is a beautiful web interface with no external dependencies (agents). Salt and Ansible are amazing, but they still have a bit of a learning curve and setup "costs". Our target market also tends to be less technically proficient, and thus not willing to dive into more robust solutions. Think cPanel for operations. SandBOx, if you'd like to try out Commando.io, let me know, and I'll get you a private beta invite.

halayli 13y ago
My Ansible learning curve was 5 minutes.
- StavrosK 13y ago
  I'm betting it would have been 2 if there were better examples. The concept is trivial to get and the reference is great, but a few more fleshed-out examples would be nice. After having used it for a while, I adore it. One minute and one command to go from a new project to a full deployment, what more can you ask.
dsl 13y ago
A giant database of SSH keys from people who can't setup basic operational tools? Jesus, and I thought the CDCs "big vault full of things that will kill you" was a terrible idea...

coolsunglasses 13y ago

Fabric can make for a nice midway point between a pile of scripts and using Chef/Puppet.

saintx 13y ago

Reading through many of the responses, they're almost all phrased in terms of "X does Y better than Z" comparisons, directly ignoring your amusing and insightful comment. At this point, the only thing that could save ordinary users from the melee of competing Open Source configuration management frameworks is an illustration by Randall Munroe. (My hacker senses are tingling. If you listen very carefully, you can just barely hear a new eponymous law being born.)

deleted 13y ago
[deleted]

StavrosK 13y ago

Here's a small, concrete example I wrote to showcase ansible: http://www.stavros.io/posts/example-provisioning-and-deployment-ansible/ http://www.stavros.io/posts/example-provisioning-and-deploym...

grandalf 13y ago

The tools are overly complicated and offer little value for installations with fewer than hundreds of servers. Checkout fabric.py and cuisine.py, both are small, simple, elegant approaches. Combined with a native package manager (like apt), fabric+cuisine is nearly as powerful as puppet or chef.