5 ms·

Servers are fun: Ansible

32 points by lauriswtf 12y ago

larrybolt 12y ago
One of the things that got me into ansible was simply doing ansible all -i ./ansible_hosts -m command -a "cat /etc/issue" From there I started going trough the possible options I could pass instead of each time doing a command, such as checking the network speed using a script etc.. As it might be useful for some I have next aliases set: export ANSIBLE_PATH="$HOME/web/clusters-servers/mycluster" alias a="ansible -i $ANSIBLE_PATH/hosts" alias ap="ansible-playbook -i $ANSIBLE_PATH/hosts" # for vagrant testing machines where I test playbooks before deploying alias atest="ansible -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s" alias aptest="ansible-playbook -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s" On of the (slightly) annoying things is after bringing up a new node you have to "bootstrap" the node so you can issue ansible commands on it, I have a simple script [1] for doing so but I'd love to hear how others do it! [1]: https://gist.github.com/larrybolt/85b59f47615be9fcb643 https://gist.github.com/larrybolt/85b59f47615be9fcb643
- falcolas 12y ago
  The raw command. Doesn't require anything to be installed on the target host, so you can set up a common playbook which will bootstrap the rest of Ansible.
- 0x0 12y ago
  For a more low-tech solution, I've found cssh / csshX ("Cluster SSH") quite the timesaver. It pops up a bunch of xterms/Terminal.app windows and lets you type in one window while sending the keystrokes to all of the ssh sessions simultaneously.
  - mpdehaan2 12y ago
    The problem with cluster SSH is it's not declarative, so if one server already has a package, and another does not, your servers will not behave the same if typing into all of them. This is why Ansible modules (and most other config tools) instead describe the desired end state, and perform the commands needed to get you to those states. Basically all config tools arose out of a need for solutions to replace parallel scripting with something more robust that accounted for potential variance between systems, and also to make things more push-button.
    - _delirium 12y ago
      I can see that for some kinds of admin tasks, but why package management? At least on Debian, the package manager is already declarative: There's an install/upgrade planner that figures out how to get from the current system state to the final desired state, given constraints such as packages' declared Requires/Conflicts/etc. lines. If the system is already at that state, it does nothing.
      - weaksauce 12y ago
        How often is setting up a package just running apt-get? There is an overwhelming majority of times where you have to change something in the config file somewhere. Ansible can do anything you want to provision your boxes.
  - invertedohm 12y ago
    Indeed, cssh is great for getting a quick visual on a few servers, but you're pretty much limited to what will fit on your screen. Ansible really shines when you scale it up a bit further, and it grows well, allowing you to add bits and pieces as needed. The hosts/inventory in ansible as a source of truth for your infrastructure is also a nice feature. You could use a selection from your ansible inventory to open a cssh with 'top' running on all your webservers, for example.
- killing_time 12y ago
  For testing ansible playbooks/roles in a vagrant VM, I find a simpler way is to use Vagrant's built-in support for Ansible as a provisioner. This way, launching your test VM is just a `vagrant up` and running ansible against it (repeatedly, if needed) is `vagrant provision`. No need to pass inventory/key parameters at the shell this way - Vagrant calls Ansible with the right settings. One thing that's easy to miss is to make Ansible use sudo when running against the Vagrant VM, since the vagrant user by convention has passwordless sudo rights (ansible.sudo = true in the Vagrantfile). http://docs.vagrantup.com/v2/provisioning/ansible.html http://docs.vagrantup.com/v2/provisioning/ansible.html is a helpful reference doc.
- spindritf 12y ago
  You can point to the hosts file in ~/.ansible.cfg hostfile = ~/.ansible/hosts Or, if you run more than one project with multiple servers each, ansible will also read an ansible.config file in the current directory. You can have a separate hosts list, options, and playbooks for each set of servers. Then just cd ~/projects/whatever and ansible will work accordingly. You can put any given set of playbooks and configs into git and share with your team so that everyone is using ansible the same way. And that repo won't even carry any auth info.
- mpdehaan2 12y ago
  For Cloud services, SSH is "cloud native" meaning authorized keys are automatically injected by the cloud provider. If using metal installs, you can also set this up in your preseed or kickstart. However, you should never have to bootstrap a machine with another script to install things like python-apt, as you can just use the "raw" command to do this for anything that is not part of the box, see http://docs.ansible.com/raw_module.html http://docs.ansible.com/raw_module.html - so it's still self-bootstrapping. disclaimer: creator of project.
  - larrybolt 12y ago
    Thanks! Didn't knew about this, or just read over it. I'm mostly using ansible for learning how to use it, don't have huge clusters yet to manage. Most of the hosts I have come from LEB (LowEndBox) offerings so therefor no way to deploy my own base image. And thank you for the work you put into ansible sir!
jamespo 12y ago
There's about 2 sentences describing what Ansible does in this article.
- lauriswtf 12y ago
  If you want to learn more about Ansible, the article links to the official docs http://docs.ansible.com/intro_getting_started.html http://docs.ansible.com/intro_getting_started.html.
willcodeforfoo 12y ago
I've long been accustomed to setting up servers one-command-at-a-time with SSH. I tried Chef and some of the others, but they seemed too complicated to get setup. Ansible's out of the box experience allows you to start with some simple but useful constructs then allow you to graduate to more and more complex usage. I started with some of the one-liners here, then setting up inventory files, then roles, some simple tasks and then (what really sold me) was templates. Even just having those configuration files in an SCM is a huge win, but I honestly squealed with joy when I was able to write a nginx configuration file that dynamically included backend servers given a particular role. And that provisioning a new backend server took a couple minutes and was automatically added to the load-balancing proxy! (not to mention was instantly sending metrics to a graphite/statsd backend because collectd was installed automatically too as part of a common role and configured to point to that log collecting server!) Magic!
Rezo 12y ago
Ansible is great, but could really have a better first time experience. The system itself is incredibly simple, but the docs and instructions are anything but, and seem to be modeled after Chef and Puppets which are frankly terrible. Instead, look at something like Vagrant as the shining example of how to do it right. Just compare http://www.ansible.com/ http://www.ansible.com/ with the excellent http://www.vagrantup.com/ http://www.vagrantup.com/ when it comes to downloading and running your first commands. After clicking around for a good while on the Ansbile site you finally find "To install on a Mac, make sure you have Homebrew, then run... ", yeah, no thanks. Oh, there's a "pip" command, if I happen to have that installed (hint: nobody on Mac has by default), but at least there's some more magic commands I can run to install that first. After running the "Looks Totally Safe To Me! (TM)" command "sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install ansible" it blows up on Mavericks with some error about something called paramiko and headers not being available. The attrition rate for newbies on Mac must be phenomenal, because you have to be really dedicated to give it another shot after this. What I'm getting at is, Python software distribution is an abysmal experience, always has been, and you should not subject your users to it. They're not Python developers, they're largely not even devs at all, they don't care about your pip's and brews. Compare this with Vagrant: Double click this here DMG, run "vagrant init", "vagrant up" and done, in nice _big_ friendly letters on the web site. Disclaimer: I use and love Ansible (on Linux), and want to see it succeed. Telling my Mac using ops team to check it out soured everyone's first impression right from the start. Rant over.
- austinhyde 12y ago
  As someone brand new to Ansible, and having been developing on OSX for years now, I'd just like to play devil's advocate. Finding the Ansible docs (http://docs.ansible.com/ http://docs.ansible.com/) was not as easy as it could be, however, the install instructions were easily accessible from there, exactly where I'd expect them: Introduction > Installation > Latest Releases via Homebrew. Second, yeah, pip sucks, I try to avoid it at all costs. However, homebrew (http://brew.sh/ http://brew.sh/) is that one thing that all OSX power users should install as soon as they touch their workstation. 90% of the software - CLI or GUI - on my MacBook is installed and managed by homebrew. I cannot recommend it highly enough. In fact, I didn't even install Ansible through their instructions. I read about it somewhere else, and thought, "Hmmm, I wonder if this is in homebrew? `brew search ansible` Oh! Sweet! `brew install ansible`" Worked perfectly the first time, and I haven't fiddled with it since.
sazpaz 12y ago
This speaks to me as I coincidentally decided to start automating my provisioning to standardize to the same vagrant environment. I'm not a sysadmin or servers guy, but these days I happen to manage most of the devops and so getting servers and machines set up is my responsibility. Ansible docs are not great for getting started, and I pretty much just learned by skimming through some Playbook, and figuring it out their — often intuitive — purpose. After a few hours I got to a close to getting my machines set up with most of my Django stack: Nginx, Gunicorn, Redis, supervisor, celery, etc. The only thing that I couldn't properly set up was PostgreSQL. And here is something that most of these automation tools lack: debugging. When my postgres roles was failing, I had no clue how to even start debugging it, so after an hour or so, I just stepped back and went in to set it up manually. I feel some of these tools need to be more friendly with devops that don't have quite the same knowledge and experience than the target audience of sysadmins and ops people. That said, I learned a ton starting to use Ansible and automated a big chunk of the process.
mindslight 12y ago
I've looked into devops tools for administering my personal collection of a dozen machines or so, and don't understand what they offer. Every one I've tried, I've come to the conclusion that it is primarily designed for dealing with large numbers of homogeneous machines, and only serve to add overhead (new syntax, a level of file name indirection) when dealing with individualized ones. What I've ended up doing instead is keeping a simple tree of config files, which get run through a templating engine and then rsynced to the target host. For simple things there is zero overhead - etc/hosts is just the plain file that goes to all machines. But this system allows me to have eg one smartd.conf which contains all my disks, which is then split between machines. A few things currently fall through the cracks (users,groups), but overall rebuilding a machine is a relatively painless process. I feel like there must be a better more standardized way though (which is why I keep looking at these tools). Any comments/experiences/etc ?

32 points by lauriswtf 12y ago

One of the things that got me into ansible was simply doing ansible all -i ./ansible_hosts -m command -a "cat /etc/issue" From there I started going trough the possible options I could pass instead of each time doing a command, such as checking the network speed using a script etc.. As it might be useful for some I have next aliases set: export ANSIBLE_PATH="$HOME/web/clusters-servers/mycluster" alias a="ansible -i $ANSIBLE_PATH/hosts" alias ap="ansible-playbook -i $ANSIBLE_PATH/hosts" # for vagrant testing machines where I test playbooks before deploying alias atest="ansible -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s" alias aptest="ansible-playbook -i $ANSIBLE_PATH/vagrant/hosts --private-key=~/.vagrant.d/insecure_private_key -u vagrant -s" On of the (slightly) annoying things is after bringing up a new node you have to "bootstrap" the node so you can issue ansible commands on it, I have a simple script [1] for doing so but I'd love to hear how others do it! [1]: https://gist.github.com/larrybolt/85b59f47615be9fcb643 https://gist.github.com/larrybolt/85b59f47615be9fcb643

falcolas 12y ago
The raw command. Doesn't require anything to be installed on the target host, so you can set up a common playbook which will bootstrap the rest of Ansible.
0x0 12y ago
For a more low-tech solution, I've found cssh / csshX ("Cluster SSH") quite the timesaver. It pops up a bunch of xterms/Terminal.app windows and lets you type in one window while sending the keystrokes to all of the ssh sessions simultaneously.
- mpdehaan2 12y ago
  The problem with cluster SSH is it's not declarative, so if one server already has a package, and another does not, your servers will not behave the same if typing into all of them. This is why Ansible modules (and most other config tools) instead describe the desired end state, and perform the commands needed to get you to those states. Basically all config tools arose out of a need for solutions to replace parallel scripting with something more robust that accounted for potential variance between systems, and also to make things more push-button.
  - _delirium 12y ago
    I can see that for some kinds of admin tasks, but why package management? At least on Debian, the package manager is already declarative: There's an install/upgrade planner that figures out how to get from the current system state to the final desired state, given constraints such as packages' declared Requires/Conflicts/etc. lines. If the system is already at that state, it does nothing.
    - weaksauce 12y ago
      How often is setting up a package just running apt-get? There is an overwhelming majority of times where you have to change something in the config file somewhere. Ansible can do anything you want to provision your boxes.
- invertedohm 12y ago
  Indeed, cssh is great for getting a quick visual on a few servers, but you're pretty much limited to what will fit on your screen. Ansible really shines when you scale it up a bit further, and it grows well, allowing you to add bits and pieces as needed. The hosts/inventory in ansible as a source of truth for your infrastructure is also a nice feature. You could use a selection from your ansible inventory to open a cssh with 'top' running on all your webservers, for example.
killing_time 12y ago
For testing ansible playbooks/roles in a vagrant VM, I find a simpler way is to use Vagrant's built-in support for Ansible as a provisioner. This way, launching your test VM is just a `vagrant up` and running ansible against it (repeatedly, if needed) is `vagrant provision`. No need to pass inventory/key parameters at the shell this way - Vagrant calls Ansible with the right settings. One thing that's easy to miss is to make Ansible use sudo when running against the Vagrant VM, since the vagrant user by convention has passwordless sudo rights (ansible.sudo = true in the Vagrantfile). http://docs.vagrantup.com/v2/provisioning/ansible.html http://docs.vagrantup.com/v2/provisioning/ansible.html is a helpful reference doc.
spindritf 12y ago
You can point to the hosts file in ~/.ansible.cfg hostfile = ~/.ansible/hosts Or, if you run more than one project with multiple servers each, ansible will also read an ansible.config file in the current directory. You can have a separate hosts list, options, and playbooks for each set of servers. Then just cd ~/projects/whatever and ansible will work accordingly. You can put any given set of playbooks and configs into git and share with your team so that everyone is using ansible the same way. And that repo won't even carry any auth info.
mpdehaan2 12y ago
For Cloud services, SSH is "cloud native" meaning authorized keys are automatically injected by the cloud provider. If using metal installs, you can also set this up in your preseed or kickstart. However, you should never have to bootstrap a machine with another script to install things like python-apt, as you can just use the "raw" command to do this for anything that is not part of the box, see http://docs.ansible.com/raw_module.html http://docs.ansible.com/raw_module.html - so it's still self-bootstrapping. disclaimer: creator of project.
- larrybolt 12y ago
  Thanks! Didn't knew about this, or just read over it. I'm mostly using ansible for learning how to use it, don't have huge clusters yet to manage. Most of the hosts I have come from LEB (LowEndBox) offerings so therefor no way to deploy my own base image. And thank you for the work you put into ansible sir!

jamespo 12y ago

There's about 2 sentences describing what Ansible does in this article.

lauriswtf 12y ago
If you want to learn more about Ansible, the article links to the official docs http://docs.ansible.com/intro_getting_started.html http://docs.ansible.com/intro_getting_started.html.

willcodeforfoo 12y ago

I've long been accustomed to setting up servers one-command-at-a-time with SSH. I tried Chef and some of the others, but they seemed too complicated to get setup. Ansible's out of the box experience allows you to start with some simple but useful constructs then allow you to graduate to more and more complex usage. I started with some of the one-liners here, then setting up inventory files, then roles, some simple tasks and then (what really sold me) was templates. Even just having those configuration files in an SCM is a huge win, but I honestly squealed with joy when I was able to write a nginx configuration file that dynamically included backend servers given a particular role. And that provisioning a new backend server took a couple minutes and was automatically added to the load-balancing proxy! (not to mention was instantly sending metrics to a graphite/statsd backend because collectd was installed automatically too as part of a common role and configured to point to that log collecting server!) Magic!

Rezo 12y ago

Ansible is great, but could really have a better first time experience. The system itself is incredibly simple, but the docs and instructions are anything but, and seem to be modeled after Chef and Puppets which are frankly terrible. Instead, look at something like Vagrant as the shining example of how to do it right. Just compare http://www.ansible.com/ http://www.ansible.com/ with the excellent http://www.vagrantup.com/ http://www.vagrantup.com/ when it comes to downloading and running your first commands. After clicking around for a good while on the Ansbile site you finally find "To install on a Mac, make sure you have Homebrew, then run... ", yeah, no thanks. Oh, there's a "pip" command, if I happen to have that installed (hint: nobody on Mac has by default), but at least there's some more magic commands I can run to install that first. After running the "Looks Totally Safe To Me! (TM)" command "sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install ansible" it blows up on Mavericks with some error about something called paramiko and headers not being available. The attrition rate for newbies on Mac must be phenomenal, because you have to be really dedicated to give it another shot after this. What I'm getting at is, Python software distribution is an abysmal experience, always has been, and you should not subject your users to it. They're not Python developers, they're largely not even devs at all, they don't care about your pip's and brews. Compare this with Vagrant: Double click this here DMG, run "vagrant init", "vagrant up" and done, in nice _big_ friendly letters on the web site. Disclaimer: I use and love Ansible (on Linux), and want to see it succeed. Telling my Mac using ops team to check it out soured everyone's first impression right from the start. Rant over.

austinhyde 12y ago
As someone brand new to Ansible, and having been developing on OSX for years now, I'd just like to play devil's advocate. Finding the Ansible docs (http://docs.ansible.com/ http://docs.ansible.com/) was not as easy as it could be, however, the install instructions were easily accessible from there, exactly where I'd expect them: Introduction > Installation > Latest Releases via Homebrew. Second, yeah, pip sucks, I try to avoid it at all costs. However, homebrew (http://brew.sh/ http://brew.sh/) is that one thing that all OSX power users should install as soon as they touch their workstation. 90% of the software - CLI or GUI - on my MacBook is installed and managed by homebrew. I cannot recommend it highly enough. In fact, I didn't even install Ansible through their instructions. I read about it somewhere else, and thought, "Hmmm, I wonder if this is in homebrew? `brew search ansible` Oh! Sweet! `brew install ansible`" Worked perfectly the first time, and I haven't fiddled with it since.

sazpaz 12y ago

This speaks to me as I coincidentally decided to start automating my provisioning to standardize to the same vagrant environment. I'm not a sysadmin or servers guy, but these days I happen to manage most of the devops and so getting servers and machines set up is my responsibility. Ansible docs are not great for getting started, and I pretty much just learned by skimming through some Playbook, and figuring it out their — often intuitive — purpose. After a few hours I got to a close to getting my machines set up with most of my Django stack: Nginx, Gunicorn, Redis, supervisor, celery, etc. The only thing that I couldn't properly set up was PostgreSQL. And here is something that most of these automation tools lack: debugging. When my postgres roles was failing, I had no clue how to even start debugging it, so after an hour or so, I just stepped back and went in to set it up manually. I feel some of these tools need to be more friendly with devops that don't have quite the same knowledge and experience than the target audience of sysadmins and ops people. That said, I learned a ton starting to use Ansible and automated a big chunk of the process.

mindslight 12y ago

I've looked into devops tools for administering my personal collection of a dozen machines or so, and don't understand what they offer. Every one I've tried, I've come to the conclusion that it is primarily designed for dealing with large numbers of homogeneous machines, and only serve to add overhead (new syntax, a level of file name indirection) when dealing with individualized ones. What I've ended up doing instead is keeping a simple tree of config files, which get run through a templating engine and then rsynced to the target host. For simple things there is zero overhead - etc/hosts is just the plain file that goes to all machines. But this system allows me to have eg one smartd.conf which contains all my disks, which is then split between machines. A few things currently fall through the cracks (users,groups), but overall rebuilding a machine is a relatively painless process. I feel like there must be a better more standardized way though (which is why I keep looking at these tools). Any comments/experiences/etc ?