12 ms·

Ansible Techniques I Wish I’d Known Earlier

346 points by oedmarap 5y ago

linsomniac 5y ago
Gotta try that console and debugger, those weren't even on my radar!
- thibaut_barrere 5y ago
  These are both great. I still have an issue with console for things requiring a sudo access, for some reason, but it is still very handy! Another trick I use routinely is this, to display a given variable across all hosts: ANSIBLE_STDOUT_CALLBACK=json poetry run ansible all -m debug -a var=vault_ssl_cert | jq ".plays[0].tasks[0].hosts"
  - zwischenzug 5y ago
    Nice, I might steal that for part II...
- mrweasel 5y ago
  Me neither, but they are certainly something I can use. I still lack something that will tell me where a variable is sourced from.
- kimbernator 5y ago
  Unfortunately, console appears to be effectively ignored by the developers and it's pretty obvious when you're in it. There's just a lot of things that you can't do. It's a shame, too, because ansible is particularly frustrating to write for me coming from chef where everything is written in pure ruby and the REPL is basically just IRB with some extra stuff. Having an interactive environment to see how the data I'm handling is being transformed would be helpful, but ansible-console just doesn't do what I need most of the time.
reilly3000 5y ago
I wish I had read this a couple of weeks ago. I am new to Ansible but needed to use it to migrate a legacy system. I’m not really accustomed to long build times and kept running out of things to do in the interim while I was waiting for it to break.
mccoyc 5y ago
Another one for the list is `--start-at-task` and `--list-tasks`, for those tasks that are statically imported. If you have a task bomb out, you can take over from where you left off. Doesn't work in all playbooks, if facts need to be set earlier, but it's still very useful and saves tons of time. Obvious, but I wish I knew that one earlier too.
corty 5y ago
Maybe a stupid redundant slightly offtopic question, but I'll ask anyways: What is the best way to couple Ansible with some git repository to automatically apply changes while still being able to trace and debug?
- edaemon 5y ago
  This definitely isn't the best way because it's far from painless, but we use GitHub Actions to run some Ansible playbooks. When there's a problem that needs debugging we add a tmate[0] session with mxschmitt's action[1] so we can step through the playbook, etc. [0]: https://tmate.io/ https://tmate.io/ [1]: https://github.com/mxschmitt/action-tmate https://github.com/mxschmitt/action-tmate
  - Datagenerator 5y ago
    Or Jenkins pipelined steps which might include setting up the pipenv to include the setup of Ansible in your git as requirement file. So it starts with Groovy and switches to Ansible etc
- WFHRenaissance 5y ago
  Build/Buy a real CI/CD system for this TBH. Once you hit a certain level of complexity, this sort of deployment strategy will mostly just amplify pain.
- xorcist 5y ago
  Tower? Open sourced by Red Hat as AWX, it is a central Ansible hub for exactly that purpose.
  - Spivak 5y ago
    Tower/AWX are bad products and they should feel bad. It’s actually easier to run Ansible via any CI system than it is to use Tower. I get what the devs are trying to do but Tower wants to own so much of the process that it’s basically not Ansible anymore. You pretty much have to write your code specifically for Tower (which means a total rewrite in mosts shops because who starts with Tower?)
whirlwin 5y ago
I have used Ansible for some years. I recently switched jobs to a company having no automation on VMs. And we are migrating our workload to managed services in the cloud (managed Kubernetes), I have to say I'm relieved I don't have to deal with Ansible again. It's probably very efficient in many ways, but just copying a file/directory with the correct options is a bit of pain in Ansible. Having not to worry about VMs anymore is a huge win for me at least. /OffTopicEnd
- yjftsjthsd-h 5y ago
  > just copying a file/directory with the correct options is a bit of pain in Ansible. How so? Do you mean you had trouble remembering what arguments to pass to the copy module? (I'm not sure what other options there are to be a pain)
  - justin_oaks 5y ago
    I'm not the one you're replying to, but I one problem I've had with Ansible directory copying is that it's terribly slow. Using the "state" parameter for files and directories is also rather odd. You specify "state: absent" to delete, "state: directory" to create a directory. It makes sense considering the declarative nature of Ansible, but it's unfamiliar to those used to writing shell scripts.
    - yjftsjthsd-h 5y ago
      Yes, ansible in general is painfully slow; I guess I assumed "painful" meant hard to write
avaika 5y ago
I'm using ansible for some years already. The biggest game changer for me was ansible molecule [0]. I integrated it to the repo, where I store my code and it just runs all the test cases for my code. It saved me a bunch of hours of investigation in test environments before I even released the code. Highly recommend to try it even for a small project. Apart of that play with ansible strategies if you have more than one server to apply the role. It really might save you some minutes of a runtime. [0]: https://molecule.readthedocs.io/en/latest/ https://molecule.readthedocs.io/en/latest/
- willejs 5y ago
  This is really great advice. Chef was doing this years ago with test kitchen and inspec (and serverspec). Most community cookbooks came with extensive test suites, and in turn made it easy to encourage people to write tests for cookbooks in house. It seems theres a real lack of testing in opensource Ansible playbooks a lot of the time, and a lot of times when I see it used in organizations.
- dheera 5y ago
  After using Ansible for a while at a company I got massively turned off by the fact the version in apt-get doesn't work and if you go to ansible.com there isn't an obvious download button, there's only a button to register for a goddamn free event. There should be a big fucking red "DOWNLOAD" button at the center of ansible.com, and not a goddamn GDPR intrusion and free event. And then if I search Google for "ansible download" I land at a documentation website with 50 billion links. No. F that. Back to shell scripts. Git clone and run "./install.sh". Most of the things I do are containerized anyway, and the heavy lifting is done by Dockerfile not Ansible, so the install.sh isn't actually that involved. Hey Ansible if you're reading this, head over to https://google.com/chrome/ https://google.com/chrome/. Make your front page look like that with a big obvious download button and no popups.
  - 1stranger 5y ago
    Google "install ansible" then follow the instructions for your OS. I'm sorry you can't read.
rakshazi 5y ago
ansible-console is a treasure! I wish I saw that article years ago...
gizdan 5y ago
Ansible is abysmal. I don't know why anyone still chooses it. It's a mess of yaml and what feels like a million yaml files that is always extremely hard to follow. Honestly writing some python, or bash is a lot easier to follow, read, and understand. The only thing it has going for it is the inventory system. I wish ansible would die already. The tips mentioned in the article are helpful, but still beyond the initial development of roles, maintaining roles is a pain no one can alleviate. It is the perl of automation. I wish people would stop using it. I've yet to start a job where people haven't regretted going towards ansible.
- _red 5y ago
  >bash and python Neither of those of idempotent. It seems likely you are coming at this from the perspective of "when I find an ansible playbook on github its hard to follow whats going on". There is a certain truth to that, but much of its complexity is probably related to said playbook developers trying to abstract away specifics and create a playbook that works on a variety of distributions / configuration options. Ansible can be as simple as a single file that proceeds sequentially like a bash script.
  - notabee 5y ago
    Yes, it can get hairy but I just have a hard time seeing an adhoc mess of python and bash doing the equivalent amount of lifting without becoming an even bigger hairball. And the thing is, you can also just run those same bash and python scripts using Ansible if you really want to, just to keep them orchestrated properly.
    - Hackbraten 5y ago
      > you can also just run those same bash and python scripts using Ansible if you really want to, just to keep them orchestrated properly Most of the time (i.e. if your Bash script generates a known file), Ansible also gives you idempotency for free, thanks to the ’creates’ parameter.
  - voidfunc 5y ago
c7DJTLrn 5y ago
At work we use Puppet and I have to say that although the DSL is much more expressive and nicer to use than YAML, Puppet's tooling sucks in comparison. There isn't really any way to test your Puppet code that I know of, you just have to apply it and check if it all looks good. Puppet doesn't have a set of good practices established either. Different companies use completely different repository structures, compiler settings, linter settings. Modules for managing common software like Docker are low quality and riddled with issues. Ansible and Puppet claim to be declarative but it's a lie. Configuration management really needs a rethink. I know Nix exists but that has a huge barrier to entry and companies don't want to use something that isn't battle tested. In my view, the ideal config management tool would be fast, testable, and provably idempotent. It would also be as far away as possible from the Python/Ruby swamp that Ansible and Puppet are stuck in.
- inshadows 5y ago
  Puppet is declarative, isn't it? You declare states and relationships, and it computes plan of action. Ansible is not declarative at all though, and it's not even funny how almost every fan says it is. It's just a sequence of actions.
  - c7DJTLrn 5y ago
    Sadly, no. Puppet masquerades as being declarative just like Ansible does. It's very easy to get into a situation where your Puppet class is just a fancy shell script which works the first time and then fails because a file doesn't exist or something. There is nothing to lint for this or to protect against this.
    - LIV2 5y ago
      At a previous job we used beaker tests as part of our pipeline to test puppet modules & environments before we would merge to prod. One of the things this tests is that the module applies and doesn't try to change anything / error out on subsequent runs. The documentation sucks though imo, every different guide you find will show a completely different way of doing it
usrme 5y ago
It's a damn shame it took until this article for me (and it appears some others) to learn about the console and the debugger; those seem like massive time savers! My own biggest helper has been to use "ANSIBLE_STDOUT_CALLBACK=yaml" wherever "ansible-playbook" is called (https://jpmens.net/2021/03/12/alter-ansible-s-output-on-debugging/ https://jpmens.net/2021/03/12/alter-ansible-s-output-on-debu...). It makes the verbose output much more readable and is especially handy for tasks that can spew thousands of lines at ones (e.g. "apt update/upgrade").
- Hackbraten 5y ago
  Thanks, that setting helps a lot! Whoever prefers configuration files over environment variables can get the same effect by adding to their ~/.ansible.cfg: [defaults] stdout_callback = yaml
  - HeadlessChild 5y ago
    One needs to lookout for their config file being overwritten [1] if another ansible.cfg exist. In my case my ~/.ansible.cfg config got overwritten by a local ./ansible.cfg (as per the order the config files are read [2]) in a project directory. In my case I workaround it by _merging_ all potential config files into one at runtime using `crudini` as demonstrated by rsguhr [3]. [1] https://github.com/ansible/proposals/issues/35 https://github.com/ansible/proposals/issues/35 [2] https://docs.ansible.com/ansible/2.4/intro_configuration.html#configuration-file https://docs.ansible.com/ansible/2.4/intro_configuration.htm... [3] https://gitlab.com/-/snippets/1851171 https://gitlab.com/-/snippets/1851171
- srvg 5y ago
  fwiw, I myself only found out about that trick sone months ago...
amanzi 5y ago
The `--step` flag will definitely be useful for me, didn't know about that. My one tip I'll add is to use the `--diff` argument when running a playbook to print out the specific changes made for each step. For debugging, use `--diff --check` to see all the changes that will be made - useful if it's been a while since you last run your playbook over a host.
- fanf2 5y ago
  The Ansible —diff and —check options were the main reason I started using it in 2013; the other reason was that it’s agentless, so I didn’t need a special vm to run it on and I could control when it might change things.
- lazyweb 5y ago
  Yes, usually working with "--diff --check" flags as well, and did not know about "--step" either. Will incorporate that into my workflow for sure.
moreati 5y ago
Over the years I've tweeted Ansible tips, and Ansible Gotchas for my own memory and others https://twitter.com/search?q=from%3Amoreati%20ansible%20tip&f=live https://twitter.com/search?q=from%3Amoreati%20ansible%20tip&... https://twitter.com/search?q=from%3Amoreati%20ansible%20gotcha&src=typed_query&f=live https://twitter.com/search?q=from%3Amoreati%20ansible%20gotc...

346 points by oedmarap 5y ago

linsomniac 5y ago

Gotta try that console and debugger, those weren't even on my radar!

thibaut_barrere 5y ago
These are both great. I still have an issue with console for things requiring a sudo access, for some reason, but it is still very handy! Another trick I use routinely is this, to display a given variable across all hosts: ANSIBLE_STDOUT_CALLBACK=json poetry run ansible all -m debug -a var=vault_ssl_cert | jq ".plays[0].tasks[0].hosts"
- zwischenzug 5y ago
  Nice, I might steal that for part II...
mrweasel 5y ago
Me neither, but they are certainly something I can use. I still lack something that will tell me where a variable is sourced from.
kimbernator 5y ago
Unfortunately, console appears to be effectively ignored by the developers and it's pretty obvious when you're in it. There's just a lot of things that you can't do. It's a shame, too, because ansible is particularly frustrating to write for me coming from chef where everything is written in pure ruby and the REPL is basically just IRB with some extra stuff. Having an interactive environment to see how the data I'm handling is being transformed would be helpful, but ansible-console just doesn't do what I need most of the time.

reilly3000 5y ago

I wish I had read this a couple of weeks ago. I am new to Ansible but needed to use it to migrate a legacy system. I’m not really accustomed to long build times and kept running out of things to do in the interim while I was waiting for it to break.

mccoyc 5y ago

Another one for the list is `--start-at-task` and `--list-tasks`, for those tasks that are statically imported. If you have a task bomb out, you can take over from where you left off. Doesn't work in all playbooks, if facts need to be set earlier, but it's still very useful and saves tons of time. Obvious, but I wish I knew that one earlier too.

corty 5y ago

Maybe a stupid redundant slightly offtopic question, but I'll ask anyways: What is the best way to couple Ansible with some git repository to automatically apply changes while still being able to trace and debug?

edaemon 5y ago
This definitely isn't the best way because it's far from painless, but we use GitHub Actions to run some Ansible playbooks. When there's a problem that needs debugging we add a tmate[0] session with mxschmitt's action[1] so we can step through the playbook, etc. [0]: https://tmate.io/ https://tmate.io/ [1]: https://github.com/mxschmitt/action-tmate https://github.com/mxschmitt/action-tmate
- Datagenerator 5y ago
  Or Jenkins pipelined steps which might include setting up the pipenv to include the setup of Ansible in your git as requirement file. So it starts with Groovy and switches to Ansible etc
WFHRenaissance 5y ago
Build/Buy a real CI/CD system for this TBH. Once you hit a certain level of complexity, this sort of deployment strategy will mostly just amplify pain.
xorcist 5y ago
Tower? Open sourced by Red Hat as AWX, it is a central Ansible hub for exactly that purpose.
- Spivak 5y ago
  Tower/AWX are bad products and they should feel bad. It’s actually easier to run Ansible via any CI system than it is to use Tower. I get what the devs are trying to do but Tower wants to own so much of the process that it’s basically not Ansible anymore. You pretty much have to write your code specifically for Tower (which means a total rewrite in mosts shops because who starts with Tower?)

whirlwin 5y ago

I have used Ansible for some years. I recently switched jobs to a company having no automation on VMs. And we are migrating our workload to managed services in the cloud (managed Kubernetes), I have to say I'm relieved I don't have to deal with Ansible again. It's probably very efficient in many ways, but just copying a file/directory with the correct options is a bit of pain in Ansible. Having not to worry about VMs anymore is a huge win for me at least. /OffTopicEnd

yjftsjthsd-h 5y ago
> just copying a file/directory with the correct options is a bit of pain in Ansible. How so? Do you mean you had trouble remembering what arguments to pass to the copy module? (I'm not sure what other options there are to be a pain)
- justin_oaks 5y ago
  I'm not the one you're replying to, but I one problem I've had with Ansible directory copying is that it's terribly slow. Using the "state" parameter for files and directories is also rather odd. You specify "state: absent" to delete, "state: directory" to create a directory. It makes sense considering the declarative nature of Ansible, but it's unfamiliar to those used to writing shell scripts.
  - yjftsjthsd-h 5y ago
    Yes, ansible in general is painfully slow; I guess I assumed "painful" meant hard to write

avaika 5y ago

I'm using ansible for some years already. The biggest game changer for me was ansible molecule [0]. I integrated it to the repo, where I store my code and it just runs all the test cases for my code. It saved me a bunch of hours of investigation in test environments before I even released the code. Highly recommend to try it even for a small project. Apart of that play with ansible strategies if you have more than one server to apply the role. It really might save you some minutes of a runtime. [0]: https://molecule.readthedocs.io/en/latest/ https://molecule.readthedocs.io/en/latest/

willejs 5y ago
This is really great advice. Chef was doing this years ago with test kitchen and inspec (and serverspec). Most community cookbooks came with extensive test suites, and in turn made it easy to encourage people to write tests for cookbooks in house. It seems theres a real lack of testing in opensource Ansible playbooks a lot of the time, and a lot of times when I see it used in organizations.
dheera 5y ago
After using Ansible for a while at a company I got massively turned off by the fact the version in apt-get doesn't work and if you go to ansible.com there isn't an obvious download button, there's only a button to register for a goddamn free event. There should be a big fucking red "DOWNLOAD" button at the center of ansible.com, and not a goddamn GDPR intrusion and free event. And then if I search Google for "ansible download" I land at a documentation website with 50 billion links. No. F that. Back to shell scripts. Git clone and run "./install.sh". Most of the things I do are containerized anyway, and the heavy lifting is done by Dockerfile not Ansible, so the install.sh isn't actually that involved. Hey Ansible if you're reading this, head over to https://google.com/chrome/ https://google.com/chrome/. Make your front page look like that with a big obvious download button and no popups.
- 1stranger 5y ago
  Google "install ansible" then follow the instructions for your OS. I'm sorry you can't read.

rakshazi 5y ago

ansible-console is a treasure! I wish I saw that article years ago...

gizdan 5y ago

Ansible is abysmal. I don't know why anyone still chooses it. It's a mess of yaml and what feels like a million yaml files that is always extremely hard to follow. Honestly writing some python, or bash is a lot easier to follow, read, and understand. The only thing it has going for it is the inventory system. I wish ansible would die already. The tips mentioned in the article are helpful, but still beyond the initial development of roles, maintaining roles is a pain no one can alleviate. It is the perl of automation. I wish people would stop using it. I've yet to start a job where people haven't regretted going towards ansible.

_red 5y ago
>bash and python Neither of those of idempotent. It seems likely you are coming at this from the perspective of "when I find an ansible playbook on github its hard to follow whats going on". There is a certain truth to that, but much of its complexity is probably related to said playbook developers trying to abstract away specifics and create a playbook that works on a variety of distributions / configuration options. Ansible can be as simple as a single file that proceeds sequentially like a bash script.
- notabee 5y ago
  Yes, it can get hairy but I just have a hard time seeing an adhoc mess of python and bash doing the equivalent amount of lifting without becoming an even bigger hairball. And the thing is, you can also just run those same bash and python scripts using Ansible if you really want to, just to keep them orchestrated properly.
  - Hackbraten 5y ago
    > you can also just run those same bash and python scripts using Ansible if you really want to, just to keep them orchestrated properly Most of the time (i.e. if your Bash script generates a known file), Ansible also gives you idempotency for free, thanks to the ’creates’ parameter.
- voidfunc 5y ago

c7DJTLrn 5y ago

At work we use Puppet and I have to say that although the DSL is much more expressive and nicer to use than YAML, Puppet's tooling sucks in comparison. There isn't really any way to test your Puppet code that I know of, you just have to apply it and check if it all looks good. Puppet doesn't have a set of good practices established either. Different companies use completely different repository structures, compiler settings, linter settings. Modules for managing common software like Docker are low quality and riddled with issues. Ansible and Puppet claim to be declarative but it's a lie. Configuration management really needs a rethink. I know Nix exists but that has a huge barrier to entry and companies don't want to use something that isn't battle tested. In my view, the ideal config management tool would be fast, testable, and provably idempotent. It would also be as far away as possible from the Python/Ruby swamp that Ansible and Puppet are stuck in.

inshadows 5y ago
Puppet is declarative, isn't it? You declare states and relationships, and it computes plan of action. Ansible is not declarative at all though, and it's not even funny how almost every fan says it is. It's just a sequence of actions.
- c7DJTLrn 5y ago
  Sadly, no. Puppet masquerades as being declarative just like Ansible does. It's very easy to get into a situation where your Puppet class is just a fancy shell script which works the first time and then fails because a file doesn't exist or something. There is nothing to lint for this or to protect against this.
  - LIV2 5y ago
    At a previous job we used beaker tests as part of our pipeline to test puppet modules & environments before we would merge to prod. One of the things this tests is that the module applies and doesn't try to change anything / error out on subsequent runs. The documentation sucks though imo, every different guide you find will show a completely different way of doing it

usrme 5y ago

It's a damn shame it took until this article for me (and it appears some others) to learn about the console and the debugger; those seem like massive time savers! My own biggest helper has been to use "ANSIBLE_STDOUT_CALLBACK=yaml" wherever "ansible-playbook" is called (https://jpmens.net/2021/03/12/alter-ansible-s-output-on-debugging/ https://jpmens.net/2021/03/12/alter-ansible-s-output-on-debu...). It makes the verbose output much more readable and is especially handy for tasks that can spew thousands of lines at ones (e.g. "apt update/upgrade").

Hackbraten 5y ago
Thanks, that setting helps a lot! Whoever prefers configuration files over environment variables can get the same effect by adding to their ~/.ansible.cfg: [defaults] stdout_callback = yaml
- HeadlessChild 5y ago
  One needs to lookout for their config file being overwritten [1] if another ansible.cfg exist. In my case my ~/.ansible.cfg config got overwritten by a local ./ansible.cfg (as per the order the config files are read [2]) in a project directory. In my case I workaround it by _merging_ all potential config files into one at runtime using `crudini` as demonstrated by rsguhr [3]. [1] https://github.com/ansible/proposals/issues/35 https://github.com/ansible/proposals/issues/35 [2] https://docs.ansible.com/ansible/2.4/intro_configuration.html#configuration-file https://docs.ansible.com/ansible/2.4/intro_configuration.htm... [3] https://gitlab.com/-/snippets/1851171 https://gitlab.com/-/snippets/1851171
srvg 5y ago
fwiw, I myself only found out about that trick sone months ago...

amanzi 5y ago

The `--step` flag will definitely be useful for me, didn't know about that. My one tip I'll add is to use the `--diff` argument when running a playbook to print out the specific changes made for each step. For debugging, use `--diff --check` to see all the changes that will be made - useful if it's been a while since you last run your playbook over a host.

fanf2 5y ago
The Ansible —diff and —check options were the main reason I started using it in 2013; the other reason was that it’s agentless, so I didn’t need a special vm to run it on and I could control when it might change things.
lazyweb 5y ago
Yes, usually working with "--diff --check" flags as well, and did not know about "--step" either. Will incorporate that into my workflow for sure.

moreati 5y ago

Over the years I've tweeted Ansible tips, and Ansible Gotchas for my own memory and others https://twitter.com/search?q=from%3Amoreati%20ansible%20tip&f=live https://twitter.com/search?q=from%3Amoreati%20ansible%20tip&... https://twitter.com/search?q=from%3Amoreati%20ansible%20gotcha&src=typed_query&f=live https://twitter.com/search?q=from%3Amoreati%20ansible%20gotc...