10 ms·

Ansible Alternatives in Python

146 points by alexeiz 6y ago

linsomniac 6y ago
I wasn't aware of the VSCode Ansible plugin, I've got to check that out. Pyinfra examples he shows don't look that different from Ansible YAML. ("apt('nginx', update=True, present=True)") I'm pretty happy with Ansible, with one exception: speed. I may play with Pyinfra, but I'm heavily invested in Ansible. But if Pyinfra is fast, I might try replacing our Fabric with it. I get pushback about how long Ansible takes for our deploys vs. Fabric.
- larsnystrom 6y ago
  So I looked at the VSCode Ansible plugin and it says it is authored by "Microsoft"[1], but the repository is owned by a Github organization called "Visual Studio China"[2] and the plugin identifier is "vscoss.vscode-ansible", when other Microsoft plugins usually has identifiers in the "ms-vscode" namespace (for example the TSLint and C/C++ extensions) or "ms-vscode-remote" or some similar namespace starting with "ms". So why is this ansible plugin such an outlier? How can I be sure that this plugin is actually authored by Microsoft and not by some other entity? Maybe I'm just being paranoid. Edit: Ok so if you click the name "Microsoft" in the vscode marketplace it will take you to a list of Microsoft authored plugins, which I guess means the organization name in the marketplace is verified somehow. So yes, I was just being paranoid. [1] https://marketplace.visualstudio.com/items?itemName=vscoss.vscode-ansible https://marketplace.visualstudio.com/items?itemName=vscoss.v... [2] https://github.com/VSChina/vscode-ansible https://github.com/VSChina/vscode-ansible
- deivid 6y ago
  try the mitogen plugin. we went from 2 hours to 15 minutes in run time
ShepherdKing 6y ago
There is no mention of Python Bundlewrap (https://bundlewrap.org/ https://bundlewrap.org/), which I have really liked, though I don't know how speed and scalability compares. I believe Bundlewrap will welcome contributors for these enhancements.
simonw 6y ago
pyinfra looks interesting. The thing I want it a way to keep configuration for a single server in a git repo, and have that server periodically run "git pull" and apply any changes. I ended up bodging together some custom scripts to do this but I really didn't want to. Maybe I can get pyinfra to do this? I don't want to have to run it on another machine and run commands over ssh though.
- moreati 6y ago
  For reference, in Ansible land the two approaches to this would be https://docs.ansible.com/ansible/latest/cli/ansible-pull.html https://docs.ansible.com/ansible/latest/cli/ansible-pull.htm... or https://github.com/ansible/awx https://github.com/ansible/awx (upstream project of Ansible Tower)
  - simonw 6y ago
    I had missed ansible-pull, looks like the exact pattern I want. Thanks!
chopraaa 6y ago
If you feel YAML is too complicated for you, a host of problems await for anything you want to do in infrastructure... Ansible has it's own set of features that are largely abstracted out by it's YAML playbook syntax. You can run it across any (or almost) system, from Linux to Windows and even networking devices. You can create modules in any language you want. For speeding up Ansible runs, a number of optimisations can be made including using Mitogen https://mitogen.networkgenomics.com/ansible_detailed.html https://mitogen.networkgenomics.com/ansible_detailed.html Alternatives listed in the blog post don't even come close to the speed and power of Ansible.
- sa46 6y ago
  > Ansible has its own set of features that are largely abstracted out by its YAML playbook syntax. I don't understand what YAML has to do with abstraction. The example from pyinfra is pretty much the same as the ansible equivalent. Whether the feature set is comparable is a different question (I agree with you, it's probably nowhere close to ansible). apt.packages( {'Install iftop'}, 'iftop', sudo=True, update=True, ) > If you feel YAML is too complicated for you Ansible YAML is a poor substitute for a real language once you add conditional logic to the mix. Another huge downside is that composition in YAML is much harder than in any programming language. A Python (or dhall, or cue) library that compiled to ansible would be pretty swell.
  - kitd 6y ago
    Ansible YAML is a poor substitute for a real language once you add conditional logic to the mix. Another huge downside is that composition in YAML is much harder than in any programming language Given that the average devops person is already stuck with yaml, many of those problems are addressed by jsonnet. But I agree, having it in the language would be preferable. I think it stems from the desire to make infrastructure declarative, which isn't really the case when your declarations are dynamic.
    - dynamite-ready 6y ago
      I've written a lot of Ansible in the past year. While I'm happy with the work, I'd admit to looking at some of the logic I've had to put together with clunky, esoteric constructs like `with_nested_items` and the super colloquial variable system, while thinking "it's great to be able to do this stuff, but htf am I going to explain this code to a newbie?". I'd like some kind of 'sensible', backwards compatible layer in Ansible that would allow me to rewrite stuff in raw Python, with 'normal' code, where it makes sense, but keep YAML for constructs that do actually look and work like lists of basic tasks.
torvald 6y ago
If one is not tied to python, NixOps is a breath of fresh air.
BossingAround 6y ago
Kubernetes already requires YAML. If you use Kubernetes, you might as well use Ansible. How YAML got so popular is a mystery to me. It's a huge pain. Then again, I've given up on fighting it, and simply learned how to use it, since I can't escape Kubernetes anyways.
- nsonha 6y ago
  What kind of logic is that? Lots of things use yaml they usually have some sort of templating engine on top, so whether the tool is friendly/adequate is depending on the templating capacity not the fact that it uses yaml, or any markup language for that matter. Also people are mostly arguing for code vs markup. yml, json, xml etc are all the same. Yml just happens to be more compact which helps with visibility in a big file.
- lazyant 6y ago
  > How YAML got so popular is a mystery to me. It's good at being human-readable. It's basically json with indents and comments. What other configuration format is out there that allows for tree structure and it's easier on humans? (not XML or json).
  - randallsquared 6y ago
    I'd say YAML's good at seeming human-readable... JSON has some surprises and a lot of limitations, but is so much simpler than YAML that it seems worth putting up with them.
  - raghava 6y ago
    > good at being human-readable If we consider high composability, non-ambiguity, clarity, expressability (even a word?) as aspects of "human readability" - YAML doesn't go very far in the scale across many of these. A language, with slightly more appealing (yet non-limiting for programmablity) declarative constructs seem very helpful for needs like this. It would be great to see any spider-web-graph of all these solutions across those aspects and thus help someone to take a wiser decision.
- unionpivo 6y ago
theonemind 6y ago
"In Python" seems like a weird constraint on the problem. Python doesn't really read like a great deployment configuration language. https://en.wikipedia.org/wiki/Infrastructure_as_code#Community_content https://en.wikipedia.org/wiki/Infrastructure_as_code#Communi... the space has a lot of players. For upgrading like one VPS and some home servers, I'd probably go with Ansible. It probably has the lowest barrier to entry, and getting an IDE just solves the problem that you created by wanting something in Python. Basic config management doesn't need an IDE.
- divbzero 6y ago
  “In Python” does seem like an odd constraint. I take the OP’s intent to be “let’s find something imperative not declarative”. If that’s the case, I’d love to see the idempotent functionality of Ansible’s modules (e.g. lineinfile) made available as simple CLI commands.
  - stonith 6y ago
    You could kind of do this using ansible-runner to dynamically generate a playbook that does what you want.
  - inshadows 6y ago
    I assume you consider Ansible to be "declarative". By that definition of "declarative", the following Bash script is also "declarative": #!/bin/bash rm -f /etc/bar mkdir -p /etc/conf.d cp -u /srv/src.txt /etc/conf.d/ because Anisble playbook is really just bunch of following statements if (!stuff_exists(stuff)) create_stuff() which the above utilities in Bash script do with the provided flags. Ansible "declarativeness" isn't anything worthy of worship. It's plain dumb definition of sequence of steps to take with added guard for each step. So let's be honest and stop pretending that this "declarativeness" is anything novel or of any value worthy of writing sequence of (guarded) commands in ad-hoc YAML language. Just add some sugar for the pattern above to a decent language and that's all that's needed.
  - Bystroushaak 6y ago
    Op is python programmer who is using python for everything for more than 13 years, so there you have the reason. I am also writing my own programming language .. in python: https://github.com/Bystroushaak/tinySelf https://github.com/Bystroushaak/tinySelf
bamboozled 6y ago
How hard is YAML to use really? The world has much more interesting problems to solve than a decent markup language. I use Ansible with zero issues daily as do many in our team. It’s perplexing coders fined it difficult or hard to use to the point where something as useful as Ansible becomes undesirable because it uses YAML? I hate to say it, but the “alternatives” are just the least good somewhat similar products.
- _frkl 6y ago
  Spot on. If you have an opinion about YAML and feel you need to make that know on the internet, I usually take it as a sign that I'll probably don't need to read the rest of your comment...
- fake-name 6y ago
  The problem isn't YAML specifically, it's that to do everything ansible does they have to extend YAML into a crappy programming language in-and-of-itself. If they're going to do that, the configuration files should just use a actual programming language. Just make them actual python files (or ruby files, or whatever). From the famous quote: > Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. The problem here isn't that they're using lisp (or python, or yaml, or whatever), it's that they're re-implementing a very mature language with lots of supporting infrastruture poorly with lots of bugs and corner cases, rather then just using the relevant (open source) programming language itself.
  - GordonS 6y ago
    > The problem isn't YAML specifically, it's that to do everything ansible does they have to extend YAML into a crappy programming language in-and-of-itself. I don't recall how ansible extends yaml, but Microsoft does something that sounds similar with their JSON-based ARM templates (which are for deploying cloud resources), adding functions for loops, parameters and variables. The result is predictably horrible. There are however at least APIs available for C# and others, so it's not the only automation option.
- 6y ago
stevenjohns 6y ago
Probably worth checking out Mitogen[0] if you want to take away some of the pain points with Ansible. It’s an extremely well built product that really complements Ansible. [0] https://mitogen.networkgenomics.com/ https://mitogen.networkgenomics.com/
- Bystroushaak 6y ago
  I've seen Mitogen, but that to me looks like lowlevel ssh paralelization library. Great project, but it doesn't seem to provide any abstractions, for example declarative package manager wrapper, that would work on multiple linux versions.
fermigier 6y ago
I have tried or used most of the projects cited in the blog post. I pretty much agree with the conclusion that Pyinfra looks the most promising at this point. I've also tried Bundlewrap (https://bundlewrap.org/ https://bundlewrap.org/) a couple of years ago though I lack the perspective to compare them. Another interesting project not cited in the post is Nuka (https://doc.bearstech.com/nuka/ https://doc.bearstech.com/nuka/). Some personal perspectives: I have used Ansible in the past and didn't like it. I have used Fabric and Fabtools many years ago (Fabtools happens to be written by a friend) and I agree that something like Fabtools is needed to make Fabric relevant. Fabric has a completely revamped version, which was quite long in the making, but not fairly stable, but AFAIK Fabric 2 doesn't (yet) have something similar to what Fabtools was to Fabric 1. I had great hope for opsmop (https://github.com/opsmop/opsmop https://github.com/opsmop/opsmop) by Ansible's creator, which was supposed to address Ansible's shortcomings, but he dropped the project (citing lack of traction) a few months after starting it in 2018. At this point I'm sticking to Fabric 2, but I'm still looking for alternatives. I'm also wondering if Augeas (https://augeas.net/ https://augeas.net/ - a configuration editing tool which parses configuration files for a large set of software in their native formats and transforms them into a tree) could be used in combination with one the of tools mentioned in the post to ease the parts related to managing configuration.
- Bystroushaak 6y ago
  Thanks for the links. I am kinda surprised by the activity on this blogpost, it's just something that I've put together in an hour or so, mostly as notes for myself. I do "exporations" where I put bunch of related stuff into the wiki node and then take an evening to go over them and do some evaluation. It wasn't really meant as a comprehensive study.
idoby 6y ago
For all its flaws, Docker does solve a lot of the pain the author mentions. Deploying postgres, for example, with a config file and keeping it running is really very easy with Docker Compose. Not affiliated with Docker except for using it, and while I do have some thoughts about design choices made by Docker, it's still a very good tool.
- dynamite-ready 6y ago
  I don't see Docker as an alternative to a configuration manager. For one, turning to containers represents a distinct architectural choice, with a clear set of tradeoffs, that a developer may or may not want to introduce to a project.
  - idoby 6y ago
    It sure does, but even if you choose not to take advantage of container tech, it's still a very easy way to deploy stuff on a single machine. Not sure why the downvotes.
    - dynamite-ready 6y ago
      I didn't downvote. I see your point, but it's a bit of a digression, tbf.
- Bystroushaak 6y ago
  I do use Docker, but that is a tool for completely different purpose. I need something else for configuration of all my physical machines; virtual server, physical server, several notebooks and so on.
INTPenis 6y ago
The author of Ansible actually tried to make a Python3 DSL based Ansible alternative called opsmop.[1] I'm not sure why but it was discontinued. I would think that if anyone could make that work it would be that particular person with their previous experience from Cobbler and Ansible. I started using Ansible when it was brand spanking new and what I've noticed over the years is that people often over complicate their Ansible use. Sometimes it's better to out source something to an RPM or a container than to use Ansible to do every single thing. 1. https://github.com/opsmop/opsmop https://github.com/opsmop/opsmop
- rezonant 6y ago
  Agreed that people tend to overcomplicate their Ansible configs. It is also OK to write a script and run it from Ansible as part of your playbook if you have a complex use case.
- jabl 6y ago
  Here's an explanation by the sensible/opsmop creator: https://medium.com/@michaeldehaan/revisiting-systems-management-assumptions-5eee2d5fa82e https://medium.com/@michaeldehaan/revisiting-systems-managem...
  - INTPenis 6y ago
    Thanks, I just skimmed it and found this. >Nobody wants to keep up with a configuration management tool, especially learn another new one (which is why creating opsmop failed), learn another YAML dialect, or maintain a full active cloud abstraction layer running on top of their cloud itself.
mongol 6y ago
I think there would be value in providing idempotent operations to all common Linux tools. This often exist, but sometimes with weird side effects. Sometimes an add operation returns a failure code because item already exist and so on. It would be quite worthwhile to go through the Ansible documentation and document a command line alternative for each. Then it would be easier to use a regular programming language to achieve the same effects.
ctippett 6y ago
I’ve gotten a lot of mileage out of using cloud-init to bootstrap VMs. The configuration is in YAML, but doesn’t require any other client side tooling. Most major distributions (incl. the public clouds) all support it and make it available in their machine images. It integrates nicely into Terraform too. https://cloudinit.readthedocs.io https://cloudinit.readthedocs.io
syllogism 6y ago
Fabric was a casualty of the Python 2/3 split. Unicode isn't a great default for the shell interaction so the project took a long time to be ported, and by then it lost all momentum. The successor project, Invoke, was finally released a year or two ago, but now it's unmaintained too.
- acdha 6y ago
  I think it’s more that Jeff Forcier is one – very productive – person and while tons of places used Fabric there were far fewer who contributed significantly to supporting it. I much prefer Python to programming in a YAML derivative with multiple bolt-on extension mechanisms but at the end of the day Ansible brings a lot to the table because there are a bunch of people employed by Red Hat to work on it full-time.
  - syllogism 6y ago
    Yes that's true, but turning it around: the task is too big (and also sucks too much) for one person to do mostly alone. And also like, why? There would've been a more active community of companies depending on it and it wouldn't have blocked out on him if it had maintained the momentum it had.
    - acdha 6y ago
      I was talking about it at the peak of that momentum. This is a common problem in open source: companies often don’t contribute or only support specific things they need. It’s much harder to get general ongoing contributions even if that would make the entire community better off long-term.
  - kstrauser 6y ago
    I count Jeff as a personal friend, and I think you’re exactly right. Everyone was riding his butt to port to Python 3, but the giant companies using Fabric weren’t exactly lining up to help. He found that incredibly frustrating.
- ivan_ah 6y ago
rcarmo 6y ago
To be honest, I do everything inside my nodes via https://cloud-init.io https://cloud-init.io these days. A single YAML file (yes, yes, I know) that typically bootstraps the machine, grabs a compose file, end done.
RNCTX 6y ago
I don't see anyone else mentioning that the selling point of using YAML in Ansible is "human readable." I see no practical benefit to requiring devops people to use one scripting language over another, when there are so many equivalent choices to accomplish the same tasks.
- acdha 6y ago
  This isn’t a simple binary flag - there’s an inflection point for most projects where YAML starts easier to read for simple tasks but you end up having to do things with loops, value manipulation, etc. which would be much easier to read in normal Python.
  - RNCTX 6y ago
    Well, that presumes the person is a regular user of Python. Maybe they write bash scripts instead. Maybe they're using a BSD and write bourne shell scripts and hate bash. Maybe they're Windows servers using powershell. As someone else stated the beauty of Ansible is that it abstracts away the most common functions so that the scripts are portable, and the person maintaining the scripts doesn't have to keep up with every syntax change in version if they keep Ansible updated. Unless you're going to convince individual open source project maintainers to agree to and convert to a somewhat rigid style guide for command line flags, API syntax, and configuration syntax (which no one has managed to do, hence the existence of distros, containers, and Ansible), you're gonna have a hard time convincing anyone to give up on markup languages in deployment tools, I think.
    - acdha 6y ago
      My point was that language complexity needs to be there once you get over a certain level of project complexity, which many projects hit. Once you need to do more than the most basic operations you’re learning something which is going to be better off learning a regular programming language which can take you much further rather than learning at least two separate syntaxes (Python and Jinja2) plus all of YAML’s quirks and duplicate syntax structure, and almost all programming tools have better IDE support, linting, and validation tools. If you want a document language, TOML is a better choice (or JSON or even non-enterprise XML). If you want to write programs, anything else is a better choice. Hashicorp’s HCL is interesting in that it’s much better designed and supported but even there the main thing which makes it work is the presence of an escape hatch when you hit a limitation.
throwaway42092 6y ago
None of this stuff works with pipelined / staged deployments, where tools cannot make changes across stages using SSH.
skywhopper 6y ago
This is fascinating to me, as someone who manages and Ansible-based config management infra that keeps thousands of servers running. The problem I have with Ansible isn’t the YAML, it’s the Python. I spend at least half my time struggling with reconciling which Python runtime and modules are installed on which hosts. And between the Python 2 EOL, the growth of ARM hosts in our inventory, a fuzzy and changing line between what’s “Ansible” or “Python” or “Jinja” in the syntax, and regular OS upgrades, it’s a lot more work than it ought to be. And at scale there are some really frustrating performance issues that aren’t entirely Python’s fault but which would be much better if we could use native code. I’ve been hoping for years that someone would build a Golang tool that could take Ansible playbooks and do the right thing with minimal tweaking, but I think we’re beyond the point where anyone is going to invest that much time and effort into new CM tooling.
- stevekemp 6y ago
  I suspect if anybody is going to write something new it won't benefit so much by being constrained by Ansible's YAML parser, etc. I toyed with writing a simple thing, inspired by Fabric, and that was moderately useful: https://github.com/skx/deployr https://github.com/skx/deployr Later I tried again, with something more puppet-like, to experiment with how you might handle dependencies in a simple and consistent fashion, ideally without using a complete language (i.e. puppet/ruby): https://github.com/skx/marionette/ https://github.com/skx/marionette/ The problem with the simpler approach is that they do only basic things, to support "everything" means writing a hell of a lot of glue, and making a lot of busywork until it is remotely useful to others. Puppet/CFengine/Chef/Salt/Ansible sometimes seem like they're a dead-end with how many things are moving to master /golden images (packer, etc), and containers.
dazoot 6y ago
What about SaltStack ?
meddlepal 6y ago
What I really want is a strongly typed version of Ansible possibly substituting the YAML bit for just calling library code in say Go or something else.
- woile 6y ago
  I'm on the same boat. I'd like to try cuelang but I haven't had the opportunity
dmsimard 6y ago
It's healthy to look at alternatives, good on OP. I wanted to comment on a specific part that mentions that Ansible's YAML is a nightmare to debug. I wouldn't describe it like that but I'm biased because I've written ARA Records Ansible exactly to make playbooks easier to understand and troubleshoot: https://github.com/ansible-community/ara https://github.com/ansible-community/ara I've been using Ansible for many years in different contexts and I think it is easy to get into the trap of "when you have a hammer, everything looks like a nail". For me, Ansible shines as an excellent abstraction layer for supporting different operating systems or as a glue between different tools. I am very much a believer of using the right tool for the job. Ansible isn't always the answer and I think that's OK.
- justaj 6y ago
  So when is Ansible not the answer?
dmarinus 6y ago
What I like most from Ansible is that a playbook is idempotent and (mostly) declarational. Biggest downside is that it's hard to debug (no realtime output) and a bit slow (but acceptable)

146 points by alexeiz 6y ago

I wasn't aware of the VSCode Ansible plugin, I've got to check that out. Pyinfra examples he shows don't look that different from Ansible YAML. ("apt('nginx', update=True, present=True)") I'm pretty happy with Ansible, with one exception: speed. I may play with Pyinfra, but I'm heavily invested in Ansible. But if Pyinfra is fast, I might try replacing our Fabric with it. I get pushback about how long Ansible takes for our deploys vs. Fabric.

larsnystrom 6y ago
So I looked at the VSCode Ansible plugin and it says it is authored by "Microsoft"[1], but the repository is owned by a Github organization called "Visual Studio China"[2] and the plugin identifier is "vscoss.vscode-ansible", when other Microsoft plugins usually has identifiers in the "ms-vscode" namespace (for example the TSLint and C/C++ extensions) or "ms-vscode-remote" or some similar namespace starting with "ms". So why is this ansible plugin such an outlier? How can I be sure that this plugin is actually authored by Microsoft and not by some other entity? Maybe I'm just being paranoid. Edit: Ok so if you click the name "Microsoft" in the vscode marketplace it will take you to a list of Microsoft authored plugins, which I guess means the organization name in the marketplace is verified somehow. So yes, I was just being paranoid. [1] https://marketplace.visualstudio.com/items?itemName=vscoss.vscode-ansible https://marketplace.visualstudio.com/items?itemName=vscoss.v... [2] https://github.com/VSChina/vscode-ansible https://github.com/VSChina/vscode-ansible
deivid 6y ago
try the mitogen plugin. we went from 2 hours to 15 minutes in run time

ShepherdKing 6y ago

There is no mention of Python Bundlewrap (https://bundlewrap.org/ https://bundlewrap.org/), which I have really liked, though I don't know how speed and scalability compares. I believe Bundlewrap will welcome contributors for these enhancements.

simonw 6y ago

pyinfra looks interesting. The thing I want it a way to keep configuration for a single server in a git repo, and have that server periodically run "git pull" and apply any changes. I ended up bodging together some custom scripts to do this but I really didn't want to. Maybe I can get pyinfra to do this? I don't want to have to run it on another machine and run commands over ssh though.

moreati 6y ago
For reference, in Ansible land the two approaches to this would be https://docs.ansible.com/ansible/latest/cli/ansible-pull.html https://docs.ansible.com/ansible/latest/cli/ansible-pull.htm... or https://github.com/ansible/awx https://github.com/ansible/awx (upstream project of Ansible Tower)
- simonw 6y ago
  I had missed ansible-pull, looks like the exact pattern I want. Thanks!

chopraaa 6y ago

If you feel YAML is too complicated for you, a host of problems await for anything you want to do in infrastructure... Ansible has it's own set of features that are largely abstracted out by it's YAML playbook syntax. You can run it across any (or almost) system, from Linux to Windows and even networking devices. You can create modules in any language you want. For speeding up Ansible runs, a number of optimisations can be made including using Mitogen https://mitogen.networkgenomics.com/ansible_detailed.html https://mitogen.networkgenomics.com/ansible_detailed.html Alternatives listed in the blog post don't even come close to the speed and power of Ansible.

sa46 6y ago
> Ansible has its own set of features that are largely abstracted out by its YAML playbook syntax. I don't understand what YAML has to do with abstraction. The example from pyinfra is pretty much the same as the ansible equivalent. Whether the feature set is comparable is a different question (I agree with you, it's probably nowhere close to ansible). apt.packages( {'Install iftop'}, 'iftop', sudo=True, update=True, ) > If you feel YAML is too complicated for you Ansible YAML is a poor substitute for a real language once you add conditional logic to the mix. Another huge downside is that composition in YAML is much harder than in any programming language. A Python (or dhall, or cue) library that compiled to ansible would be pretty swell.
- kitd 6y ago
  Ansible YAML is a poor substitute for a real language once you add conditional logic to the mix. Another huge downside is that composition in YAML is much harder than in any programming language Given that the average devops person is already stuck with yaml, many of those problems are addressed by jsonnet. But I agree, having it in the language would be preferable. I think it stems from the desire to make infrastructure declarative, which isn't really the case when your declarations are dynamic.
  - dynamite-ready 6y ago
    I've written a lot of Ansible in the past year. While I'm happy with the work, I'd admit to looking at some of the logic I've had to put together with clunky, esoteric constructs like `with_nested_items` and the super colloquial variable system, while thinking "it's great to be able to do this stuff, but htf am I going to explain this code to a newbie?". I'd like some kind of 'sensible', backwards compatible layer in Ansible that would allow me to rewrite stuff in raw Python, with 'normal' code, where it makes sense, but keep YAML for constructs that do actually look and work like lists of basic tasks.

torvald 6y ago

If one is not tied to python, NixOps is a breath of fresh air.

BossingAround 6y ago

Kubernetes already requires YAML. If you use Kubernetes, you might as well use Ansible. How YAML got so popular is a mystery to me. It's a huge pain. Then again, I've given up on fighting it, and simply learned how to use it, since I can't escape Kubernetes anyways.

nsonha 6y ago
What kind of logic is that? Lots of things use yaml they usually have some sort of templating engine on top, so whether the tool is friendly/adequate is depending on the templating capacity not the fact that it uses yaml, or any markup language for that matter. Also people are mostly arguing for code vs markup. yml, json, xml etc are all the same. Yml just happens to be more compact which helps with visibility in a big file.
lazyant 6y ago
> How YAML got so popular is a mystery to me. It's good at being human-readable. It's basically json with indents and comments. What other configuration format is out there that allows for tree structure and it's easier on humans? (not XML or json).
- randallsquared 6y ago
  I'd say YAML's good at seeming human-readable... JSON has some surprises and a lot of limitations, but is so much simpler than YAML that it seems worth putting up with them.
- raghava 6y ago
  > good at being human-readable If we consider high composability, non-ambiguity, clarity, expressability (even a word?) as aspects of "human readability" - YAML doesn't go very far in the scale across many of these. A language, with slightly more appealing (yet non-limiting for programmablity) declarative constructs seem very helpful for needs like this. It would be great to see any spider-web-graph of all these solutions across those aspects and thus help someone to take a wiser decision.
unionpivo 6y ago

theonemind 6y ago

"In Python" seems like a weird constraint on the problem. Python doesn't really read like a great deployment configuration language. https://en.wikipedia.org/wiki/Infrastructure_as_code#Community_content https://en.wikipedia.org/wiki/Infrastructure_as_code#Communi... the space has a lot of players. For upgrading like one VPS and some home servers, I'd probably go with Ansible. It probably has the lowest barrier to entry, and getting an IDE just solves the problem that you created by wanting something in Python. Basic config management doesn't need an IDE.

divbzero 6y ago
“In Python” does seem like an odd constraint. I take the OP’s intent to be “let’s find something imperative not declarative”. If that’s the case, I’d love to see the idempotent functionality of Ansible’s modules (e.g. lineinfile) made available as simple CLI commands.
- stonith 6y ago
  You could kind of do this using ansible-runner to dynamically generate a playbook that does what you want.
- inshadows 6y ago
  I assume you consider Ansible to be "declarative". By that definition of "declarative", the following Bash script is also "declarative": #!/bin/bash rm -f /etc/bar mkdir -p /etc/conf.d cp -u /srv/src.txt /etc/conf.d/ because Anisble playbook is really just bunch of following statements if (!stuff_exists(stuff)) create_stuff() which the above utilities in Bash script do with the provided flags. Ansible "declarativeness" isn't anything worthy of worship. It's plain dumb definition of sequence of steps to take with added guard for each step. So let's be honest and stop pretending that this "declarativeness" is anything novel or of any value worthy of writing sequence of (guarded) commands in ad-hoc YAML language. Just add some sugar for the pattern above to a decent language and that's all that's needed.
- Bystroushaak 6y ago
  Op is python programmer who is using python for everything for more than 13 years, so there you have the reason. I am also writing my own programming language .. in python: https://github.com/Bystroushaak/tinySelf https://github.com/Bystroushaak/tinySelf

bamboozled 6y ago

How hard is YAML to use really? The world has much more interesting problems to solve than a decent markup language. I use Ansible with zero issues daily as do many in our team. It’s perplexing coders fined it difficult or hard to use to the point where something as useful as Ansible becomes undesirable because it uses YAML? I hate to say it, but the “alternatives” are just the least good somewhat similar products.

_frkl 6y ago
Spot on. If you have an opinion about YAML and feel you need to make that know on the internet, I usually take it as a sign that I'll probably don't need to read the rest of your comment...
fake-name 6y ago
The problem isn't YAML specifically, it's that to do everything ansible does they have to extend YAML into a crappy programming language in-and-of-itself. If they're going to do that, the configuration files should just use a actual programming language. Just make them actual python files (or ruby files, or whatever). From the famous quote: > Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp. The problem here isn't that they're using lisp (or python, or yaml, or whatever), it's that they're re-implementing a very mature language with lots of supporting infrastruture poorly with lots of bugs and corner cases, rather then just using the relevant (open source) programming language itself.
- GordonS 6y ago
  > The problem isn't YAML specifically, it's that to do everything ansible does they have to extend YAML into a crappy programming language in-and-of-itself. I don't recall how ansible extends yaml, but Microsoft does something that sounds similar with their JSON-based ARM templates (which are for deploying cloud resources), adding functions for loops, parameters and variables. The result is predictably horrible. There are however at least APIs available for C# and others, so it's not the only automation option.
6y ago

stevenjohns 6y ago

Probably worth checking out Mitogen[0] if you want to take away some of the pain points with Ansible. It’s an extremely well built product that really complements Ansible. [0] https://mitogen.networkgenomics.com/ https://mitogen.networkgenomics.com/

Bystroushaak 6y ago
I've seen Mitogen, but that to me looks like lowlevel ssh paralelization library. Great project, but it doesn't seem to provide any abstractions, for example declarative package manager wrapper, that would work on multiple linux versions.

fermigier 6y ago

I have tried or used most of the projects cited in the blog post. I pretty much agree with the conclusion that Pyinfra looks the most promising at this point. I've also tried Bundlewrap (https://bundlewrap.org/ https://bundlewrap.org/) a couple of years ago though I lack the perspective to compare them. Another interesting project not cited in the post is Nuka (https://doc.bearstech.com/nuka/ https://doc.bearstech.com/nuka/). Some personal perspectives: I have used Ansible in the past and didn't like it. I have used Fabric and Fabtools many years ago (Fabtools happens to be written by a friend) and I agree that something like Fabtools is needed to make Fabric relevant. Fabric has a completely revamped version, which was quite long in the making, but not fairly stable, but AFAIK Fabric 2 doesn't (yet) have something similar to what Fabtools was to Fabric 1. I had great hope for opsmop (https://github.com/opsmop/opsmop https://github.com/opsmop/opsmop) by Ansible's creator, which was supposed to address Ansible's shortcomings, but he dropped the project (citing lack of traction) a few months after starting it in 2018. At this point I'm sticking to Fabric 2, but I'm still looking for alternatives. I'm also wondering if Augeas (https://augeas.net/ https://augeas.net/ - a configuration editing tool which parses configuration files for a large set of software in their native formats and transforms them into a tree) could be used in combination with one the of tools mentioned in the post to ease the parts related to managing configuration.

Bystroushaak 6y ago
Thanks for the links. I am kinda surprised by the activity on this blogpost, it's just something that I've put together in an hour or so, mostly as notes for myself. I do "exporations" where I put bunch of related stuff into the wiki node and then take an evening to go over them and do some evaluation. It wasn't really meant as a comprehensive study.

idoby 6y ago

For all its flaws, Docker does solve a lot of the pain the author mentions. Deploying postgres, for example, with a config file and keeping it running is really very easy with Docker Compose. Not affiliated with Docker except for using it, and while I do have some thoughts about design choices made by Docker, it's still a very good tool.

dynamite-ready 6y ago
I don't see Docker as an alternative to a configuration manager. For one, turning to containers represents a distinct architectural choice, with a clear set of tradeoffs, that a developer may or may not want to introduce to a project.
- idoby 6y ago
  It sure does, but even if you choose not to take advantage of container tech, it's still a very easy way to deploy stuff on a single machine. Not sure why the downvotes.
  - dynamite-ready 6y ago
    I didn't downvote. I see your point, but it's a bit of a digression, tbf.
Bystroushaak 6y ago
I do use Docker, but that is a tool for completely different purpose. I need something else for configuration of all my physical machines; virtual server, physical server, several notebooks and so on.

INTPenis 6y ago

The author of Ansible actually tried to make a Python3 DSL based Ansible alternative called opsmop.[1] I'm not sure why but it was discontinued. I would think that if anyone could make that work it would be that particular person with their previous experience from Cobbler and Ansible. I started using Ansible when it was brand spanking new and what I've noticed over the years is that people often over complicate their Ansible use. Sometimes it's better to out source something to an RPM or a container than to use Ansible to do every single thing. 1. https://github.com/opsmop/opsmop https://github.com/opsmop/opsmop

rezonant 6y ago
Agreed that people tend to overcomplicate their Ansible configs. It is also OK to write a script and run it from Ansible as part of your playbook if you have a complex use case.
jabl 6y ago
Here's an explanation by the sensible/opsmop creator: https://medium.com/@michaeldehaan/revisiting-systems-management-assumptions-5eee2d5fa82e https://medium.com/@michaeldehaan/revisiting-systems-managem...
- INTPenis 6y ago
  Thanks, I just skimmed it and found this. >Nobody wants to keep up with a configuration management tool, especially learn another new one (which is why creating opsmop failed), learn another YAML dialect, or maintain a full active cloud abstraction layer running on top of their cloud itself.

mongol 6y ago

I think there would be value in providing idempotent operations to all common Linux tools. This often exist, but sometimes with weird side effects. Sometimes an add operation returns a failure code because item already exist and so on. It would be quite worthwhile to go through the Ansible documentation and document a command line alternative for each. Then it would be easier to use a regular programming language to achieve the same effects.

ctippett 6y ago

I’ve gotten a lot of mileage out of using cloud-init to bootstrap VMs. The configuration is in YAML, but doesn’t require any other client side tooling. Most major distributions (incl. the public clouds) all support it and make it available in their machine images. It integrates nicely into Terraform too. https://cloudinit.readthedocs.io https://cloudinit.readthedocs.io

syllogism 6y ago

Fabric was a casualty of the Python 2/3 split. Unicode isn't a great default for the shell interaction so the project took a long time to be ported, and by then it lost all momentum. The successor project, Invoke, was finally released a year or two ago, but now it's unmaintained too.

acdha 6y ago
I think it’s more that Jeff Forcier is one – very productive – person and while tons of places used Fabric there were far fewer who contributed significantly to supporting it. I much prefer Python to programming in a YAML derivative with multiple bolt-on extension mechanisms but at the end of the day Ansible brings a lot to the table because there are a bunch of people employed by Red Hat to work on it full-time.
- syllogism 6y ago
  Yes that's true, but turning it around: the task is too big (and also sucks too much) for one person to do mostly alone. And also like, why? There would've been a more active community of companies depending on it and it wouldn't have blocked out on him if it had maintained the momentum it had.
  - acdha 6y ago
    I was talking about it at the peak of that momentum. This is a common problem in open source: companies often don’t contribute or only support specific things they need. It’s much harder to get general ongoing contributions even if that would make the entire community better off long-term.
- kstrauser 6y ago
  I count Jeff as a personal friend, and I think you’re exactly right. Everyone was riding his butt to port to Python 3, but the giant companies using Fabric weren’t exactly lining up to help. He found that incredibly frustrating.
ivan_ah 6y ago

rcarmo 6y ago

To be honest, I do everything inside my nodes via https://cloud-init.io https://cloud-init.io these days. A single YAML file (yes, yes, I know) that typically bootstraps the machine, grabs a compose file, end done.

RNCTX 6y ago

I don't see anyone else mentioning that the selling point of using YAML in Ansible is "human readable." I see no practical benefit to requiring devops people to use one scripting language over another, when there are so many equivalent choices to accomplish the same tasks.

acdha 6y ago
This isn’t a simple binary flag - there’s an inflection point for most projects where YAML starts easier to read for simple tasks but you end up having to do things with loops, value manipulation, etc. which would be much easier to read in normal Python.
- RNCTX 6y ago
  Well, that presumes the person is a regular user of Python. Maybe they write bash scripts instead. Maybe they're using a BSD and write bourne shell scripts and hate bash. Maybe they're Windows servers using powershell. As someone else stated the beauty of Ansible is that it abstracts away the most common functions so that the scripts are portable, and the person maintaining the scripts doesn't have to keep up with every syntax change in version if they keep Ansible updated. Unless you're going to convince individual open source project maintainers to agree to and convert to a somewhat rigid style guide for command line flags, API syntax, and configuration syntax (which no one has managed to do, hence the existence of distros, containers, and Ansible), you're gonna have a hard time convincing anyone to give up on markup languages in deployment tools, I think.
  - acdha 6y ago
    My point was that language complexity needs to be there once you get over a certain level of project complexity, which many projects hit. Once you need to do more than the most basic operations you’re learning something which is going to be better off learning a regular programming language which can take you much further rather than learning at least two separate syntaxes (Python and Jinja2) plus all of YAML’s quirks and duplicate syntax structure, and almost all programming tools have better IDE support, linting, and validation tools. If you want a document language, TOML is a better choice (or JSON or even non-enterprise XML). If you want to write programs, anything else is a better choice. Hashicorp’s HCL is interesting in that it’s much better designed and supported but even there the main thing which makes it work is the presence of an escape hatch when you hit a limitation.

throwaway42092 6y ago

None of this stuff works with pipelined / staged deployments, where tools cannot make changes across stages using SSH.

skywhopper 6y ago

This is fascinating to me, as someone who manages and Ansible-based config management infra that keeps thousands of servers running. The problem I have with Ansible isn’t the YAML, it’s the Python. I spend at least half my time struggling with reconciling which Python runtime and modules are installed on which hosts. And between the Python 2 EOL, the growth of ARM hosts in our inventory, a fuzzy and changing line between what’s “Ansible” or “Python” or “Jinja” in the syntax, and regular OS upgrades, it’s a lot more work than it ought to be. And at scale there are some really frustrating performance issues that aren’t entirely Python’s fault but which would be much better if we could use native code. I’ve been hoping for years that someone would build a Golang tool that could take Ansible playbooks and do the right thing with minimal tweaking, but I think we’re beyond the point where anyone is going to invest that much time and effort into new CM tooling.

stevekemp 6y ago
I suspect if anybody is going to write something new it won't benefit so much by being constrained by Ansible's YAML parser, etc. I toyed with writing a simple thing, inspired by Fabric, and that was moderately useful: https://github.com/skx/deployr https://github.com/skx/deployr Later I tried again, with something more puppet-like, to experiment with how you might handle dependencies in a simple and consistent fashion, ideally without using a complete language (i.e. puppet/ruby): https://github.com/skx/marionette/ https://github.com/skx/marionette/ The problem with the simpler approach is that they do only basic things, to support "everything" means writing a hell of a lot of glue, and making a lot of busywork until it is remotely useful to others. Puppet/CFengine/Chef/Salt/Ansible sometimes seem like they're a dead-end with how many things are moving to master /golden images (packer, etc), and containers.

dazoot 6y ago

What about SaltStack ?

meddlepal 6y ago

What I really want is a strongly typed version of Ansible possibly substituting the YAML bit for just calling library code in say Go or something else.

woile 6y ago
I'm on the same boat. I'd like to try cuelang but I haven't had the opportunity

dmsimard 6y ago

It's healthy to look at alternatives, good on OP. I wanted to comment on a specific part that mentions that Ansible's YAML is a nightmare to debug. I wouldn't describe it like that but I'm biased because I've written ARA Records Ansible exactly to make playbooks easier to understand and troubleshoot: https://github.com/ansible-community/ara https://github.com/ansible-community/ara I've been using Ansible for many years in different contexts and I think it is easy to get into the trap of "when you have a hammer, everything looks like a nail". For me, Ansible shines as an excellent abstraction layer for supporting different operating systems or as a glue between different tools. I am very much a believer of using the right tool for the job. Ansible isn't always the answer and I think that's OK.

justaj 6y ago
So when is Ansible not the answer?

dmarinus 6y ago

What I like most from Ansible is that a playbook is idempotent and (mostly) declarational. Biggest downside is that it's hard to debug (no realtime output) and a bit slow (but acceptable)