5 ms·

I'm hopping someday people will provide a friendly Python API on top of ansible so that I don't have to write its YAML DSL anymore. The "it must be idempotent"

I'm hopping someday people will provide a friendly Python API on top of ansible so that I don't have to write its YAML DSL anymore. The "it must be idempotent" argument lead to putting conditional into a a language that was never meant to be, doing the whole "store" danse that could be done with simple function calls and whole lot of conventions instead of a proof tested import system. It's calling python code behind the scene anyway, and nothing reads the playbook apart from ansible itself. Ansible is a great tool, but I strongly believe the experience would be 100% better without learning, crafting and debugging those twisted .yml.

geerlingguy 7y ago
_Technically_ there's a Python API on it: https://docs.ansible.com/ansible/latest/dev_guide/developing_api.html https://docs.ansible.com/ansible/latest/dev_guide/developing... There's also a utility to help with the running of Ansible playbooks via Python (https://github.com/ansible/ansible-runner https://github.com/ansible/ansible-runner). However, I've said this many times before—if your playbooks start mixing Python into tasks, you're either doing something wrong (it could be architected better), or it's time to write a custom Ansible module so the Python goes into a `.py` file and not into YAML. 99% of my playbooks are strict YAML with maybe a few Jinja filters, which make everything very readable. But I've also seen a ton of YAML that makes me want to barf.
- BiteCode_dev 7y ago
  Keyword being "friendly". See, if hello world is: - name: Hello Ansible hosts: ansibleclient01.local tasks: - name: Copy '/tmp/testfile.txt' into 'hello world'. copy: content: hello worldn dest: /tmp/testfile.txt Then it would become: from icandream import playbook pb = playbook("Hello Ansible", hosts=["ansibleclient01.local"]) with db.define_task(Copy '/tmp/testfile.txt' into 'hello world') as task: task.copy(content="hello worldn", dest="/tmp/testfile.txt) Instant benefits: - Python interpretter is very good at telling you error messages. - You get an exception system, stack trace and a debugger for free - The entire python tooling ecosystem, including full monthy IDE support, linters and formaters, is at your disposal - Any more complex example is going to be much cleaner than the yaml version, exponentially. And no need to live in the "yaml/python module" dichotomy to keep PB clean. - No need to reinvent the wheel. Python has elegant import, namespaces, ways to pass data around, conditionals, etc., - You can import Python libs. Event if it's just for declarative stuff like pendulum, appdirs.. The only cons are: - idempotence needs to be enforced culturally. It's already the case for people writing the module behind the scene, and a good API + some warning tricks would be enough IMO. - can't read the files outside of Python. But nobody does it anyway, and we can always let the Python files output the yaml representation if needed. Now I don't hold my breath because it's a lot of work and the ansible community usually don't like the idea at all, and I'm not willing to do it. Fair is fair.
mindwok 7y ago
Yes! Totally agree. Ansible's largest problem for me is trying to wrangle YAML into a domain it doesn't belong. This is why I prefer puppets approach, which is a domain specific language that maintains the idea of YAML in that its mostly declarative, desired state, but it still provides sane constructs for looping, conditionals and types. Ansible would benefit greatly from this.
nerdbaggy 7y ago
Salt modules can do this. You can call them right from python code and do whatever you want.
- _ix 7y ago
  I’ve worked with Puppet, Ansible, and Salt professionally in the past few years. Of the three, I’m unsure which I’d choose for greenfield, but I think I would rather inherit a legacy Salt codebase than one in Puppet or Ansible.
  - blakesterz 7y ago
    I've been working with puppet and ansible, and I'm sure I'd rather take on an ansible. I've been exposed to both and ansible has been far easier to figure out for me. Obviously this will be different for everyone depending on what you already know and who wrote that thing that you're inheriting.
justinsaccount 7y ago
> It's calling python code behind the scene anyway It doesn't even really do that. It sets up a rube goldberg mechanism to push a python program containing a base64 encoded zip file containing the module it wants to execute it. Then it runs a shell to run python to run that module. This is why ansible is 2000x slower than it should be. see https://github.com/ansible/ansible/issues/36275 https://github.com/ansible/ansible/issues/36275
- BiteCode_dev 7y ago
  Yes but part of it is necessary because it's agentless, which is an awesome feature. Getting rid of the yaml would make things faster, but compare to the cost of the network, I don't think it would be that much faster.
- expialidocious 7y ago
  I like that, rube goldberg mechanism, good way to describe it. How it uses ssh also slows it down. Mitigen speeds it up significantly. https://mitogen.networkgenomics.com/ https://mitogen.networkgenomics.com/
  - geerlingguy 7y ago
    Mitogen is a nice speedup if it works for your use case. Some of the improvements from the mitogen architecture have slowly worked their way into Ansible core, but it seems that a number of the major shifts can lead to breakage in some scenarios. Hopefully more of the low hanging fruit can make it into Ansible someday...
jerrya 7y ago
What I find interesting & frustrating with ansible is that it is idempotent, but not everything is reversible (or is it? I am a noob with ansible.) So how do people test their ansible playbooks if they can't easily reverse their state?
- BiteCode_dev 7y ago
  Virtual machines. If linux only, docker is great for that because it's so light, and it easier to compare layers of an image.
- expialidocious 7y ago
  I've seen molecule mentioned a lot, but I've never used it. My understanding is that it will spin up a container, vm or cloud image for you, run your playbooks, validate the result and tear down for you. Seems like a lot of trouble. Red Hat maintains it, I believe. https://github.com/ansible-community/molecule https://github.com/ansible-community/molecule
- geerlingguy 7y ago
  Most of the time I'm using Ansible to build containers or AMIs (or other images), which are then deployed, and overwritten for updates. Other times, I write a secondary playbook to roll back some deployment if needed. (Though sometimes it's as simple as setting a `state=present` to `state=absent` for a few tasks and that's all that's required.
- notabee 7y ago
  Ansible seems to be better at declaring a particular configuration at build time, which should be repeatable and tested, and then perhaps running limited updates in a more controlled fashion later. The destination hosts should be disposable and recreated for any significant changes. If you write playbooks to change existing state frequently you have to be very careful to write in tasks that check the previous state. If you can version the roles and playbooks carefully then you can have some idea of the systems' state. For simple idempotency, like changing an app's config template and restarting a daemon, the modules have you covered, but for more complex dependencies between services it can get hairy. Some unpredictable things can happen. Example story: systemd-tmpfiles on CentOS 7 has a poorly documented behavior of skipping files in /tmp or /var/tmp that are both root-owned and read-only. It's not something in the options or config, just a default behavior (that was removed in later versions). We had a playbook that would clone a git repo into /var/tmp and then copy the required files into their appropriate place. The git module worked fine to update the repo each run, until several weeks passed and systemd-tmpfiles blew away most, but not all, of the repo files. Some files with just read permission remained. The module could handle either all the repo files existing or having the repo disappear just fine, but not a partially deleted repo. It would error out on that task. So now there's a new dependency on a changed systemd-tmpfiles config to skip that directory, and the partially deleted mess must be deleted for a run to work on existing hosts with the old config. There's lots of stuff like that that can accrue.
throw0101a 7y ago
> I'm hopping someday people will provide a friendly Python API on top of ansible so that I don't have to write its YAML DSL anymore. Ansible is used in the network automation space as well, but some people found they wanted more flexibility so they created 'raw' Python frameworks: * https://nornir.readthedocs.io/en/latest/ https://nornir.readthedocs.io/en/latest/ * https://napalm-automation.net https://napalm-automation.net Presentation: * https://www.youtube.com/watch?v=3uIk0WQLHZk https://www.youtube.com/watch?v=3uIk0WQLHZk
- BiteCode_dev 7y ago
  Nice. Is it compatible with ansible? Because ansible has one thing that can't be beaten: the ecosystem is huge. You have recipes for everything.
  - throw0101a 7y ago
    I have no idea. AFAICT these were developed because some people found Ansible too limiting, so instead of the 'narrowness' of Ansible modules you can program just about any behaviour you want.
_ernesto 7y ago
Ansible's creator was actually working on a new project that was a pure Python DSL: https://github.com/mpdehaan/opsmop https://github.com/mpdehaan/opsmop I was really excited about it, but unfortunately he stopped work on it since it wasn't getting much traction.
- geerlingguy 7y ago
  The main problem I have with any kind of DSL (Python, Ruby, PHP, Go-based, whatever) is that it usually doesn't abstract enough of the underlying language away, meaning if you're not already a [language XYZ] developer or willing to learn some of the ins-and-outs of the underlying language, you're not going to be very productive. With Ansible, you can go _very_ far before you have to know anything about Python (well, most of the time—annoyingly, there are situations like naming groups where Python's variable syntax leaks through currently). It would be like making people know basic Go syntax just to write or edit a Kubernetes manifest—that immediately limits its appeal to those who have at least a passing interest in Go (which is a much smaller audience than 'willing to learn YAML, which is used all over the place').
  - BiteCode_dev 7y ago
    Basic Python syntax is enough for most playbooks, and it's no more difficult to learn than the ansible yaml dsl. Anybody provisioning a machine knows a little bit of bash, and most probably python. Python is way more "used all over the place" than yaml. Worst of all, when you are a beginner, ansible yaml errors are an utter mystery.

geerlingguy 7y ago

_Technically_ there's a Python API on it: https://docs.ansible.com/ansible/latest/dev_guide/developing_api.html https://docs.ansible.com/ansible/latest/dev_guide/developing... There's also a utility to help with the running of Ansible playbooks via Python (https://github.com/ansible/ansible-runner https://github.com/ansible/ansible-runner). However, I've said this many times before—if your playbooks start mixing Python into tasks, you're either doing something wrong (it could be architected better), or it's time to write a custom Ansible module so the Python goes into a `.py` file and not into YAML. 99% of my playbooks are strict YAML with maybe a few Jinja filters, which make everything very readable. But I've also seen a ton of YAML that makes me want to barf.

BiteCode_dev 7y ago
Keyword being "friendly". See, if hello world is: - name: Hello Ansible hosts: ansibleclient01.local tasks: - name: Copy '/tmp/testfile.txt' into 'hello world'. copy: content: hello worldn dest: /tmp/testfile.txt Then it would become: from icandream import playbook pb = playbook("Hello Ansible", hosts=["ansibleclient01.local"]) with db.define_task(Copy '/tmp/testfile.txt' into 'hello world') as task: task.copy(content="hello worldn", dest="/tmp/testfile.txt) Instant benefits: - Python interpretter is very good at telling you error messages. - You get an exception system, stack trace and a debugger for free - The entire python tooling ecosystem, including full monthy IDE support, linters and formaters, is at your disposal - Any more complex example is going to be much cleaner than the yaml version, exponentially. And no need to live in the "yaml/python module" dichotomy to keep PB clean. - No need to reinvent the wheel. Python has elegant import, namespaces, ways to pass data around, conditionals, etc., - You can import Python libs. Event if it's just for declarative stuff like pendulum, appdirs.. The only cons are: - idempotence needs to be enforced culturally. It's already the case for people writing the module behind the scene, and a good API + some warning tricks would be enough IMO. - can't read the files outside of Python. But nobody does it anyway, and we can always let the Python files output the yaml representation if needed. Now I don't hold my breath because it's a lot of work and the ansible community usually don't like the idea at all, and I'm not willing to do it. Fair is fair.

mindwok 7y ago

Yes! Totally agree. Ansible's largest problem for me is trying to wrangle YAML into a domain it doesn't belong. This is why I prefer puppets approach, which is a domain specific language that maintains the idea of YAML in that its mostly declarative, desired state, but it still provides sane constructs for looping, conditionals and types. Ansible would benefit greatly from this.

nerdbaggy 7y ago

Salt modules can do this. You can call them right from python code and do whatever you want.

_ix 7y ago
I’ve worked with Puppet, Ansible, and Salt professionally in the past few years. Of the three, I’m unsure which I’d choose for greenfield, but I think I would rather inherit a legacy Salt codebase than one in Puppet or Ansible.
- blakesterz 7y ago
  I've been working with puppet and ansible, and I'm sure I'd rather take on an ansible. I've been exposed to both and ansible has been far easier to figure out for me. Obviously this will be different for everyone depending on what you already know and who wrote that thing that you're inheriting.

justinsaccount 7y ago

> It's calling python code behind the scene anyway It doesn't even really do that. It sets up a rube goldberg mechanism to push a python program containing a base64 encoded zip file containing the module it wants to execute it. Then it runs a shell to run python to run that module. This is why ansible is 2000x slower than it should be. see https://github.com/ansible/ansible/issues/36275 https://github.com/ansible/ansible/issues/36275

BiteCode_dev 7y ago
Yes but part of it is necessary because it's agentless, which is an awesome feature. Getting rid of the yaml would make things faster, but compare to the cost of the network, I don't think it would be that much faster.
expialidocious 7y ago
I like that, rube goldberg mechanism, good way to describe it. How it uses ssh also slows it down. Mitigen speeds it up significantly. https://mitogen.networkgenomics.com/ https://mitogen.networkgenomics.com/
- geerlingguy 7y ago
  Mitogen is a nice speedup if it works for your use case. Some of the improvements from the mitogen architecture have slowly worked their way into Ansible core, but it seems that a number of the major shifts can lead to breakage in some scenarios. Hopefully more of the low hanging fruit can make it into Ansible someday...

jerrya 7y ago

What I find interesting & frustrating with ansible is that it is idempotent, but not everything is reversible (or is it? I am a noob with ansible.) So how do people test their ansible playbooks if they can't easily reverse their state?

BiteCode_dev 7y ago
Virtual machines. If linux only, docker is great for that because it's so light, and it easier to compare layers of an image.
expialidocious 7y ago
I've seen molecule mentioned a lot, but I've never used it. My understanding is that it will spin up a container, vm or cloud image for you, run your playbooks, validate the result and tear down for you. Seems like a lot of trouble. Red Hat maintains it, I believe. https://github.com/ansible-community/molecule https://github.com/ansible-community/molecule
geerlingguy 7y ago
Most of the time I'm using Ansible to build containers or AMIs (or other images), which are then deployed, and overwritten for updates. Other times, I write a secondary playbook to roll back some deployment if needed. (Though sometimes it's as simple as setting a `state=present` to `state=absent` for a few tasks and that's all that's required.
notabee 7y ago
Ansible seems to be better at declaring a particular configuration at build time, which should be repeatable and tested, and then perhaps running limited updates in a more controlled fashion later. The destination hosts should be disposable and recreated for any significant changes. If you write playbooks to change existing state frequently you have to be very careful to write in tasks that check the previous state. If you can version the roles and playbooks carefully then you can have some idea of the systems' state. For simple idempotency, like changing an app's config template and restarting a daemon, the modules have you covered, but for more complex dependencies between services it can get hairy. Some unpredictable things can happen. Example story: systemd-tmpfiles on CentOS 7 has a poorly documented behavior of skipping files in /tmp or /var/tmp that are both root-owned and read-only. It's not something in the options or config, just a default behavior (that was removed in later versions). We had a playbook that would clone a git repo into /var/tmp and then copy the required files into their appropriate place. The git module worked fine to update the repo each run, until several weeks passed and systemd-tmpfiles blew away most, but not all, of the repo files. Some files with just read permission remained. The module could handle either all the repo files existing or having the repo disappear just fine, but not a partially deleted repo. It would error out on that task. So now there's a new dependency on a changed systemd-tmpfiles config to skip that directory, and the partially deleted mess must be deleted for a run to work on existing hosts with the old config. There's lots of stuff like that that can accrue.

throw0101a 7y ago

> I'm hopping someday people will provide a friendly Python API on top of ansible so that I don't have to write its YAML DSL anymore. Ansible is used in the network automation space as well, but some people found they wanted more flexibility so they created 'raw' Python frameworks: * https://nornir.readthedocs.io/en/latest/ https://nornir.readthedocs.io/en/latest/ * https://napalm-automation.net https://napalm-automation.net Presentation: * https://www.youtube.com/watch?v=3uIk0WQLHZk https://www.youtube.com/watch?v=3uIk0WQLHZk

BiteCode_dev 7y ago
Nice. Is it compatible with ansible? Because ansible has one thing that can't be beaten: the ecosystem is huge. You have recipes for everything.
- throw0101a 7y ago
  I have no idea. AFAICT these were developed because some people found Ansible too limiting, so instead of the 'narrowness' of Ansible modules you can program just about any behaviour you want.

_ernesto 7y ago

Ansible's creator was actually working on a new project that was a pure Python DSL: https://github.com/mpdehaan/opsmop https://github.com/mpdehaan/opsmop I was really excited about it, but unfortunately he stopped work on it since it wasn't getting much traction.

geerlingguy 7y ago
The main problem I have with any kind of DSL (Python, Ruby, PHP, Go-based, whatever) is that it usually doesn't abstract enough of the underlying language away, meaning if you're not already a [language XYZ] developer or willing to learn some of the ins-and-outs of the underlying language, you're not going to be very productive. With Ansible, you can go _very_ far before you have to know anything about Python (well, most of the time—annoyingly, there are situations like naming groups where Python's variable syntax leaks through currently). It would be like making people know basic Go syntax just to write or edit a Kubernetes manifest—that immediately limits its appeal to those who have at least a passing interest in Go (which is a much smaller audience than 'willing to learn YAML, which is used all over the place').
- BiteCode_dev 7y ago
  Basic Python syntax is enough for most playbooks, and it's no more difficult to learn than the ansible yaml dsl. Anybody provisioning a machine knows a little bit of bash, and most probably python. Python is way more "used all over the place" than yaml. Worst of all, when you are a beginner, ansible yaml errors are an utter mystery.