6 ms·

First of all, congratulations to the team of Ansible and these tools have benefited us in numerous ways that cannot be understated. Having said that, this caug

First of all, congratulations to the team of Ansible and these tools have benefited us in numerous ways that cannot be understated. Having said that, this caught my eye: > There may be backwards incompatibilities in the core playbook language. Please see the porting guide for details. Doing incompatible changes is not something specific to Ansible, for example Puppet has also done it time and again. We've been using both Ansible and Puppet in my previous job, and we always found it mildly annoying that upgrading a system (Linux, FreeBSD) would, in addition to the other "usual" dangers, bring along the danger of the new version not having a package for the "old" version of Puppet or Ansible that we were using. Which would force us to divert attention to the automation tool's problems instead of using the automation tool to solve problems. I do understand that nothing can stand still and everything must evolve and change, but at some point this acquires the flavour of changes for the sake of changes. Especially when regressions happen and things that were working perfectly are now breaking, it is not exactly pleasant to have to devote time to them. It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged, but the syntax of a manifest or playbook that does a couple of simple operations cannot remain stable. It's not like those tools were created yesterday, in which case it would be reasonable to expect changes in their first years.

AJAlabs 5y ago
With verison 4.0.0, they've implemented semantic versioning which should make life easier for avoiding breaking backwards compatibility changes. https://semver.org https://semver.org
- asiachick 5y ago
  Does that make it easier or does it just mean you'll be more aware upfront that to get the security fixed or support you need for some new feature you need to upgrade to an incompatible version
- jacobsenscott 5y ago
  All semver does is tell you you are pinned to some obsolete tool, and you are going to go through a soul sucking 6 months of yak shaving to stay up to date. (rails programmer since v3 here) Semver doesn't make life easier. Making backward compatible changes is what makes life easier.
zhengyi13 5y ago
I feel you: in our environment, we stayed on Puppet 3 for exactly the sort of reason you call out for the longest time, and upgrading our Puppet codebase to be 4+ compatible was a major effort. OTOH, in the case of Ansible, I've had good success with `pip install --user` to get a local (to my user) Ansible install that's separate from the system, and it looks like specific versions of software can be requested.
colechristensen 5y ago
The reason why kubernetes et al. are so popular is that the orchestration tools like ansible, puppet, salt, chef... they all failed to deliver ultimately great software. Plagued by mediocre quality external modules and a constant churn of semi-backwards incompatible changes which required frequent maintenance, they have just been hard to use. You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted. They were often somewhere both not opinionated and too opinionated, and generally suffered from being "DSL"s domain-specific languages which were really just syntatic magic in various languages which really needed to be actual programming languages or more explicitly policy setting frameworks. They were in the middle of several extremes and tried to be lots of things at once and generally just didn't do a great job, unfortunately.
- binoct 5y ago
  > You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted. How does Kubernetes not also result in the same? I'm not questioning plenty of improvements and the somewhat different domain it brings to the table overall but genuinely curious. From what I've seen any tool designed to integrate and manage complex, disparate systems is going to end up with lots special cases, domain knowledge, and require digging into implementation details on a regular basis for anything other than absolute common-case uses.
  - pm90 5y ago
    One of the best things about K8s is its pretty stable API. We use GKE and the upgrades happen without issues 99% of the time. The other benefit that k8s got for "free" was with containers: developers could stuff in whatever environment, software and configs they wanted inside the container. This removed most of the "my app broke because some shared lib didn't install properly" sorts of issues. This separation b/w software configuration and application configuration is extremely powerful.
  - waheoo 5y ago
    Who said it doesn't? Look at its schema it's horrific. The reason it's popular is that it's backed by Google and solves problems Docker the company couldn't. That doesn't mean k8s is by any means not suffering the same problems of being a dsl that should be something else. It also doesn't mean k8s won't suffer the same weirdness between platforms (AWS vs GCE vs GKE).
geodel 5y ago
I am not much familiar with Ansible/Puppet etc. However working in Java and dealing with frameworks like Spring/boot, kafka etc I notice a phenomenon of large number of breaking changes, specially when one update in couple of years as opposed to update with every release. Now when I dig into changes it just points to extremely poor design to begin with. It just couldn't handle slight change in requirement without breaking changes. I do not think it is a case of hindsight 20/20. These are basic things like object need multiple fields but it is rigidly defined with 2 fixed strings and now one need object V2 to update. Or all functionality dumped at base URL so no scope of adding sub-functionality without breaking the whole pattern. I can understand no one can plan for how complex functionality evolve over time. But here I see perennial amateur hour with every release. And it is doubly irritating when these frameworks are already many layer deep stack of bullshit abstractions so ideally they should be able to handle changes without breaking.
- deleted 5y ago
  [deleted]
- mech422 5y ago
  I tend to think this is an issue with 'designing' systems vs 'evolving' them. When you build something to 'scratch your own itch' and it suddenly becomes popular, lots of stuff you didn't think of tends to pop up. Maybe all the market studies, focus groups, ad nauseum in 'traditional' BigCo development actually did have some value. Ansible at least had the advantage of being a 'broad' tool from the get go. So a lot of the common use cases were covered. But I'm sure surprises still popped up :-)
- overtomanu 5y ago
  Can you give some examples of breaking changes in spring boot? Thinking of using it for some projects
kbenson 5y ago
> It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged, but the syntax of a manifest or playbook that does a couple of simple operations cannot remain stable. It's clear when you deal with both types of systems. Perl optimized for stability and backwards compatibility at the cost of new features and new language conveniences. This is great and a strength when all your competitors are playing catch-up or don't have enough market share to make an impact, but once that's not the case, anything new they bring to the table is a point in their favor, and you get what we have now, the long slow death of Perl. Backwards compatibility is wonderful until there's a really useful feature that it prevents, then all of a sudden it's a more complex story because it's not just preventing busywork, it's preventing progress.
- bredren 5y ago
  This is happening in Django land right now.
- taneq 5y ago
  There seems to have been a strong swing in the past decade or so towards busywork being not just acceptable but in some way laudable. "Doing the work" to keep your product using all latest versions is noble, whether or not you actually gain anything by upgrading. I can't shake the feeling that it all ties in to the industry wide push for SaaS and subscription models and evergreen software, all of which share an underlying motivation of everlasting job security for software companies and the people who work for them. It's a similar attitude to the parts of the OSS community who feel that anyone complaining about poor documentation or usability just "doesn't want to do the work" to understand the system.
pfranz 5y ago
I've never used things like Ansible or Puppet in a professional environment, but I can appreciate how difficult my recommendation is. If you're using a technology for a significant part of your business, decouple it from the OS. Where that line is drawn is subjective. I don't even think backwards incompatibilities are the biggest reason. In practice, the problem is often around bugfixes and upgrades. Upgrades are much easier if you can do them independently. It sucks to have to upgrade the kernel and other tooling just to grab a small bugfix--or hold off on a transition waiting until both OS issues and code base issues are resolved. > It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged In all likelihood they don't. For open source projects they probably have a lot of workarounds and cruft to support a variety of environments and spans of versions they are both written against and tested against. Internal code is written and tested against a specific version relying on your proprietary environment.
- saargrin 5y ago
  what do you mean decouple it from the OS ? this is not coupled to OS to begin with,ansible is just a bunch of python modules in a long trenchcoat
  - pfranz 5y ago
    > we always found it mildly annoying that upgrading a system (Linux, FreeBSD) would [...] bring along the danger of the new version not having a package for the "old" version of Puppet or Ansible that we were using. If this is the only issue, it sounds only mildly annoying. Either upgrade your tooling to use the newer version of be prepared to package up that old version of Puppet/Ansible--as long as you control what version is installed and are comfortable with the range of versions the package manager offers. The type of scenario I had in mind is relying on something like Python, Ruby, Perl, etc. Something the OS may also deeply rely on. This came up a lot when macOS stopped updating a lot of GPL2 stuff to avoid shipping GPL3 or when they announced they'd stop shipping Python, Perl, and Ruby. You also see it when people try and change system Python version outside of the package manager in Linux and can no longer boot--the same thing can just as easily happen with libraries.
notatoad 5y ago
if it was just incompatible syntax changes, that would be one thing. but it's not - it's incompatible administration and structure changes. they just stabilized the new modules shift a couple months ago, which restructured all their documentation and issue tracking, breaking uncountable links and closing thousands of open issues. and that wasn't the first time they'd done that. at this point i've started transitioning as much of my administration stuff as i can away from ansible. there's some good stuff in their codebase, but it's easier for me to find it in the source code and copy-paste it into my own projects than it is for me to work within the ansible structure. ansible solved a lot of problems for us, but over the time we've been using it i'm pretty sure it's caused more problems than it's solved.
- open0 5y ago
  I wrote my own remote scriptrunner thing because I hate Ansible so much.
  - brightball 5y ago
    I did this before with perl’s SSH::Batch. You can install it from CPAN and get really simple ability to run code on groups of servers in parallel. Works great. Eventually I switched to Ansible because it made the more complicated things a lot simpler.
  - dheera 5y ago
    I use bash scripts too on almost all my systems. Containerize deployments. Dockerfiles are a godsend and work indefinitely.
- yjftsjthsd-h 5y ago
  > at this point i've started transitioning as much of my administration stuff as i can away from ansible Any preferred alternatives you can share?
  - Fizzadar 5y ago
    There's pyinfra (https://github.com/Fizzadar/pyinfra https://github.com/Fizzadar/pyinfra) if you like Python (disclaimer: I started the project), heavily inspired by Ansible just without the yaml/jinja2 hybrid stuff.
dheera 5y ago
One would think at the very least you should be able to specify a version of your script some place and it be able to parse all prior versions.
pbreit 5y ago
How is this possible in 2021: "Due to a limitation in pip, if you are upgrading from Ansible 3 (or earlier), you need to uninstall Ansible and Ansible Base before installing Ansible 4"
- sam_goody 5y ago
  https://xkcd.com/1987/ https://xkcd.com/1987/ Python does some things really well, but their package managers leave what to be desired [especially for the uninitiated].
strzibny 5y ago
Yes, that's another reason I am sticking with Bash for teaching for my book on deployments. Your Bash will only change if your tools change. If you want to know how things work under the hood, you don't want to go through a rabbit hole of Python modules that also change beneath you.
j1elo 5y ago
It's fun how one's own perpectives and opinions change and adapt with the experience of facing real-world problems over years of professional (or otherwise any kind of) experience. I was contrary to the idea of 3rd party software repositories, and was of the opinion that one should strive to depend as much as possible from system provided packages. But the same experience that you mention now, is what gave my point of view a 180 degree change, and now I'd rather install something from pip or NPM. I can just pin the desired version, and they more or less have the policy of holding past versions in a frozen state, so if I know for a fact that version 1.2.3 is exactly right for my needs, I'll keep using that one regardless of system. The weak point of packaging systems (my experience is Debian and Ubuntu) is that normally there are not many (if _any_ at all) alternative versions offered for the same software. They are built with the general assumption that you should use the latest ones, and of course they don't even attempt to offer a history of versions of every package. So every time you upgrade, it happens what you say, "divert attention to the tool's problems instead of using the tool to solve problems" (very good wording by the way, I liked it a lot)

AJAlabs 5y ago

With verison 4.0.0, they've implemented semantic versioning which should make life easier for avoiding breaking backwards compatibility changes. https://semver.org https://semver.org

asiachick 5y ago
Does that make it easier or does it just mean you'll be more aware upfront that to get the security fixed or support you need for some new feature you need to upgrade to an incompatible version
jacobsenscott 5y ago
All semver does is tell you you are pinned to some obsolete tool, and you are going to go through a soul sucking 6 months of yak shaving to stay up to date. (rails programmer since v3 here) Semver doesn't make life easier. Making backward compatible changes is what makes life easier.

zhengyi13 5y ago

I feel you: in our environment, we stayed on Puppet 3 for exactly the sort of reason you call out for the longest time, and upgrading our Puppet codebase to be 4+ compatible was a major effort. OTOH, in the case of Ansible, I've had good success with `pip install --user` to get a local (to my user) Ansible install that's separate from the system, and it looks like specific versions of software can be requested.

colechristensen 5y ago

The reason why kubernetes et al. are so popular is that the orchestration tools like ansible, puppet, salt, chef... they all failed to deliver ultimately great software. Plagued by mediocre quality external modules and a constant churn of semi-backwards incompatible changes which required frequent maintenance, they have just been hard to use. You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted. They were often somewhere both not opinionated and too opinionated, and generally suffered from being "DSL"s domain-specific languages which were really just syntatic magic in various languages which really needed to be actual programming languages or more explicitly policy setting frameworks. They were in the middle of several extremes and tried to be lots of things at once and generally just didn't do a great job, unfortunately.

binoct 5y ago
> You ended up with lots of magic an internal domain knowledge about how to do things, and often had to get clever to get the result you wanted. How does Kubernetes not also result in the same? I'm not questioning plenty of improvements and the somewhat different domain it brings to the table overall but genuinely curious. From what I've seen any tool designed to integrate and manage complex, disparate systems is going to end up with lots special cases, domain knowledge, and require digging into implementation details on a regular basis for anything other than absolute common-case uses.
- pm90 5y ago
  One of the best things about K8s is its pretty stable API. We use GKE and the upgrades happen without issues 99% of the time. The other benefit that k8s got for "free" was with containers: developers could stuff in whatever environment, software and configs they wanted inside the container. This removed most of the "my app broke because some shared lib didn't install properly" sorts of issues. This separation b/w software configuration and application configuration is extremely powerful.
- waheoo 5y ago
  Who said it doesn't? Look at its schema it's horrific. The reason it's popular is that it's backed by Google and solves problems Docker the company couldn't. That doesn't mean k8s is by any means not suffering the same problems of being a dsl that should be something else. It also doesn't mean k8s won't suffer the same weirdness between platforms (AWS vs GCE vs GKE).

geodel 5y ago

I am not much familiar with Ansible/Puppet etc. However working in Java and dealing with frameworks like Spring/boot, kafka etc I notice a phenomenon of large number of breaking changes, specially when one update in couple of years as opposed to update with every release. Now when I dig into changes it just points to extremely poor design to begin with. It just couldn't handle slight change in requirement without breaking changes. I do not think it is a case of hindsight 20/20. These are basic things like object need multiple fields but it is rigidly defined with 2 fixed strings and now one need object V2 to update. Or all functionality dumped at base URL so no scope of adding sub-functionality without breaking the whole pattern. I can understand no one can plan for how complex functionality evolve over time. But here I see perennial amateur hour with every release. And it is doubly irritating when these frameworks are already many layer deep stack of bullshit abstractions so ideally they should be able to handle changes without breaking.

deleted 5y ago
[deleted]
mech422 5y ago
I tend to think this is an issue with 'designing' systems vs 'evolving' them. When you build something to 'scratch your own itch' and it suddenly becomes popular, lots of stuff you didn't think of tends to pop up. Maybe all the market studies, focus groups, ad nauseum in 'traditional' BigCo development actually did have some value. Ansible at least had the advantage of being a 'broad' tool from the get go. So a lot of the common use cases were covered. But I'm sure surprises still popped up :-)
overtomanu 5y ago
Can you give some examples of breaking changes in spring boot? Thinking of using it for some projects

kbenson 5y ago

> It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged, but the syntax of a manifest or playbook that does a couple of simple operations cannot remain stable. It's clear when you deal with both types of systems. Perl optimized for stability and backwards compatibility at the cost of new features and new language conveniences. This is great and a strength when all your competitors are playing catch-up or don't have enough market share to make an impact, but once that's not the case, anything new they bring to the table is a point in their favor, and you get what we have now, the long slow death of Perl. Backwards compatibility is wonderful until there's a really useful feature that it prevents, then all of a sudden it's a more complex story because it's not just preventing busywork, it's preventing progress.

bredren 5y ago
This is happening in Django land right now.
taneq 5y ago
There seems to have been a strong swing in the past decade or so towards busywork being not just acceptable but in some way laudable. "Doing the work" to keep your product using all latest versions is noble, whether or not you actually gain anything by upgrading. I can't shake the feeling that it all ties in to the industry wide push for SaaS and subscription models and evergreen software, all of which share an underlying motivation of everlasting job security for software companies and the people who work for them. It's a similar attitude to the parts of the OSS community who feel that anyone complaining about poor documentation or usability just "doesn't want to do the work" to understand the system.

pfranz 5y ago

I've never used things like Ansible or Puppet in a professional environment, but I can appreciate how difficult my recommendation is. If you're using a technology for a significant part of your business, decouple it from the OS. Where that line is drawn is subjective. I don't even think backwards incompatibilities are the biggest reason. In practice, the problem is often around bugfixes and upgrades. Upgrades are much easier if you can do them independently. It sucks to have to upgrade the kernel and other tooling just to grab a small bugfix--or hold off on a transition waiting until both OS issues and code base issues are resolved. > It's not exactly clear how can Perl programs or shell scripts or Makefiles from 20 years ago play perfectly fine unchanged In all likelihood they don't. For open source projects they probably have a lot of workarounds and cruft to support a variety of environments and spans of versions they are both written against and tested against. Internal code is written and tested against a specific version relying on your proprietary environment.

saargrin 5y ago
what do you mean decouple it from the OS ? this is not coupled to OS to begin with,ansible is just a bunch of python modules in a long trenchcoat
- pfranz 5y ago
  > we always found it mildly annoying that upgrading a system (Linux, FreeBSD) would [...] bring along the danger of the new version not having a package for the "old" version of Puppet or Ansible that we were using. If this is the only issue, it sounds only mildly annoying. Either upgrade your tooling to use the newer version of be prepared to package up that old version of Puppet/Ansible--as long as you control what version is installed and are comfortable with the range of versions the package manager offers. The type of scenario I had in mind is relying on something like Python, Ruby, Perl, etc. Something the OS may also deeply rely on. This came up a lot when macOS stopped updating a lot of GPL2 stuff to avoid shipping GPL3 or when they announced they'd stop shipping Python, Perl, and Ruby. You also see it when people try and change system Python version outside of the package manager in Linux and can no longer boot--the same thing can just as easily happen with libraries.

notatoad 5y ago

if it was just incompatible syntax changes, that would be one thing. but it's not - it's incompatible administration and structure changes. they just stabilized the new modules shift a couple months ago, which restructured all their documentation and issue tracking, breaking uncountable links and closing thousands of open issues. and that wasn't the first time they'd done that. at this point i've started transitioning as much of my administration stuff as i can away from ansible. there's some good stuff in their codebase, but it's easier for me to find it in the source code and copy-paste it into my own projects than it is for me to work within the ansible structure. ansible solved a lot of problems for us, but over the time we've been using it i'm pretty sure it's caused more problems than it's solved.

open0 5y ago
I wrote my own remote scriptrunner thing because I hate Ansible so much.
- brightball 5y ago
  I did this before with perl’s SSH::Batch. You can install it from CPAN and get really simple ability to run code on groups of servers in parallel. Works great. Eventually I switched to Ansible because it made the more complicated things a lot simpler.
- dheera 5y ago
  I use bash scripts too on almost all my systems. Containerize deployments. Dockerfiles are a godsend and work indefinitely.
yjftsjthsd-h 5y ago
> at this point i've started transitioning as much of my administration stuff as i can away from ansible Any preferred alternatives you can share?
- Fizzadar 5y ago
  There's pyinfra (https://github.com/Fizzadar/pyinfra https://github.com/Fizzadar/pyinfra) if you like Python (disclaimer: I started the project), heavily inspired by Ansible just without the yaml/jinja2 hybrid stuff.

dheera 5y ago

One would think at the very least you should be able to specify a version of your script some place and it be able to parse all prior versions.

pbreit 5y ago

How is this possible in 2021: "Due to a limitation in pip, if you are upgrading from Ansible 3 (or earlier), you need to uninstall Ansible and Ansible Base before installing Ansible 4"

sam_goody 5y ago
https://xkcd.com/1987/ https://xkcd.com/1987/ Python does some things really well, but their package managers leave what to be desired [especially for the uninitiated].

strzibny 5y ago

Yes, that's another reason I am sticking with Bash for teaching for my book on deployments. Your Bash will only change if your tools change. If you want to know how things work under the hood, you don't want to go through a rabbit hole of Python modules that also change beneath you.

j1elo 5y ago

It's fun how one's own perpectives and opinions change and adapt with the experience of facing real-world problems over years of professional (or otherwise any kind of) experience. I was contrary to the idea of 3rd party software repositories, and was of the opinion that one should strive to depend as much as possible from system provided packages. But the same experience that you mention now, is what gave my point of view a 180 degree change, and now I'd rather install something from pip or NPM. I can just pin the desired version, and they more or less have the policy of holding past versions in a frozen state, so if I know for a fact that version 1.2.3 is exactly right for my needs, I'll keep using that one regardless of system. The weak point of packaging systems (my experience is Debian and Ubuntu) is that normally there are not many (if _any_ at all) alternative versions offered for the same software. They are built with the general assumption that you should use the latest ones, and of course they don't even attempt to offer a history of versions of every package. So every time you upgrade, it happens what you say, "divert attention to the tool's problems instead of using the tool to solve problems" (very good wording by the way, I liked it a lot)