5 ms·

I've kept an eye on this space---automation is not my day job--and I have to say that I still cannot put my finger on Puppet or Chef or Salt or Ansible.

by hanlec 11y ago

duggan 11y ago
Automation is my day job and I'm still not convinced any of them is a significant improvement over a well maintained set of shell scripts and Makefiles.
- berkes 11y ago
  Automation is not my day job, and I am very happy that there are frameworks (I work with Chef) with communities that offer standards, guidelines and libraries for a well maintained set of shell scripts and Makefiles. Instead of having to build and maintain them all myself.
  - otterley 11y ago
    I've sadly learned that given enough time, you'll likely end up mostly using your own set of private cookbooks, even for configuring things community cookbooks already exist for. The quality of the community cookbooks varies wildly from "pretty good" to "toxic waste," and the likelihood of getting fixes upstream fast enough to reflect back into one's own environment is generally low, even for good maintainers. And Chef's strict adherence to X.Y.Z semver for cookbooks (even though cookbooks don't fit semver well) and inability to declare "my cookbook foo_prime can be used whenever cookbook foo is called" makes it near impossible to maintain your own branch while you wait for your fixes to make it upstream. This makes it difficult to try to get things done when you encounter a cookbook that just doesn't work quite right. The cookbook ecosystem is not yet mature enough at this point in time to rely on.
  - duggan 11y ago
    I agree that having standards and best practices is a boon, especially if they can be packaged up in an easy to use way. Like anything, Chef brings its own set of idiosyncrasies to the table. However, I don't like how it brings all of Ruby's idiosyncrasies along for the ride too. If you're not immediately comfortable and familiar with both ruby and the quirks of its ecosystem, you end up investing a lot of time into getting a development environment up and running (and having it reproducible by other members of your team). Most infrastructure tools, if they're any use, are rapidly evolving. Conway's law applies here, and the maintainability of Chef mirrors that of its ruby ecosystem roots - people tend to freeze dependencies rather than upgrade. That's an antipattern we don't need in infrastructure. Honestly I just think we can do better. For me, I'm not sure (read: I really don't know, I'm not being facetious) the costs outweigh the benefits, but for someone totally new, perhaps it helps them get stuff done faster.
- kolev 11y ago
  Same here. I recently found out about Babashka [0] and Declarative Bash [1], but I've written something similar already. It's much easier to find talent that knows Bash and can learn a readable library of Bash scripts within a day vs finding talent that knows the flavor of the day (every year people are migrating from the old favor to the new - Bash => CFEngine => Pupper => Chef => Salt => Ansible => ... => Bash). [0] https://github.com/richo/babashka https://github.com/richo/babashka [1] https://github.com/threatgrid/declarative.bash https://github.com/threatgrid/declarative.bash P.S. I've recently been fascinated by Rex [2] (They should fix their text logo though - I find it terrible). [2] http://www.rexify.org/ http://www.rexify.org/
  - e12e 11y ago
    First I thought this was some dialect of Rexx[1], but then I saw: "Easy to learn, it's just plain Perl". A well. No thank you, I'll stick with ansible. Not that I think Rexx would be a good idea either, it just reminded me of ARexx for AmigaOS... [ed: forgot to ask: you find it easy to find talent that can not only learn some bash script library, but write new code without introducing some gaping hole somewhere?] [1] http://www.rexxla.org/ http://www.rexxla.org/
    - kolev 11y ago
      I'm sorry, but with Rex you don't use much of Perl. It's a DSL, which is based on Perl.
- crdoconnor 11y ago
  IMHO puppet isn't significantly better than bash scripts. There are some improvements, but it has significant drawbacks too. Ansible is way better though, mainly due to: * Idempotence * Templating * Being declarative (turing completeness in a language = more likely to be buggy)
  - e12e 11y ago
    I agree with you, but re: templating: if you're using shell scripts, why not use m4 as well? Shell of course has templating -- but it's a little dangerous...: template=teapot size=short build=stout \ cat <<eof > teapot.conf I'm a $template - $size and $build. I live at $(hostname). eof [ed: m4 is a little like cpp (the c pre-prosessor) on steroids (or acid). To get a feel for how it relates to templating for config files etc, see eg: http://box.matto.nl/m4.html http://box.matto.nl/m4.html ]
    - mercurial 11y ago
      At first glance, I don't see that m4 does anything jinja2 (Ansible's templating system) doesn't do better.
      - e12e 11y ago
        If you're going to use shell and not ansible, then why not use shell with m4. I'm not suggesting shell+m4 is better than ansible, or that ansible+m4 (wtf?! right?) would be a good idea. Just why not use a readily available template/macro language along with shell scripts, if you've decided to use shell scripts. I suppose you could use jinja and bash -- but why pull in two full interpreters as a dependency?
      - 23david 11y ago
        If we're discussing templating languages, I definitely prefer Mako over Jinja2 and think it's much more appropriate for devops templating. But Jinja2 is the default in a lot of projects, including Saltstack and Ansible, and it's hard to fight the currents all the time.
- e12e 11y ago
  Right. But half-way, somewhat maintained ansible is much better than the similarly maintained shell scripts and Makefiles... And you get somewhat more portability with it. mattjaynes mentioned his blogpost in a previous discussion on this: https://devopsu.com/blog/ansible-vs-shell-scripts/ https://devopsu.com/blog/ansible-vs-shell-scripts/ I think it illustrates the difference well. I agree that you probably could do much of the same with discipline, source control, shell scripts and Makefiles. But there's a reason why we use other languages than shell scripts. Shell scripts are finky, and it takes years for people to stop making "rookie" mistakes. If they ever do. "[" vs "[[", ${var} vs $var, ksh vs bash... the shell gives you way too much rope (and I like shell scripts :-).
  - duggan 11y ago
    I should probably couch this in the caveat that I am using Packer and Terraform to control the infrastructure itself :) So machines are effectively configured "offline", tested, and replaced. It's not enough though (usual things like databases and development tweaks like config changes are problematic) so I'm back into the world of config management to see how it can help. Matt makes some good points, and we broadly agree on controlling infrastructure, I just think that most of the CM systems are inadequate abstractions - they don't go far enough, and it leaves them largely as convenience wrappers around shell, but ironically enough, usually without the debuggability of shell. I've worked with Chef for years, and recently have started seeing what I can do with Ansible. For me, the major thing in Chef's favour was the combination of attribute driven configuration and integration with things like Test Kitchen and ServerSpec. Chef requries buy in to the ruby ecosystem though, and that's just not something I want to expose non-ruby developers to (it's just too brittle), which is why I'm exploring Ansible. What I'm finding, though, is that for any reasonably interesting tasks, I'm still forced to execute "command" strings, which is not an improvement over shell, and gives people the false impression that it's idempotent (or even just safer). Also most of these systems are trying to upsell me a centralized management system and UI, handing over the keys to my infrastructure, which is just not something I'm willing to do. Edit: I'd also add that cross-OS portability of CM (if that's what you're referring to), in my experience, a bit of a red herring.
    - 11y ago
- skywhopper 11y ago
  Automation is also my day job. I've got a ton of shell scripts and a ton of Ansible playbooks. No reason they can't work together. For some tasks, shell scripts are better/clearer/easier, and you can call those scripts from Ansible. For other tasks, Ansible provides better tools, and hey, you can call Ansible from your shell scripts! I've added a few ansible-friendly options to some of my shell scripts like adding an --ansible-mode flag that then outputs just a one-line "OK" or "Changed" message that then I have Ansible parse to determine if something changed during the run. Or if you're more ambitious you can just turn your shell scripts into Ansible modules by having them take arguments in a slightly different way and output JSON. A free templating system (Jinja2) and cross-platform capabilities are nothing to shake a stick at either. Not to mention, Ansible does a good job of helping organize your hosts and parameters in one place, and forces you to modularize your "scripts" (in the form of playbooks and roles) in a beneficial way. It's definitely worth a look. You can start small, and use it only for one thing, and grow from there. That's one of its biggest benefits over Puppet and Chef, which don't really have a way to go small-scale.
  - duggan 11y ago
    I do like the "mass SSH" functionality that Ansible provides, it's something accomplished years ago with Capistrano, but I certainly prefer having it in Python (now that I've wrangled together a dynamic inventory interface to Terraform's tfstate files). Shell is not what you want to go to for orchestration, but, essential as it is, I think it's a separate concern from managing the configuration of the machine. You're absolutely right about being able to start small as a major point in its favour, though. Not enough infrastructure tooling accomplishes this.

I've kept an eye on this space---automation is not my day job--and I have to say that I still cannot put my finger on Puppet or Chef or Salt or Ansible.

by hanlec 11y ago

I've kept an eye on this space---automation is not my day job--and I have to say that I still cannot put my finger on Puppet or Chef or Salt or Ansible.

duggan 11y ago

Automation is my day job and I'm still not convinced any of them is a significant improvement over a well maintained set of shell scripts and Makefiles.

berkes 11y ago
Automation is not my day job, and I am very happy that there are frameworks (I work with Chef) with communities that offer standards, guidelines and libraries for a well maintained set of shell scripts and Makefiles. Instead of having to build and maintain them all myself.
- otterley 11y ago
  I've sadly learned that given enough time, you'll likely end up mostly using your own set of private cookbooks, even for configuring things community cookbooks already exist for. The quality of the community cookbooks varies wildly from "pretty good" to "toxic waste," and the likelihood of getting fixes upstream fast enough to reflect back into one's own environment is generally low, even for good maintainers. And Chef's strict adherence to X.Y.Z semver for cookbooks (even though cookbooks don't fit semver well) and inability to declare "my cookbook foo_prime can be used whenever cookbook foo is called" makes it near impossible to maintain your own branch while you wait for your fixes to make it upstream. This makes it difficult to try to get things done when you encounter a cookbook that just doesn't work quite right. The cookbook ecosystem is not yet mature enough at this point in time to rely on.
- duggan 11y ago
  I agree that having standards and best practices is a boon, especially if they can be packaged up in an easy to use way. Like anything, Chef brings its own set of idiosyncrasies to the table. However, I don't like how it brings all of Ruby's idiosyncrasies along for the ride too. If you're not immediately comfortable and familiar with both ruby and the quirks of its ecosystem, you end up investing a lot of time into getting a development environment up and running (and having it reproducible by other members of your team). Most infrastructure tools, if they're any use, are rapidly evolving. Conway's law applies here, and the maintainability of Chef mirrors that of its ruby ecosystem roots - people tend to freeze dependencies rather than upgrade. That's an antipattern we don't need in infrastructure. Honestly I just think we can do better. For me, I'm not sure (read: I really don't know, I'm not being facetious) the costs outweigh the benefits, but for someone totally new, perhaps it helps them get stuff done faster.
kolev 11y ago
Same here. I recently found out about Babashka [0] and Declarative Bash [1], but I've written something similar already. It's much easier to find talent that knows Bash and can learn a readable library of Bash scripts within a day vs finding talent that knows the flavor of the day (every year people are migrating from the old favor to the new - Bash => CFEngine => Pupper => Chef => Salt => Ansible => ... => Bash). [0] https://github.com/richo/babashka https://github.com/richo/babashka [1] https://github.com/threatgrid/declarative.bash https://github.com/threatgrid/declarative.bash P.S. I've recently been fascinated by Rex [2] (They should fix their text logo though - I find it terrible). [2] http://www.rexify.org/ http://www.rexify.org/
- e12e 11y ago
  First I thought this was some dialect of Rexx[1], but then I saw: "Easy to learn, it's just plain Perl". A well. No thank you, I'll stick with ansible. Not that I think Rexx would be a good idea either, it just reminded me of ARexx for AmigaOS... [ed: forgot to ask: you find it easy to find talent that can not only learn some bash script library, but write new code without introducing some gaping hole somewhere?] [1] http://www.rexxla.org/ http://www.rexxla.org/
  - kolev 11y ago
    I'm sorry, but with Rex you don't use much of Perl. It's a DSL, which is based on Perl.
crdoconnor 11y ago
IMHO puppet isn't significantly better than bash scripts. There are some improvements, but it has significant drawbacks too. Ansible is way better though, mainly due to: * Idempotence * Templating * Being declarative (turing completeness in a language = more likely to be buggy)
- e12e 11y ago
  I agree with you, but re: templating: if you're using shell scripts, why not use m4 as well? Shell of course has templating -- but it's a little dangerous...: template=teapot size=short build=stout \ cat <<eof > teapot.conf I'm a $template - $size and $build. I live at $(hostname). eof [ed: m4 is a little like cpp (the c pre-prosessor) on steroids (or acid). To get a feel for how it relates to templating for config files etc, see eg: http://box.matto.nl/m4.html http://box.matto.nl/m4.html ]
  - mercurial 11y ago
    At first glance, I don't see that m4 does anything jinja2 (Ansible's templating system) doesn't do better.
    - e12e 11y ago
      If you're going to use shell and not ansible, then why not use shell with m4. I'm not suggesting shell+m4 is better than ansible, or that ansible+m4 (wtf?! right?) would be a good idea. Just why not use a readily available template/macro language along with shell scripts, if you've decided to use shell scripts. I suppose you could use jinja and bash -- but why pull in two full interpreters as a dependency?
    - 23david 11y ago
      If we're discussing templating languages, I definitely prefer Mako over Jinja2 and think it's much more appropriate for devops templating. But Jinja2 is the default in a lot of projects, including Saltstack and Ansible, and it's hard to fight the currents all the time.
e12e 11y ago
Right. But half-way, somewhat maintained ansible is much better than the similarly maintained shell scripts and Makefiles... And you get somewhat more portability with it. mattjaynes mentioned his blogpost in a previous discussion on this: https://devopsu.com/blog/ansible-vs-shell-scripts/ https://devopsu.com/blog/ansible-vs-shell-scripts/ I think it illustrates the difference well. I agree that you probably could do much of the same with discipline, source control, shell scripts and Makefiles. But there's a reason why we use other languages than shell scripts. Shell scripts are finky, and it takes years for people to stop making "rookie" mistakes. If they ever do. "[" vs "[[", ${var} vs $var, ksh vs bash... the shell gives you way too much rope (and I like shell scripts :-).
- duggan 11y ago
  I should probably couch this in the caveat that I am using Packer and Terraform to control the infrastructure itself :) So machines are effectively configured "offline", tested, and replaced. It's not enough though (usual things like databases and development tweaks like config changes are problematic) so I'm back into the world of config management to see how it can help. Matt makes some good points, and we broadly agree on controlling infrastructure, I just think that most of the CM systems are inadequate abstractions - they don't go far enough, and it leaves them largely as convenience wrappers around shell, but ironically enough, usually without the debuggability of shell. I've worked with Chef for years, and recently have started seeing what I can do with Ansible. For me, the major thing in Chef's favour was the combination of attribute driven configuration and integration with things like Test Kitchen and ServerSpec. Chef requries buy in to the ruby ecosystem though, and that's just not something I want to expose non-ruby developers to (it's just too brittle), which is why I'm exploring Ansible. What I'm finding, though, is that for any reasonably interesting tasks, I'm still forced to execute "command" strings, which is not an improvement over shell, and gives people the false impression that it's idempotent (or even just safer). Also most of these systems are trying to upsell me a centralized management system and UI, handing over the keys to my infrastructure, which is just not something I'm willing to do. Edit: I'd also add that cross-OS portability of CM (if that's what you're referring to), in my experience, a bit of a red herring.
  - 11y ago
skywhopper 11y ago
Automation is also my day job. I've got a ton of shell scripts and a ton of Ansible playbooks. No reason they can't work together. For some tasks, shell scripts are better/clearer/easier, and you can call those scripts from Ansible. For other tasks, Ansible provides better tools, and hey, you can call Ansible from your shell scripts! I've added a few ansible-friendly options to some of my shell scripts like adding an --ansible-mode flag that then outputs just a one-line "OK" or "Changed" message that then I have Ansible parse to determine if something changed during the run. Or if you're more ambitious you can just turn your shell scripts into Ansible modules by having them take arguments in a slightly different way and output JSON. A free templating system (Jinja2) and cross-platform capabilities are nothing to shake a stick at either. Not to mention, Ansible does a good job of helping organize your hosts and parameters in one place, and forces you to modularize your "scripts" (in the form of playbooks and roles) in a beneficial way. It's definitely worth a look. You can start small, and use it only for one thing, and grow from there. That's one of its biggest benefits over Puppet and Chef, which don't really have a way to go small-scale.
- duggan 11y ago
  I do like the "mass SSH" functionality that Ansible provides, it's something accomplished years ago with Capistrano, but I certainly prefer having it in Python (now that I've wrangled together a dynamic inventory interface to Terraform's tfstate files). Shell is not what you want to go to for orchestration, but, essential as it is, I think it's a separate concern from managing the configuration of the machine. You're absolutely right about being able to start small as a major point in its favour, though. Not enough infrastructure tooling accomplishes this.