8 ms·

This recent trend of articles wrt to devops it making me want to post here. Not trying to be too snarky, but why can't people just use shell scripts for automa

This recent trend of articles wrt to devops it making me want to post here. Not trying to be too snarky, but why can't people just use shell scripts for automation? They are easy to repair, in a language you (should) know, reproducible, and testable. They are native to your target, supremely portable, etc, etc.

cheeseprocedure 11y ago
Because it's not worth re-implementing the functionality available in Jenkins, Chef, etc. etc. etc.
- stonogo 11y ago
  This is a misunderstanding of the situation. Jenkins specifically has no business being anywhere near a production deployment; it is a development tool, despite "devops" people tending to use it as a web-interfaced crond. Automation should be tooled to the production environment by people who are familiar with the production requirements and -- most importantly -- by the people who will be expected to maintain the environment and respond to critical issues.
  - boon 11y ago
    What's wrong with using jenkins to invoke the local tooling?
  - rev_bird 11y ago
    > Jenkins specifically has no business being anywhere near a production deployment I'm curious about this -- what's wrong with using the same deployment tool for running dev _and_ the other environments? Jenkins itself is a life-saver for, like you said, doing development builds/testing/reporting, but if we've got one script that deploys an image to production, why not trigger it with a Jenkins job, if only to keep everything centralized and maintain coherent deployment histories?
- nickysielicki 11y ago
  Why do you think most people don't use cfengine?
fweespeech 11y ago
How would you deploy a cluster with highly dynamic portions with shell scripts? You end up with a process that pulls it from a database or something to fill out the configuration. At which point, you might as well just use a script in python or an existing system.
herge 11y ago
Bash scripts (to pick the most popular shell language) are hard to write, hard to debug (no stack traces), have weird historical artifacts, like not having named arguments to functions, or [[ $foo == true ]] or a whole slew of gotchas that make writing and maintaining them unpleasant. If your application is written in ruby or python, why should you learn another (worse scripting, but that's just my opinion) language to deploy it?
- _wmd 11y ago
  > hard to write, hard to debug (no stack traces), have weird historical artifacts Only slightly being sarcy, but this perfectly describes my experience of Salt, Ansible and Chef, but instead of a handful of 'ps' commands to figure out what's going on, you have Python/Ruby interpreters on both sides of the SSH connection, giant gumps of weird SSH command lines, a big C++ library and multitudes of TCP ports and home-grown crypto (Salt+ZeroMQ), perfectly descriptionless error messages like "Timed out", "command failed", or even worse, something you never had to deal with in shell: "template syntax error". I have a personal axe to grind in this department (been thinking about this style of system since the mid 2000s, but never dared start yet another project), so it's entirely possible my expectations are higher than normal here. Still.. processing YAML with Jinja2? Seriously, bash or the most arcane old Makefile syntax was better than this.
  - herge 11y ago
    Yeah, I agree with you, the tools are lacking. My favorite to date is ansible, but it's a pain to guess the syntax to the magic language that ansible has encoded inside yaml. But then that's maybe that's more of an ansible 1.1 or 1.2 issue, not sure if it's relevant anymore. I kinda wish I could just write python scripts that would use ansible as a library.
    - willthames 11y ago
      You can use ansible as a library from python: http://docs.ansible.com/developing_api.html http://docs.ansible.com/developing_api.html
chimeracoder 11y ago
> Not trying to be too snarky, but why can't people just use shell scripts for automation? This is why I love Ansible. Unlike many of the alternatives, Ansible doesn't try to invent a radically different DSL. Instead, ansible uses YAML playbooks that degrade very nicely to shell commands[0]. You can literally take almost any shell script and turn it into an Ansible playbook with a series of Vim regexes - or heck, write a simple sed script to translate it automatically. Of course, if you use Ansible modules instead of the `command` directive, you get a lot of performance benefits and guarantees of idempotency. But it doesn't try to reinvent the wheel; it just builds on top of it. It makes it really easy to hack together an Ansible script that is immediately understood by anyone who's used bash/sh and it makes it possible to improve upon it incrementally by replacing shell commands with the relevant Ansible modules. I've used other automation tools, and the thing that I really disliked about them was that it was so difficult to migrate an existing workflow to a new paradigm. In 2015, systems work is still very imperative, so while declarative syntax is philosophically attractive, I still find it way more practical to use the tools that require smaller mental leaps. [0] e.g., this thing was hacked together from a shell script, and I never bothered to clean it up, but it's still already more readable than the equivalent shell script: https://github.com/ChimeraCoder/znc-kibana-playbooks/blob/master/addswap.yml https://github.com/ChimeraCoder/znc-kibana-playbooks/blob/ma...
- falcolas 11y ago
  My biggest problem, well two problems with Ansible: The maintainer is unusually hostile to user contributions, especially when compared to, say, Salt. A colleague of mine tried to contribute a (fully functional, tested, and documented) change to the MySQL user module to support passing in hashes for user creation - the pull request was closed with a comment like "this can go in a community module". Nevermind that the MySQL user module is a core module... It's also horrendously slow when your playbooks grow. The constant copying back and forth over SSH can be downright crippling. Just running a "green" (the server is in the target state and no changes need to be made) playbook against a local vagrant instance can take over a minute for my current monitoring server playbook. As a point of comparison, Salt completes in about 5 seconds. That said, it's still one of the best "no setup" tools, for the cases where you can't install a minion of one flavor or another. (Yes, theoretically the salt-ssh command line tool exists, but it requires NOPASSWORD sudo on the target server to run)
falcolas 11y ago
For the same reason we don't write all of our backend tools in C: there are better languages for this domain, and better libraries which we can leverage. For example, when running Puppet or Salt, I don't have to worry about which package manager is installed, or mucking about with checksums to see if a file needs to be replaced, or templating, or variables, or... If I'm being honest, just the thought of writing dozens of sed and awk commands to mimic the templating offered by Jinja gives me hives. That said, after fighting for a few years with Puppet, then Ansible, then Salt, I'm thinking that bash and make files are sorely underestimated for some of the simpler provisioning tasks. Bootstrapping a Salt Minion or a salt master is a lot more straightforward in bash than it is in Salt itself.
billybofh 11y ago
Having been doing sysadmin duties for more years than I care to remember and having gone through the gradual process of manual configuration -> shell scripts -> cfengine -> puppet, I'd say despite problems in tools like puppet/salt/etc they are way, way more reliable and sysadmin friendly than shell scripts. For just one example, I've recently begun deploying CentOS 7 boxes whereas previously we mostly used 6 (and some 5). Largely the existing puppet modules kept on working despite major OS changes (hello systemd!). If I'd been maintaining a set of bash (or as used to be, csh) scripts I'd hate to imagine the amount of spaghetti I'd have to have mashed up. We've also got almost every Linux distribution out there you can think of, versions going back a decade or more, Solaris boxes, BSD's and not long got rid of our last AIX box (and I'm going to ignore the last couple of OpenVMS machines). Handling it with hand-rolled scripts was really not fun compared to puppet.
- fweespeech 11y ago
  Congratulations on getting rid of your last AIX box. I wish I could :|
rachelbythebay 11y ago
Your simple script is someone else's bad day. Imagine a shell script that needs to do a dozen things in sequence. Is it just a list of 12 commands? If so, do they even have error checking, whether by "set -e" or with explicit checks? What if it does fail and it gets restarted? Are any of the steps going to cause problems if they run twice or are they smart enough to not do anything this time? How about if you have "critical sections", such that you always have to do A-B-C, and if any of them fail, you have to undo them in reverse, C-B-A? Do any of the parts need locking so that multiple instances don't run at once? Do you need timeouts on any of your calls in case something gets stuck? What cleans up the locks if you exit uncleanly? Will another instance be able to start up and continue? You can handle all of these things in a shell script, but that's not the question. The question is: will someone who's "just writing a quick script" think that far ahead? My experience so far suggests the answer is usually no. It's not the language. It's the mindset.
xorcist 11y ago
It's certainly possible to do with shell scripts and by stringing lots of small utilities such as expect together. But I've been there, and after a few years and a handful of consultants came and went, it's all a big ball of mud. I know how many of my colleagues get spaces in file names, tmp races, and error handling right in their shell scripts, and they're not many. What Puppet et al brings to the table is a bit of structure and best practice. After a two day introduction, anyone can start making changes to our infrastructure, and I'm pretty confident that the new hire in two years' time will understand it as well. And if I need to restore a previous environment, it's all in version control, and will work since it defines state as opposed to transforming it. But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment. Suddenly you can start doing things like tell an application to connect to its auth service, and it'll know which one it is or even mock it if required. Your monitoring software knows every app in every environment, and can configure itself, so it's literally impossible for someone to forget to set up monitoring for a new service. It's a huge win. Once you used to it, you never really understand how you managed without. But you really need to go all in on it to see the benefits. Automated systems integration is possible only when you store configuration in one and only one place.
- Goladus 11y ago
  > What Puppet et al brings to the table is a bit of structure and best practice. > But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment. I agree that is the big win. Though in theory you don't need a configuration management tool to get it. Just sufficient discipline and attention to best-practices. What the config management tools provide, here, is a framework for combining "site-wide" configuration, "local/discovered" facts, and "human-assigned" roles and classes and exposing it all, consistently, to a specialized scripting language. The library of modules/cookbooks/etc. also often take care of a lot tedious logic/details that would otherwise have to be in the code (although this part can still get in the way...).
afarrell 11y ago
Why do I not do everything in bash? Because the syntax for if statements and while loops is godawful. I've taught it to myself at least 30 times. Also, there is no debugger, so I can't step through my code like I can with pub. Also, the tooling around automated testing is worse.

cheeseprocedure 11y ago

Because it's not worth re-implementing the functionality available in Jenkins, Chef, etc. etc. etc.

stonogo 11y ago
This is a misunderstanding of the situation. Jenkins specifically has no business being anywhere near a production deployment; it is a development tool, despite "devops" people tending to use it as a web-interfaced crond. Automation should be tooled to the production environment by people who are familiar with the production requirements and -- most importantly -- by the people who will be expected to maintain the environment and respond to critical issues.
- boon 11y ago
  What's wrong with using jenkins to invoke the local tooling?
- rev_bird 11y ago
  > Jenkins specifically has no business being anywhere near a production deployment I'm curious about this -- what's wrong with using the same deployment tool for running dev _and_ the other environments? Jenkins itself is a life-saver for, like you said, doing development builds/testing/reporting, but if we've got one script that deploys an image to production, why not trigger it with a Jenkins job, if only to keep everything centralized and maintain coherent deployment histories?
nickysielicki 11y ago
Why do you think most people don't use cfengine?

fweespeech 11y ago

How would you deploy a cluster with highly dynamic portions with shell scripts? You end up with a process that pulls it from a database or something to fill out the configuration. At which point, you might as well just use a script in python or an existing system.

herge 11y ago

Bash scripts (to pick the most popular shell language) are hard to write, hard to debug (no stack traces), have weird historical artifacts, like not having named arguments to functions, or [[ $foo == true ]] or a whole slew of gotchas that make writing and maintaining them unpleasant. If your application is written in ruby or python, why should you learn another (worse scripting, but that's just my opinion) language to deploy it?

_wmd 11y ago
> hard to write, hard to debug (no stack traces), have weird historical artifacts Only slightly being sarcy, but this perfectly describes my experience of Salt, Ansible and Chef, but instead of a handful of 'ps' commands to figure out what's going on, you have Python/Ruby interpreters on both sides of the SSH connection, giant gumps of weird SSH command lines, a big C++ library and multitudes of TCP ports and home-grown crypto (Salt+ZeroMQ), perfectly descriptionless error messages like "Timed out", "command failed", or even worse, something you never had to deal with in shell: "template syntax error". I have a personal axe to grind in this department (been thinking about this style of system since the mid 2000s, but never dared start yet another project), so it's entirely possible my expectations are higher than normal here. Still.. processing YAML with Jinja2? Seriously, bash or the most arcane old Makefile syntax was better than this.
- herge 11y ago
  Yeah, I agree with you, the tools are lacking. My favorite to date is ansible, but it's a pain to guess the syntax to the magic language that ansible has encoded inside yaml. But then that's maybe that's more of an ansible 1.1 or 1.2 issue, not sure if it's relevant anymore. I kinda wish I could just write python scripts that would use ansible as a library.
  - willthames 11y ago
    You can use ansible as a library from python: http://docs.ansible.com/developing_api.html http://docs.ansible.com/developing_api.html

chimeracoder 11y ago

> Not trying to be too snarky, but why can't people just use shell scripts for automation? This is why I love Ansible. Unlike many of the alternatives, Ansible doesn't try to invent a radically different DSL. Instead, ansible uses YAML playbooks that degrade very nicely to shell commands[0]. You can literally take almost any shell script and turn it into an Ansible playbook with a series of Vim regexes - or heck, write a simple sed script to translate it automatically. Of course, if you use Ansible modules instead of the `command` directive, you get a lot of performance benefits and guarantees of idempotency. But it doesn't try to reinvent the wheel; it just builds on top of it. It makes it really easy to hack together an Ansible script that is immediately understood by anyone who's used bash/sh and it makes it possible to improve upon it incrementally by replacing shell commands with the relevant Ansible modules. I've used other automation tools, and the thing that I really disliked about them was that it was so difficult to migrate an existing workflow to a new paradigm. In 2015, systems work is still very imperative, so while declarative syntax is philosophically attractive, I still find it way more practical to use the tools that require smaller mental leaps. [0] e.g., this thing was hacked together from a shell script, and I never bothered to clean it up, but it's still already more readable than the equivalent shell script: https://github.com/ChimeraCoder/znc-kibana-playbooks/blob/master/addswap.yml https://github.com/ChimeraCoder/znc-kibana-playbooks/blob/ma...

falcolas 11y ago
My biggest problem, well two problems with Ansible: The maintainer is unusually hostile to user contributions, especially when compared to, say, Salt. A colleague of mine tried to contribute a (fully functional, tested, and documented) change to the MySQL user module to support passing in hashes for user creation - the pull request was closed with a comment like "this can go in a community module". Nevermind that the MySQL user module is a core module... It's also horrendously slow when your playbooks grow. The constant copying back and forth over SSH can be downright crippling. Just running a "green" (the server is in the target state and no changes need to be made) playbook against a local vagrant instance can take over a minute for my current monitoring server playbook. As a point of comparison, Salt completes in about 5 seconds. That said, it's still one of the best "no setup" tools, for the cases where you can't install a minion of one flavor or another. (Yes, theoretically the salt-ssh command line tool exists, but it requires NOPASSWORD sudo on the target server to run)

falcolas 11y ago

For the same reason we don't write all of our backend tools in C: there are better languages for this domain, and better libraries which we can leverage. For example, when running Puppet or Salt, I don't have to worry about which package manager is installed, or mucking about with checksums to see if a file needs to be replaced, or templating, or variables, or... If I'm being honest, just the thought of writing dozens of sed and awk commands to mimic the templating offered by Jinja gives me hives. That said, after fighting for a few years with Puppet, then Ansible, then Salt, I'm thinking that bash and make files are sorely underestimated for some of the simpler provisioning tasks. Bootstrapping a Salt Minion or a salt master is a lot more straightforward in bash than it is in Salt itself.

billybofh 11y ago

Having been doing sysadmin duties for more years than I care to remember and having gone through the gradual process of manual configuration -> shell scripts -> cfengine -> puppet, I'd say despite problems in tools like puppet/salt/etc they are way, way more reliable and sysadmin friendly than shell scripts. For just one example, I've recently begun deploying CentOS 7 boxes whereas previously we mostly used 6 (and some 5). Largely the existing puppet modules kept on working despite major OS changes (hello systemd!). If I'd been maintaining a set of bash (or as used to be, csh) scripts I'd hate to imagine the amount of spaghetti I'd have to have mashed up. We've also got almost every Linux distribution out there you can think of, versions going back a decade or more, Solaris boxes, BSD's and not long got rid of our last AIX box (and I'm going to ignore the last couple of OpenVMS machines). Handling it with hand-rolled scripts was really not fun compared to puppet.

fweespeech 11y ago
Congratulations on getting rid of your last AIX box. I wish I could :|

rachelbythebay 11y ago

Your simple script is someone else's bad day. Imagine a shell script that needs to do a dozen things in sequence. Is it just a list of 12 commands? If so, do they even have error checking, whether by "set -e" or with explicit checks? What if it does fail and it gets restarted? Are any of the steps going to cause problems if they run twice or are they smart enough to not do anything this time? How about if you have "critical sections", such that you always have to do A-B-C, and if any of them fail, you have to undo them in reverse, C-B-A? Do any of the parts need locking so that multiple instances don't run at once? Do you need timeouts on any of your calls in case something gets stuck? What cleans up the locks if you exit uncleanly? Will another instance be able to start up and continue? You can handle all of these things in a shell script, but that's not the question. The question is: will someone who's "just writing a quick script" think that far ahead? My experience so far suggests the answer is usually no. It's not the language. It's the mindset.

xorcist 11y ago

It's certainly possible to do with shell scripts and by stringing lots of small utilities such as expect together. But I've been there, and after a few years and a handful of consultants came and went, it's all a big ball of mud. I know how many of my colleagues get spaces in file names, tmp races, and error handling right in their shell scripts, and they're not many. What Puppet et al brings to the table is a bit of structure and best practice. After a two day introduction, anyone can start making changes to our infrastructure, and I'm pretty confident that the new hire in two years' time will understand it as well. And if I need to restore a previous environment, it's all in version control, and will work since it defines state as opposed to transforming it. But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment. Suddenly you can start doing things like tell an application to connect to its auth service, and it'll know which one it is or even mock it if required. Your monitoring software knows every app in every environment, and can configure itself, so it's literally impossible for someone to forget to set up monitoring for a new service. It's a huge win. Once you used to it, you never really understand how you managed without. But you really need to go all in on it to see the benefits. Automated systems integration is possible only when you store configuration in one and only one place.

Goladus 11y ago
> What Puppet et al brings to the table is a bit of structure and best practice. > But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment. I agree that is the big win. Though in theory you don't need a configuration management tool to get it. Just sufficient discipline and attention to best-practices. What the config management tools provide, here, is a framework for combining "site-wide" configuration, "local/discovered" facts, and "human-assigned" roles and classes and exposing it all, consistently, to a specialized scripting language. The library of modules/cookbooks/etc. also often take care of a lot tedious logic/details that would otherwise have to be in the code (although this part can still get in the way...).

afarrell 11y ago

Why do I not do everything in bash? Because the syntax for if statements and while loops is godawful. I've taught it to myself at least 30 times. Also, there is no debugger, so I can't step through my code like I can with pub. Also, the tooling around automated testing is worse.