9 ms·

What I learned from others' shell scripts

• captn3m0 13y ago
  A good gem I found recently was to use the big version of the command line flags. So instead of seeing -s -q 1, you should use arguments like --max-depth. Increases the readability of scripts by a huge margin.
  • unhammer 13y ago
    Note that some of those long arguments don't work on OS X or BSD, so do some tests if you want to be portable.
    • nonchalance 13y ago
      To be fair, with some commands there are flags which do different things in OSX/BSD and in GNU. So you should verify that all arguments, both short and long, do the same thing As an example, `sed -i` is used for in-place editing. BSD/OSX variants expect an extension argument to be provided after the flag, with an empty string for no backup (sed -i '' 's/foo/bar/' foo.bar) On the other hand, gnu expects the backup extension in the flag (like -W in C compilers), so the command is parsed as if 's/foo/bar/' was the file to open
• praptak 13y ago
  Shell scripts have their use but at the point when your script needs output coloring or printing debug information, it is time to switch to Python. Edit: Place a general disclaimer to weaken the absolute tone of the above statement. Rule of thumb, exceptions apply, yadda, yadda. In other words, I somewhat agree with most responses that disagreed with the statement above :-)
  • mhd 13y ago
    Or Perl, of course. Although I've seen some pretty big collections of shell scripts used for deployment, configuration etc, and all that in a rather heterogeneous Unix environment. Then again, that was Korn Shell, which supports larger programs a bit better than plain bash.
    • cbsmith 13y ago
      Korn shell really ought to be considered "Bourne shell compatible Perl". ;-)
  • fizerkhan 13y ago
    Yes, you are correct. But some people crazy to write everything in shell scripts.
    • Fuxy 13y ago
      I find shell scripts confuses the heck out of me whenever they try to do something a little more complicated the piping. Even this left me a but confused like how do the functions get the input parameters does the * in echo -e "$RED$*$NORMAL" have something to do with it? There's just so many obscure ways of doing the same thing not to mention there's very few good places to teach you shell scripting and shell scripting best practices. I'd rather just go with python. It's easier to read.
      • coolj 13y ago
        http://www.tldp.org/LDP/abs/html/ http://www.tldp.org/LDP/abs/html/ TLDP's Advanced Bash-Scripting Guide is a great resource for learning bash. For example, this section talks about the meaning of "$*": http://www.tldp.org/LDP/abs/html/internalvariables.html#ARGLIST http://www.tldp.org/LDP/abs/html/internalvariables.html#ARGL...
        • barrkel 13y ago
          Free advice: you almost never want $. Almost always what you want is $@, usually in the form "$@". It doesn't matter for this example, but $ is usually a code smell.
      • unhammer 13y ago
        http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide is the ultimate bash guide, along with http://mywiki.wooledge.org/BashFAQ http://mywiki.wooledge.org/BashFAQ For learning the differences between POSIX sh and bash (which can trip you up if you want to run on systems without bash, or if you simply don't know that #!/bin/sh is different from #!/bin/bash), see http://mywiki.wooledge.org/Bashism http://mywiki.wooledge.org/Bashism And regarding how to parse your example, the $* in a string expands to all input parameters to the function, separated by the first character of the IFS variable (e.g. space).
  • 616c 13y ago
    I get what you mean, but there are reasons for shell scripts to stay shell scripts, and not move to Python. One such project [0] is patching Android to allow for a privacy framework. What would Python offer me for diffing and patching that cleanly written shell scripts would not? Shell is still good for one thing: piping commands together. You can do that in Python, but I think even complicated output coloring and debugging scripts should stick to shell when the focus is piping utilities in and out of each other. Python can do that, but it is not its sweet spot, in my opinion. I am sure others disagree. Different tools for different jobs, blah blah. [0] https://github.com/mateor/auto-patcher https://github.com/mateor/auto-patcher
    • praptak 13y ago
      Agreed. Piping and a few of the other syntax niceties ('&&' for conditionally chaining processes is my favorite) are a hard match for even the best process-management library you might have in a general purpose language. Maybe a Lisp DSL could match that, I don't know.
      • klibertp 13y ago
        Check it out and say for yourself - I didn't try it yet, but as a Lisp fan I certainly will: http://www.scsh.net/docu/docu.html http://www.scsh.net/docu/html/man-Z-H-3.html#node_chap_2 I especially love the acknowledgements section :) http://www.scsh.net/docu/html/man.html
        • 616c 13y ago
          Does it still work well? I am very interested in this. I really like Chicken Scheme because it compiles to portable C, and I was wondering why there has not yet been a Scheme shell this morning. Guess I was naive to think there was not without a proper Google-fu session. UPDATE: I take that back, it appears some still works on it. That page just seems outdated, by a while now. https://github.com/scheme/scsh/commits/master https://github.com/scheme/scsh/commits/master
          • 13y ago
  • kamaal 13y ago
    Python is an ideal replacement for small Java programs, not Shell. The moment your script needs any more than 10 regular expressions, or dealing with >3 files all in a complex interplay- use of Perl becomes inevitable. And that is something like the very utmost basic thing you can do with Perl. Python is more like tried-to-be-scripting-but-is-a-web-framework language.
    • rubinelli 13y ago
      I agree that Python is a general-purpose language that could replace much of the Java code you see around, but I never got the feeling that it is particularly well-suited for the web. For one thing, with significant whitespace, it's harder to embed snippets of code in HTML or vice versa, so you have to work with a templating library right from the beginning. Some will see this as a blessing in disguise, but for very small projects or mostly self-contained components, having logic and presentation in one file is much more productive.
      • kamaal 13y ago
        Ironically bulk of all the Python out there is web code. In fact Django, Twisted and Zope is all the Python code there is. Scripting was never Python's forte. Scripting is all about succinctness, power and providing as much power with fewer constructs and restrictions. Which happens to be exactly the very opposite of Python goals, and some thing which more or less as a mission statement Python tries to achieve. This is why Python will likely never be a very successful scripting language. Python was always a web language for frustrated java programmers who couldn't put with java's problems anymore. Much of Python's success is in web frame work area. Which was previously Java's territory.
        • PommeDeTerre 13y ago
          Yes, Python is obviously used for web development. But it's very short-sighted, and even absurd, to claim that the "bulk of all the Python out there is web code". It's even more absurd to claim that "Django, Twisted and Zope is all the Python code there is." That's utter nonsense, in fact. There are numerous non-web applications that use Python extensively, whether they're partially or fully implemented in Python, or whether they can be scripted using Python. Then there are the numerous libraries and frameworks for Python, from GUI toolkits through to scientific computation packages. Many, many organizations use Python internally for a very wide variety of tasks and systems, without broadcasting such use loudly, if at all. This ranges from one-off scripts up to entire multi-million-line software systems. I sure hope that you're joking, but it just isn't coming across as a joke. Nobody can seriously claim that "Python will likely never be a very successful scripting language" when it has undoubtedly been one of the most successful, and versatile, programming and scripting languages around for many years now.
  • noonespecial 13y ago
    I use these languages in ascending order for sys-admin/config: shell -> perl -> python. On OpenWRT systems, sometimes even microperl won't fit. All you've got is shell. (and sh only, none of this fancy bash stuff(1)) Its good to be able to do things at many levels. (1) http://stackoverflow.com/questions/5725296/difference-between-sh-and-bash* http://stackoverflow.com/questions/5725296/difference-betwee...
    • Torgo 13y ago
      At my first "real" job I used to work on a RS-6000, a couple hundred-grand machine. The admin wouldn't install _anything_ non-stock other than a JVM on it. It taught me how to get around using only BSD Unix tools and Bourne Shell. It was an enormous pain in the butt, but what I learned has been utterly invaluable over the course of my career.
    • reidrac 13y ago
      Well, it's more like: shell -> awk -> perl -> python. I use awk from time to time, specially when I need to process a file line by line and the task is too simple to use python (ie. an awk one-liner will do!).
      • noonespecial 13y ago
        Very true. I really need to sharpen my awk/sed/grep skills, but damn, some of those 400 character awk "one-liners" hurt my brain. Perl regexes are rightfully known for being cryptic but those awk statements make me cry for my mama.
  • michaelry 13y ago
    True if python is available, but on a small embedded system this is not always the case.
  • unhammer 13y ago
    I tend to think of colouring and debug output as standard shell script boilerplate. I typically switch to Python (or similar) when I need something like multidimensional arrays / dicts (not portable in awk), or in general want to run more complex parsing or transformations on input. The more advanced string functions of bash I always have to look up (is it ## or %% that removes from the start of the string?), and for/while loops are in general a lot slower. But for file system operations, stringing together commands, keeping track of backgrounded commands (using multiple threads without even thinking about it), and even turning arbitrary programs into "daemons" (using fifo's), nothing beats shell scripting.
    • jzwinck 13y ago
      You say nothing beats shell scripting for: - Filesystem operations. How many times have we seen programs using "ls * .foo" when they needed to use "find -name '*.foo'" to avoid command-line length limits? I answered this very question on StackOverflow this week. How many seemingly-capable software shops will churn out shell scripts which misbehave when a path has a space or other "weird" character in it? Even Apple fell prey to that one, about ten years ago, and destroyed some users' data. - Stringing together commands. This encourages abominations like "cat myfile | grep foo | awk '...'". Just use awk if you're into that, but the shell has a knack for "tricking" people into spawning extra processes that are not really needed (indeed this is one of the most frequent performance sinks in shell scripts). And what about error handling for the several subprocesses? It's usually ignored for N-1 of them. - Keeping track of backgrounded commands (using multiple threads without even thinking about it). Yes, you can use multiple cores without thinking--that can be cool. But what if you want to do N units of work on many fewer than N cores? You ought to use a pool, but there's no such thing in Bash. Maybe you're clever and use "xargs -P" for this, but most people don't. - Turning arbitrary programs into "daemons". I use start-stop-daemon for that (it's included in Debian, and I easily wrote a workalike in Python when I had to use a system that didn't support it natively). Just about the only thing here that shell scripts are really good for is doing things "without even thinking about it." Once when I was asked why shell scripting was not a good idea for production programs, I reviewed a smallish sample Bash script that had been deployed. I found a dozen latent bugs, 50% of which would have never have happened with Python (or Go, or...).
  • Millennium 13y ago
    Python is great for many things, but sometimes you need something lighter. That's not to say that Python is heavy -Java is heavy- but there are environments so constrained that even Python is not light enough. This is where shell scripts can come in handy.
• arek2 13y ago
  Shouldn't it be: which curl >/dev/null 2>&1 ? (what I learned from man bash)
  • infinity0 13y ago
    One "moment of clarity" I had is to realise that 2>&1 is a RIGHT-TO-LEFT assignment - file descriptor 1 gets assigned to file descriptor 2. This is why (to redirect both stdout and stderr to $file) you have to do e.g. 1>$file 2>&1 rather than the more obvious-looking 2>&1 1>$file.
  • chrismorgan 13y ago
    `2>&1 > /dev/null` will drop stdout and write stderr to stdout. `> /dev/null 2>&1` will drop both stdout and stderr. For myself, with `which` I would just use `> /dev/null`, as `which` writes to stdout and not stderr: if anything comes through stderr, you probably want to know about it.
    • arek2 13y ago
      All right, but he wrote: "The 2>&1 > /dev/null puts both output stream and error stream to /dev/null (which means nothing printed on console).". So the intention was different. PS. I think "which" will never produce anything on stderr
      • fizerkhan 13y ago
        As Chris said, `2>&1 > /dev/null` will drop stdout and write stderr to stdout. If you do not want anything to printed on console, you can use it. Now only i came to know that `which` never produce the stderr. So it is enough to use `>/dev/null`.
  • torso 13y ago
    If you're using Bash, you can use this shorthand: which curl >& /dev/null
    • arek2 13y ago
• chrismorgan 13y ago
  I don't like the `require_curl` example. Here's what is done there: OK=0 FAIL=1 function require_curl() { which curl 2>&1 > /dev/null if [ $? -eq 0 ] then return $OK fi return $FAIL } (`2>&1 > /dev/null` drops stdout and writes stderr to stdout—not what was meant. `which` doesn't write to stderr, so I drop that part.) That can be shortened significantly by using the return code directly in the if branch: function require_curl() { if which curl > /dev/null then return $OK fi return $FAIL } Or by just using the return code directly: function require_curl() { which curl > /dev/null return $? } And as it will return the return code of the last statement executed: function require_curl() { which curl > /dev/null }
  • fizerkhan 13y ago
    Thanks, It is very cool. I really donot know `which` does not write to stderr.
    • mordae 13y ago
      When you are not sure and don't want to waste your time investigating, you can do &>/dev/null in bash and get rid of them both at the same time.
      • damncabbage 13y ago
        ... But only in Bash 4. This doesn't appear to work with stock OS X (Bash 3).
        • lelf 13y ago
          Betty:srcs lelf$ perl -E 'say "out"; say STDERR "err";' &> out && cat out err out Betty:srcs lelf$ GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin12)
  • switch007 13y ago
    I believe "return $?" is the default behaviour if you don't have a return statement (i.e. you can require_curl || echo "no curl")
• jalcine 13y ago
  Might inject those `tput` color commands into my shell To those curious: https://github.com/jalcine/dotfiles https://github.com/jalcine/dotfiles
• lelf 13y ago
  function require_curl() { which "curl" > /dev/null; } function require_curl() { which -s "curl"; } function debug { ((DEBUG)) && echo ">>> $*"; } Last one is bash, which -s is BSD'ish
  • cbsmith 13y ago
    Really, just say no to which: https://news.ycombinator.com/item?id=6210440 https://news.ycombinator.com/item?id=6210440
  • mrb 13y ago
    To make debug() POSIX compliant: debug() { [ "$DEBUG" ] && echo ">>> $*"; }
• essrinn 13y ago
  Shell scripts frequently fail to handle directory names with spaces in them. Example from the linked article: APP_ROOT=`dirname $0` This is not going to do what you expect if the current script is run via a command with spaces in one of the preceding directories' name. It should be: APP_ROOT="`dirname "$0"`" Even this would fail if your immediate parent directory's name ended with a newline character, but should handle any other whitespace without issue.
  • damncabbage 13y ago
    The outer quotes aren't required: APP_ROOT=`dirname "$0"` (See http://www.tldp.org/LDP/abs/html/varassignment.html#EX16 http://www.tldp.org/LDP/abs/html/varassignment.html#EX16) Also, even if ^J or ^M were present in a parent directory's name, the command above would still work. Whitespace-like characters take some getting used to with Bash.
• mordae 13y ago
  My eyes bleed. So many unquoted expansions! APP_ROOT=`dirname $0` filename=`basename $filepath .html` should read: APP_ROOT=`dirname "$0"` filename=`basename "$filepath" .html` Plus if your script supports both `--version` and `--help` with proper formats, you can easily generate a manual page with `help2man`. The help output example is far from that. Also, he does not mention getopt at all. The magic line for that is: eval "set -- $(getopt -o hV -l help,version -- "${@}")" || exit $? echo "${*}" It does this: $ ./test foo bar --help --version --help --version -- foo bar Which can be parsed using a `while` loop with `shift`. For other tips on shell scripting best practices I recommend Gentoo ebuilds. They tend to be quite well written. And always read scripts from other people before running them yourself.
  • 616c 13y ago
    I always wanted a shell best practices book (for formatting, beyond the TLDP Advanced Bash Guide, or dare I even say the POSIX spec). I will definitely checkout the Gentoo ebuild scripts. I used to look at Debian in my youth to try and find someone to imitate. However, are there are any people writing lint or prettify or Perl Critic like progs or web services that will properly wrap and do things you mention? Would there be interest in such a thing? UPDATE: I found this stuff. Does anyone use these things? https://trac.id.ethz.ch/projects/bashcritic/ https://trac.id.ethz.ch/projects/bashcritic/ http://man.he.net/man1/checkbashisms http://man.he.net/man1/checkbashisms http://stackoverflow.com/questions/3668665/is-there-a-static-analysis-tool-like-lint-or-perlcritic-for-shell-scripts http://stackoverflow.com/questions/3668665/is-there-a-static...
    • 616c 13y ago
      Also found very cool meet me in the middle type projects, where you can interact with Shell in different programming languages, or "transpile" to it like CoffeeScript -> Javascript. The most interesting so far: Sh.py [0] (mentioned here before [1]) and plumbum [2], which is surprisingly extensive. [0] http://amoffat.github.io/sh/ http://amoffat.github.io/sh/ [1] http://news.ycombinator.com/item?id=4531645 http://news.ycombinator.com/item?id=4531645 [2] https://github.com/tomerfiliba/plumbum https://github.com/tomerfiliba/plumbum
• arnehormann 13y ago
  Huh, no mention of "set -e -u"? Do yourself a favor and follow this, too: http://www.davidpashley.com/articles/writing-robust-shell-scripts/ http://www.davidpashley.com/articles/writing-robust-shell-sc...
  • lysium 13y ago
    I second that! Every shell script should start with "set -e -u": exit if any command fails (except if called as a condition) or if an unset variable is used. Usually, you never want your script to continue running after a command unexpectedly fails or you use an unset variable. Saves so much time debugging!
  • unhammer 13y ago
    That's a great article :-) Regarding traps, I struggled for a while with getting a robust and simple way to kill backgrounded scripts on exit. I would have a script do stuff like "sort bigfile & pid=$!; runlongtask; wait $pid", and on Ctrl-C I wanted the sort command to stop too. The trick is to use "kill 0", which kills the non-interactive script and its subprocesses (see "man 2 kill"). So say you want to both remove some temp directory and kill all subprocesses on Ctrl-C, put this at the top of your script: trap 'rm -rf "$tmp"; kill 0' EXIT
  • unhammer 13y ago
    Also, regarding the lockfile examples, Linux users get a much simpler method using flock: http://mywiki.wooledge.org/BashFAQ/045?highlight=%28flock%29 http://mywiki.wooledge.org/BashFAQ/045?highlight=%28flock%29
• louwrentius 13y ago
  A while ago I was working on a bash function library. The library itself may not be that usefull but it also contains some examples on how to color output and move the cursor around. http://code.google.com/p/bsfl/ http://code.google.com/p/bsfl/
• pavs 13y ago
  As someone who is learning PHP as his first programming language, I am surprised to see similarity in terms to syntax with shell scripts. Are there other popular language with php-like sytax?
  • username42 13y ago
    The syntax of php was inspired by Perl (and C). Perl was inspired by sh/sed/awk.
  • pessimizer 13y ago
    Really algol-like. Most modern procedural languages are as much like PHP as bash is. https://en.wikipedia.org/wiki/ALGOL https://en.wikipedia.org/wiki/ALGOL
• louwrentius 13y ago
  Browse this website and you will learn a lot about bash. Opinionated at times, but that's what I like. If you use Bash scripting you've might encountered this site already, but for what it's worth: http://mywiki.wooledge.org/BashFAQ http://mywiki.wooledge.org/BashFAQ
• ballard 13y ago
  I wrote a silly script for noob openbsders like yours' truly: it builds the usage from functions' comments in the script on-the-fly. https://gist.github.com/6120072 https://gist.github.com/6120072
• herbig 13y ago
  You've got a trailing comma in your about json.
• derekp7 13y ago
  My favorite discovery is how to efficiently do RPC style calls in BASH. Let's say you have some functions and variables you want to execute on a remote server. Just do the following: ssh remotehost " $(declare -p var1 var2 var3) $(declare -f func1 func2 remotemain) remotemain" In this example, var1, var2, var3 and func1, func2 are support variables/functions for the function "remotemain". This pushes all those to the remote side, then calls remotemain.
  • e40 13y ago
    Wow, that is very cool. The ways in which bash is used never cease to amaze me.
  • dmytrish 13y ago
    And don't forget that you can still set the remote environment variables, use stderr output of the remote command, pipe it and so on: $ ssh remotehost "arecord | gzip -c" | gunzip -c | aplay records raw pcm stream an a remote machine, gzip's it and gunzip's and plays on the local machine.
• xxtjaxx 13y ago
  http://mywiki.wooledge.org/Bashism http://mywiki.wooledge.org/Bashism
• jfb 13y ago
  All this article does is make me wish there were an alternative to typical Borne shell garbage. No sane person would ask for a wretched pile of hacks like this.
  • pekk 13y ago
    I guess you'll just have to wait for someone to develop Perl, then...