5 ms·

Rich’s sh (Posix shell) tricks

54 points by networked 11y ago

julian37 11y ago
Great resource, bookmarked! However, as the author hints at in the last paragraph, if you can get away with it (i.e., if you don't need to support Minix or something) I believe it's better to just save yourself the pain and go with #!/bin/bash or even better, #!/usr/bin/env python. Both are ubiquitous on many systems and much more predictable.
- delan 11y ago
  That’s more true for Linux than other Unix-like systems. Solaris comes with Bourne, FreeBSD comes with that and csh defaulting to Bourne, OpenBSD comes with those and pdksh defaulting to pdksh, and so on.
ufo 11y ago
> If seeing the above hacks has inspired anyone to write a program in a real language rather than sh/Bash/whatever, or to fix corner case bugs arising from the badness of the shell language, I will be happy. What languages does the rest of HN consider to be good alternatives to shell? When I try writing scripts in Python I find myself missing string interpolation and redirection/piping syntax. I tried to learn Perl but I fount it to be too quirky for my tastes. In the end I always come back to sh, even though I know I will regret it later.
- rcfox 11y ago
  For Python, have you tried the sh library? http://amoffat.github.io/sh/ http://amoffat.github.io/sh/ String interpolation isn't nearly as nice as Perl or Bash, but it's possible. http://stackoverflow.com/a/4450610/55 http://stackoverflow.com/a/4450610/55
- zzalpha 11y ago
  You find Perl "too quirky" compared to... POSIX shell scripts. Remarkable. ;)
- dahart 11y ago
  For anything launching processes, I always regret it if I don't use a bash script. (Not sure if you're lumping or separating sh from bash...) I like Python for anything that processes a pile of data, but doesn't need to glue many other programs together. I like Perl if and only if it's a one liner or very straightforward text filter, I just prefer Perl to situations that might otherwise require sed/awk. But for shell scripts, like makefiles, I'd love to see a better alternative, but I just don't think one's out there. For certain kinds of tasks, I don't think a good alternative exists.
- viraptor 11y ago
  I don't think the piping syntax is a big issue. If you use it more than once in a module, just define something like this: (haven't checked, may have errors) def pipe_them(args*): last = subprocess.Popen(args[0], stdout=subprocess.PIPE) for p in args[1:]: last = subprocess.Popen(p, stdin=last.stdout, stdout=subprocess.PIPE) return last proc = pipe_them(['something'], ['something_else', 'args'], ['final_one']) proc.communicate() Interpolation shouldn't be a huge problem either with `% locals()` (if it's a small one-off script).
- jheriko 11y ago
  I think the case he is getting it is when you should be writing an actual program using something like C where you do have access to absolutely everything. He alludes to this in his first comment about exactly when to use shell scripts. > build or bootstrap scripts I agree that python and perl are not good alternatives to shell scripts... even if they are more compatible with e.g. windows environments.
- rbehrends 11y ago
  Tcl, esp. in its form of "tclsh". No, I'm serious. While Tcl has some limitations, it can also be much closer to actual shell programming due to its syntactic flexibility (for example, the DSL to write shell pipelines embodied in the "exec" command). Plus, there's a fairly simple standalone implementation in the form of JimTcl [1], which is used (e.g.) as the configuration tool for autosetup [2]. [1] http://jim.tcl.tk/index.html/doc/www/www/index.html http://jim.tcl.tk/index.html/doc/www/www/index.html [2] https://msteveb.github.io/autosetup/ https://msteveb.github.io/autosetup/
raboukhalil 11y ago
Neat tricks, thanks for sharing! In case anyone's interested, I recently came across a neat trick in Bash to make sure the user specifies a command line argument. Instead of: if [ -z "$1" ]; then echo "Error!!"; exit; fi This basically does the same thing and is more elegant (especially if you need to do this for many arguments): var=${1?Error!!} If the user specifies an argument, it will be saved in $var; if not, Bash will output an error message and will stop execution of the program.
- bburky 11y ago
  Nice trick. It looks like this is actually a POSIX shell feature too, so feel free to use this in portable shell scripts even. Here's the rest of the parameter expansion options. I usually only remember half of them. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...
jheriko 11y ago
You can tell this guy is smart: > I ... consider programming sh for any purpose ... as an extremely misguided endeavor.
id_ris 11y ago
just when I thought I was becoming competent at shell scripts, a post like this reminds me of all the issues that I hadn't even known existed.
101914 11y ago
For shells that lack echo -e and printf (yes, there are such shells; at least, I use one): echoe(){ local a=${#1}; echo $1 |dd bs=1 count=$a;}
Myrmornis 11y ago
fnmatch () { case "$2" in $1) return 0 ;; *) return 1 ;; esac ; } So much for needing Bash’s “[[” command... I've bookmarked this article. However, I think the author advocates too strongly for custom functions which "know better" than native functionality. It seems to be written more for the situation of "my personal shell scripts" than a "codebase". I wouldn't, for example, want to work in a codebase where people have redefined `echo` using the shell function he suggests. With a different name, perhaps.

54 points by networked 11y ago

Great resource, bookmarked! However, as the author hints at in the last paragraph, if you can get away with it (i.e., if you don't need to support Minix or something) I believe it's better to just save yourself the pain and go with #!/bin/bash or even better, #!/usr/bin/env python. Both are ubiquitous on many systems and much more predictable.

delan 11y ago
That’s more true for Linux than other Unix-like systems. Solaris comes with Bourne, FreeBSD comes with that and csh defaulting to Bourne, OpenBSD comes with those and pdksh defaulting to pdksh, and so on.

ufo 11y ago

> If seeing the above hacks has inspired anyone to write a program in a real language rather than sh/Bash/whatever, or to fix corner case bugs arising from the badness of the shell language, I will be happy. What languages does the rest of HN consider to be good alternatives to shell? When I try writing scripts in Python I find myself missing string interpolation and redirection/piping syntax. I tried to learn Perl but I fount it to be too quirky for my tastes. In the end I always come back to sh, even though I know I will regret it later.

rcfox 11y ago
For Python, have you tried the sh library? http://amoffat.github.io/sh/ http://amoffat.github.io/sh/ String interpolation isn't nearly as nice as Perl or Bash, but it's possible. http://stackoverflow.com/a/4450610/55 http://stackoverflow.com/a/4450610/55
zzalpha 11y ago
You find Perl "too quirky" compared to... POSIX shell scripts. Remarkable. ;)
dahart 11y ago
For anything launching processes, I always regret it if I don't use a bash script. (Not sure if you're lumping or separating sh from bash...) I like Python for anything that processes a pile of data, but doesn't need to glue many other programs together. I like Perl if and only if it's a one liner or very straightforward text filter, I just prefer Perl to situations that might otherwise require sed/awk. But for shell scripts, like makefiles, I'd love to see a better alternative, but I just don't think one's out there. For certain kinds of tasks, I don't think a good alternative exists.
viraptor 11y ago
I don't think the piping syntax is a big issue. If you use it more than once in a module, just define something like this: (haven't checked, may have errors) def pipe_them(args*): last = subprocess.Popen(args[0], stdout=subprocess.PIPE) for p in args[1:]: last = subprocess.Popen(p, stdin=last.stdout, stdout=subprocess.PIPE) return last proc = pipe_them(['something'], ['something_else', 'args'], ['final_one']) proc.communicate() Interpolation shouldn't be a huge problem either with `% locals()` (if it's a small one-off script).
jheriko 11y ago
I think the case he is getting it is when you should be writing an actual program using something like C where you do have access to absolutely everything. He alludes to this in his first comment about exactly when to use shell scripts. > build or bootstrap scripts I agree that python and perl are not good alternatives to shell scripts... even if they are more compatible with e.g. windows environments.
rbehrends 11y ago
Tcl, esp. in its form of "tclsh". No, I'm serious. While Tcl has some limitations, it can also be much closer to actual shell programming due to its syntactic flexibility (for example, the DSL to write shell pipelines embodied in the "exec" command). Plus, there's a fairly simple standalone implementation in the form of JimTcl [1], which is used (e.g.) as the configuration tool for autosetup [2]. [1] http://jim.tcl.tk/index.html/doc/www/www/index.html http://jim.tcl.tk/index.html/doc/www/www/index.html [2] https://msteveb.github.io/autosetup/ https://msteveb.github.io/autosetup/

raboukhalil 11y ago

Neat tricks, thanks for sharing! In case anyone's interested, I recently came across a neat trick in Bash to make sure the user specifies a command line argument. Instead of: if [ -z "$1" ]; then echo "Error!!"; exit; fi This basically does the same thing and is more elegant (especially if you need to do this for many arguments): var=${1?Error!!} If the user specifies an argument, it will be saved in $var; if not, Bash will output an error message and will stop execution of the program.

bburky 11y ago
Nice trick. It looks like this is actually a POSIX shell feature too, so feel free to use this in portable shell scripts even. Here's the rest of the parameter expansion options. I usually only remember half of them. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

jheriko 11y ago

You can tell this guy is smart: > I ... consider programming sh for any purpose ... as an extremely misguided endeavor.

id_ris 11y ago

just when I thought I was becoming competent at shell scripts, a post like this reminds me of all the issues that I hadn't even known existed.

101914 11y ago

For shells that lack echo -e and printf (yes, there are such shells; at least, I use one): echoe(){ local a=${#1}; echo $1 |dd bs=1 count=$a;}

Myrmornis 11y ago

fnmatch () { case "$2" in $1) return 0 ;; *) return 1 ;; esac ; } So much for needing Bash’s “[[” command... I've bookmarked this article. However, I think the author advocates too strongly for custom functions which "know better" than native functionality. It seems to be written more for the situation of "my personal shell scripts" than a "codebase". I wouldn't, for example, want to work in a codebase where people have redefined `echo` using the shell function he suggests. With a different name, perhaps.