13 ms·

Insufficiently known Posix shell features

140 points by rcfox 16y ago

pyre 16y ago
Are those really POSIX or just bash? Namely the $() example. I was under the impression that only `` was POSIX, and that $() was bash/zsh-specific (maybe ksh too). Also, ${CC:=gcc} is only more readable than: [ -z "$CC" ] || CC=gcc if you know what that syntax means. There are a ton of syntax in bash/zsh to modify varables inside of the ${} construct. I usually have to look them up every time that I see them, even when I was the one the wrote the original code. I would say that more people know the 'test' syntax than the ${:=} syntax. > 3. You can assign one variable to another without quoting This is somewhat annoying, because something like this: command="ls -l" $command /path/to/dir doesn't work. It tries to run a command called "ls -l" which probably doesn't exist. Passing that to exec will work, though. A more common example of this issue would be trying to build a command-line in the shell script before executing it, you might have base level of options: base_options="-l -d --debug=1" command $base_options $opt1 $opt2 But $base_options gets passed as a single parameter to command.
- apenwarr 16y ago
  $() is standard, not a bashism. Every code example in the article was tested on multiple shells. Use $() and be happy, and I recommend never touching the backquotes again. They're a disaster as soon as you need to nest them. ${CC:=gcc} is a special case of syntax, but one of the points in the article is: as with any programming language, you should actually learn it so you can write better code. If people start using this one, it won't be any lesser known than any other weird sh syntax. The other variants (=, -, :-, etc) the article recommends against anyway. As for your comments on #3, both of your examples are untrue; they work fine in a POSIX shell. Try it. (They fail in zsh, when zsh isn't in "sh compatibility" mode, so perhaps you're using that. Try switching it to sh compatibility mode.) Either way, those examples aren't actually related to point #3 in the article.
  - pyre 16y ago
    > Use $() and be happy, and I recommend never touching > the backquotes again. Any clue why the 'sh' filetype in Vim highlights $() as an error unless it reads "#!/bin/bash" on the first line of the file? (Zsh actually has its own filetype, but unfortunately no folding built-in support like the sh filetype) There's gotta be a reasoning behind that. Maybe $() isn't POSIX, but unofficially standard b/c most shells support it? [ Note (IIRC) in vim shell filetypes are like this: sh=(sh, bash, ksh) zsh=(zsh) csh=(csh, tcsh) ] > Either way, those examples aren't actually related to point > #3 in the article. They were sort of related in that I was talking about the shell implicitly quoting the contents of the variable. I was just pointing out a different context where (I believe) that it also does something similar. > They fail in zsh, when zsh isn't in "sh compatibility" mode, > so perhaps you're using that. I ran into issues with that a few years ago, and then steered clear of using things in that way. I may well have been using zsh at the time as I was using it for a while before switching to bash, and then back again to zsh (and I didn't keep notes on that timeline).
    - apenwarr 16y ago
      Looks like $() is really really POSIX: http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_03 http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu... In general, beware of trusting the people who wrote your editor instead just trying it to see if it works :)
      - masklinn 16y ago
        I think it would be nifty if every title linked to the relevant section of the POSIX spec (maybe with a [POSIX] link next to the title itself). Also, OpenBSD's pdksh seems to stick quite closely to POSIX shell features, it's probably a good test environment. > In general, beware of trusting the people who wrote your editor instead just trying it to see if it works :) That runs the risk of encountering an additional feature of the current shell compared to POSIX, this is very much the source of all bashisms: just trying to see if it works.
zacharyvoase 16y ago
Here's another (much cleaner) way of reading two variables: $ { read A; read B } < <(echo "C"; echo "D") $ echo $A C $ echo $B D See http://tldp.org/LDP/abs/html/process-sub.html http://tldp.org/LDP/abs/html/process-sub.html for more information.
- apenwarr 16y ago
  That only works in bash.
  - rbanffy 16y ago
    Is there any modern serious OS that does not provide bash?
    - durin42 16y ago
      Yes, and /bin/sh is often not bash. A sufficiently hosed system may give you only /bin/sh until you can repair things, and knowing these tricks in those cases can be a real lifesaver.
      - rbanffy 16y ago
        A sufficiently hosed system may not even boot. Under normal circumstances, however, you have a full set of shells, depending on how... feature-rich is the OS you are using.
    - naner 16y ago
      Is there any modern serious OS that does not provide bash? By default? Windows (heh). And most Unix servers don't come with bash by default. Though I think you can install it on most anything, it is not a good idea to change the default root shell on Unix servers. That being said, OSX and Linux have Bash as the default shell. If that's all you deal with then go ahead and learn Bash (and the other GNU utilities) instead of POSIX sh. Unless you're an admin there's no point in being so pedantic.
mansr 16y ago
For writing portable scritps, I strongly advise against using anything not in the POSIX spec. No matter how many shells you've tested it in, there will always be one somewhere that doesn't implement it. Conversely, if a POSIX feature is found to be broken in at least one moderately common shell, it should be avoided or worked around if at all practical.
- asb 16y ago
  I like the ability to have local variables though, which isn't in the POSIX standard. However, the Debian Policy Manual does state that any /bin/sh shell should be posix-compliant and support local variables (along with a couple of other minor extensions). This means I'm fairly comfortable using them despite their absence from the standard. http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts http://www.debian.org/doc/debian-policy/ch-files.html#s-scri...
- apenwarr 16y ago
  This is generally good advice, but redo tries to do something else: it actually tests all your shells, and picks the one that supports the features we're expecting. That way we can set a minimum baseline. All the features in the article are tested by redo, so at least in your redo scripts, you can assume those features are available. Maybe someone can generalize the "redo-sh" feature somehow.
- Create 16y ago
  I would just like to mention, that in order to really take the pain out of portable scripting, the easiest way to go is python scripts - and forget about the differences between win32, bash, csh and the rest.
  - bch 16y ago
    That's what I do (s/Python/Tcl/). I feel pretty comfortable doing it, but a -little- uneasy. I feel like I'm cheating; I usually prescribe (and follow) "as dumb as possible (but no dumber)" rule, and use /bin/sh as my interactive shell to keep from relying on non-portable "creature-features". Using /bin/sh satisfies the "as dumb as possible..." rule because it's basic, standard, and everywhere. However, as soon as a shell script starts getting moderately complicated, I feel more comfortable moving to something (Tcl) that's (to my mind) less obtuse and less surprising. edit: and then I read http://news.ycombinator.com/item?id=2271004 http://news.ycombinator.com/item?id=2271004 further below, which makes Tcl (or python, or ...) look even better.

140 points by rcfox 16y ago

pyre 16y ago

Are those really POSIX or just bash? Namely the $() example. I was under the impression that only `` was POSIX, and that $() was bash/zsh-specific (maybe ksh too). Also, ${CC:=gcc} is only more readable than: [ -z "$CC" ] || CC=gcc if you know what that syntax means. There are a ton of syntax in bash/zsh to modify varables inside of the ${} construct. I usually have to look them up every time that I see them, even when I was the one the wrote the original code. I would say that more people know the 'test' syntax than the ${:=} syntax. > 3. You can assign one variable to another without quoting This is somewhat annoying, because something like this: command="ls -l" $command /path/to/dir doesn't work. It tries to run a command called "ls -l" which probably doesn't exist. Passing that to exec will work, though. A more common example of this issue would be trying to build a command-line in the shell script before executing it, you might have base level of options: base_options="-l -d --debug=1" command $base_options $opt1 $opt2 But $base_options gets passed as a single parameter to command.

apenwarr 16y ago
$() is standard, not a bashism. Every code example in the article was tested on multiple shells. Use $() and be happy, and I recommend never touching the backquotes again. They're a disaster as soon as you need to nest them. ${CC:=gcc} is a special case of syntax, but one of the points in the article is: as with any programming language, you should actually learn it so you can write better code. If people start using this one, it won't be any lesser known than any other weird sh syntax. The other variants (=, -, :-, etc) the article recommends against anyway. As for your comments on #3, both of your examples are untrue; they work fine in a POSIX shell. Try it. (They fail in zsh, when zsh isn't in "sh compatibility" mode, so perhaps you're using that. Try switching it to sh compatibility mode.) Either way, those examples aren't actually related to point #3 in the article.
- pyre 16y ago
  > Use $() and be happy, and I recommend never touching > the backquotes again. Any clue why the 'sh' filetype in Vim highlights $() as an error unless it reads "#!/bin/bash" on the first line of the file? (Zsh actually has its own filetype, but unfortunately no folding built-in support like the sh filetype) There's gotta be a reasoning behind that. Maybe $() isn't POSIX, but unofficially standard b/c most shells support it? [ Note (IIRC) in vim shell filetypes are like this: sh=(sh, bash, ksh) zsh=(zsh) csh=(csh, tcsh) ] > Either way, those examples aren't actually related to point > #3 in the article. They were sort of related in that I was talking about the shell implicitly quoting the contents of the variable. I was just pointing out a different context where (I believe) that it also does something similar. > They fail in zsh, when zsh isn't in "sh compatibility" mode, > so perhaps you're using that. I ran into issues with that a few years ago, and then steered clear of using things in that way. I may well have been using zsh at the time as I was using it for a while before switching to bash, and then back again to zsh (and I didn't keep notes on that timeline).
  - apenwarr 16y ago
    Looks like $() is really really POSIX: http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_03 http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu... In general, beware of trusting the people who wrote your editor instead just trying it to see if it works :)
    - masklinn 16y ago
      I think it would be nifty if every title linked to the relevant section of the POSIX spec (maybe with a [POSIX] link next to the title itself). Also, OpenBSD's pdksh seems to stick quite closely to POSIX shell features, it's probably a good test environment. > In general, beware of trusting the people who wrote your editor instead just trying it to see if it works :) That runs the risk of encountering an additional feature of the current shell compared to POSIX, this is very much the source of all bashisms: just trying to see if it works.

zacharyvoase 16y ago

Here's another (much cleaner) way of reading two variables: $ { read A; read B } < <(echo "C"; echo "D") $ echo $A C $ echo $B D See http://tldp.org/LDP/abs/html/process-sub.html http://tldp.org/LDP/abs/html/process-sub.html for more information.

apenwarr 16y ago
That only works in bash.
- rbanffy 16y ago
  Is there any modern serious OS that does not provide bash?
  - durin42 16y ago
    Yes, and /bin/sh is often not bash. A sufficiently hosed system may give you only /bin/sh until you can repair things, and knowing these tricks in those cases can be a real lifesaver.
    - rbanffy 16y ago
      A sufficiently hosed system may not even boot. Under normal circumstances, however, you have a full set of shells, depending on how... feature-rich is the OS you are using.
  - naner 16y ago
    Is there any modern serious OS that does not provide bash? By default? Windows (heh). And most Unix servers don't come with bash by default. Though I think you can install it on most anything, it is not a good idea to change the default root shell on Unix servers. That being said, OSX and Linux have Bash as the default shell. If that's all you deal with then go ahead and learn Bash (and the other GNU utilities) instead of POSIX sh. Unless you're an admin there's no point in being so pedantic.

mansr 16y ago

For writing portable scritps, I strongly advise against using anything not in the POSIX spec. No matter how many shells you've tested it in, there will always be one somewhere that doesn't implement it. Conversely, if a POSIX feature is found to be broken in at least one moderately common shell, it should be avoided or worked around if at all practical.

asb 16y ago
I like the ability to have local variables though, which isn't in the POSIX standard. However, the Debian Policy Manual does state that any /bin/sh shell should be posix-compliant and support local variables (along with a couple of other minor extensions). This means I'm fairly comfortable using them despite their absence from the standard. http://www.debian.org/doc/debian-policy/ch-files.html#s-scripts http://www.debian.org/doc/debian-policy/ch-files.html#s-scri...
apenwarr 16y ago
This is generally good advice, but redo tries to do something else: it actually tests all your shells, and picks the one that supports the features we're expecting. That way we can set a minimum baseline. All the features in the article are tested by redo, so at least in your redo scripts, you can assume those features are available. Maybe someone can generalize the "redo-sh" feature somehow.
Create 16y ago
I would just like to mention, that in order to really take the pain out of portable scripting, the easiest way to go is python scripts - and forget about the differences between win32, bash, csh and the rest.
- bch 16y ago
  That's what I do (s/Python/Tcl/). I feel pretty comfortable doing it, but a -little- uneasy. I feel like I'm cheating; I usually prescribe (and follow) "as dumb as possible (but no dumber)" rule, and use /bin/sh as my interactive shell to keep from relying on non-portable "creature-features". Using /bin/sh satisfies the "as dumb as possible..." rule because it's basic, standard, and everywhere. However, as soon as a shell script starts getting moderately complicated, I feel more comfortable moving to something (Tcl) that's (to my mind) less obtuse and less surprising. edit: and then I read http://news.ycombinator.com/item?id=2271004 http://news.ycombinator.com/item?id=2271004 further below, which makes Tcl (or python, or ...) look even better.