5 ms·

I am hoping this appears at https://pubs.opengroup.org/onlinepubs/9699919799/ https://pubs.opengroup.org/onlinepubs/9699919799/ soon. This is the link I use mo

by susam 2y ago

I am hoping this appears at https://pubs.opengroup.org/onlinepubs/9699919799/ https://pubs.opengroup.org/onlinepubs/9699919799/ soon. This is the link I use most often to go through the specification. In fact, I owe a lot of my shell scripting skills to this online resource. As a specific example, the seemingly simple matter of when the shell decides to split a string based on $IFS and when it does not were quite confusing to me until I went through the specification here: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V... For example, if a="foo bar" then ls $a will split the value into two fields (thus two arguments to ls). Of course we should surround $a with double-quotes to avoid the field splitting. However the following is fine: case $a in No field splitting occurs here. However, to be kind to your code reviewer, you might want to double-quote this anyway for the sake of simplicity and consistency. Behaviour like this is specified in sections "Field Splitting" and "Case Conditional Construct" of the aforementioned link. Specification documents like this were formative in in my journey toward learning to write shell scripts confidently.

thomashabets2 2y ago
I make a habit to always quote strings with "${a?}". That way a typo variable won't blindly go ahead and do the wrong thing.
- matrss 2y ago
  I can recommend starting every bash script with set -euxo pipefail The "u" has basically the same effect as the question mark, but for every variable usage.
  - ko1nksm 2y ago
    set -o pipefail is now POSIX compliant. Use it in all POSIX shell scripts, not just bash scripts.
    - matrss 2y ago
      Oh wow, I didn't know that. I am pretty sure I had a sh trip up on it just recently, so I thought it still was bash only. Yes, use it everywhere possible.
      - yjftsjthsd-h 2y ago
        Emphasis on now - it's new in this version, so probably expect a little bit of delay before it's actually available everywhere. But yes, excellent addition and I'm happy to have it available more broadly now.
  - cesarb 2y ago
    I do the same, except for the "x" (that is, "set -euo pipefail"); depending on what you're doing, "set -x" might be helpful, might be too much noise, or it could even break things which were not expecting the extra output (and in the worst case, it might end up echoing secret tokens into your build logs).
    - matrss 2y ago
      I find it to be a good default when writing a script, but yes, it can get noisy and potentially leak stuff, if that is what your script deals with. That hasn't been a concern in the settings I usually use it, though. I am not sure how it could break anything though, unless you are parsing stderr of your script in a subsequent step, which would seem unusual anyway.
  - tommiegannert 2y ago
    Oh, it seems set -u was fixed with regards to arrays in 2011 or so: https://stackoverflow.com/questions/7577052/unbound-variable-error-in-bash-when-expanding-empty-array https://stackoverflow.com/questions/7577052/unbound-variable... Maybe I can start using it again. (I think I noticed that issue while I was at Google, and they used an older version of Bash.)
  - Maledictus 2y ago
    I prefer set -Eeuxo pipefail, so my ERR trap is inherited in functions.
thebeardisred 2y ago
The note on the front page of the Austin Groups' website says as much with regards to publication: > June 14, 2024: IEEE Std 1003.1-2024 has been published by IEEE. The Open Group Base Specifications, Issue 8 has been published by The Open Group. At this stage only PDF is available. The HTML edition to follow soon. https://www.opengroup.org/austin/ https://www.opengroup.org/austin/
lordmauve 2y ago
> However the following is fine: > > case $a in > > No field splitting occurs here This kind of bullshit is how I made a career rewriting people's buggy shell scripts in Python
- kstrauser 2y ago
  Wow, no kidding. I’ve been writing little shell scripts to do random things for literally decades and this is the first time I heard about it. I also rewrite my stuff in Python as soon as it becomes nontrivial.
chubot 2y ago
I think many people are even more surprised by this: x=$a # not split! It means the same thing as x="$a" They were taught that you have to quote everything, which is a reasonable rule to follow, but it's not true. --- I never wrote about this on the Oils blog (https://www.oilshell.org/ https://www.oilshell.org/ ), but the post would be titled: Shell Has Context Sensitive Evaluation Basically the two contexts you should think of are: (1) EVAL WORD SEQUENCE This occurs in 2 places in POSIX shell: ls $x$y # simple command is a sequence of words for i in $x$y; do echo $i; done # for loop And 1 place in bash: a=( $x$y ) # array literal In these cases, the shell "wants" a sequence of strings, not a single one. So it does splitting. --- (2) EVAL WORD TO STRING But there are many other contexts where the shell does not "want" a sequence of strings. It wants a SINGLE string. And conversely, it actually JOINS arrays of strings, rather than splitting. Usually "$@" is an array / sequence of strings, while $@ or $* is a string, roughly speaking. But the shell doesn't want sequences of strings in MANY cases, e.g. a=$@ # I only want 1 string here, so I JOIN rather than splitting echo hi > "$@" # redirect arg (not all shells agree though!) case "$@" in ... esac # as you point out So the bottom line is that variables aren't really strings OR arrays of strings. Whatever the shell wants, it converts it to. And shells also DISAGREE on the specifics of those rules. POSIX shell has the array "$@", but arrays in general are not in POSIX. --- And even worse, think about this case: local x=$a Does it behave like an assignment, which wants a single string? Or does it behave like a simple command, which wants a sequence? You can look at it both ways. The bottom line is that assignment builtins are special and they don't follow the normal rules of simple commands. Shells have differed, but POSIX decided on this awhile ago. --- This is all of course mind numbing trivia that has no real reason for existing ... YSH fixes it, and it's now pure native C++, no more Python. YSH Doesn't Require Quoting Everywhere - https://www.oilshell.org/blog/2021/04/simple-word-eval.html https://www.oilshell.org/blog/2021/04/simple-word-eval.html (Oil was renamed to YSH since this blog post was written) Simple Word Evaluation in Unix Shell - https://www.oilshell.org/release/latest/doc/simple-word-eval.html https://www.oilshell.org/release/latest/doc/simple-word-eval... In YSH you can tell just by looking it's a single string or an array. ls $a # identical to ls "$a" ls @myarray # splice an array It never "molests" your variables. There's no auto-conversion, and you can upgrade to those rules with shopt --set ysh:upgrade
- 2y ago

by susam 2y ago

thomashabets2 2y ago

I make a habit to always quote strings with "${a?}". That way a typo variable won't blindly go ahead and do the wrong thing.

matrss 2y ago
I can recommend starting every bash script with set -euxo pipefail The "u" has basically the same effect as the question mark, but for every variable usage.
- ko1nksm 2y ago
  set -o pipefail is now POSIX compliant. Use it in all POSIX shell scripts, not just bash scripts.
  - matrss 2y ago
    Oh wow, I didn't know that. I am pretty sure I had a sh trip up on it just recently, so I thought it still was bash only. Yes, use it everywhere possible.
    - yjftsjthsd-h 2y ago
      Emphasis on now - it's new in this version, so probably expect a little bit of delay before it's actually available everywhere. But yes, excellent addition and I'm happy to have it available more broadly now.
- cesarb 2y ago
  I do the same, except for the "x" (that is, "set -euo pipefail"); depending on what you're doing, "set -x" might be helpful, might be too much noise, or it could even break things which were not expecting the extra output (and in the worst case, it might end up echoing secret tokens into your build logs).
  - matrss 2y ago
    I find it to be a good default when writing a script, but yes, it can get noisy and potentially leak stuff, if that is what your script deals with. That hasn't been a concern in the settings I usually use it, though. I am not sure how it could break anything though, unless you are parsing stderr of your script in a subsequent step, which would seem unusual anyway.
- tommiegannert 2y ago
  Oh, it seems set -u was fixed with regards to arrays in 2011 or so: https://stackoverflow.com/questions/7577052/unbound-variable-error-in-bash-when-expanding-empty-array https://stackoverflow.com/questions/7577052/unbound-variable... Maybe I can start using it again. (I think I noticed that issue while I was at Google, and they used an older version of Bash.)
- Maledictus 2y ago
  I prefer set -Eeuxo pipefail, so my ERR trap is inherited in functions.

thebeardisred 2y ago

The note on the front page of the Austin Groups' website says as much with regards to publication: > June 14, 2024: IEEE Std 1003.1-2024 has been published by IEEE. The Open Group Base Specifications, Issue 8 has been published by The Open Group. At this stage only PDF is available. The HTML edition to follow soon. https://www.opengroup.org/austin/ https://www.opengroup.org/austin/

lordmauve 2y ago

> However the following is fine: > > case $a in > > No field splitting occurs here This kind of bullshit is how I made a career rewriting people's buggy shell scripts in Python

kstrauser 2y ago
Wow, no kidding. I’ve been writing little shell scripts to do random things for literally decades and this is the first time I heard about it. I also rewrite my stuff in Python as soon as it becomes nontrivial.

chubot 2y ago

I think many people are even more surprised by this: x=$a # not split! It means the same thing as x="$a" They were taught that you have to quote everything, which is a reasonable rule to follow, but it's not true. --- I never wrote about this on the Oils blog (https://www.oilshell.org/ https://www.oilshell.org/ ), but the post would be titled: Shell Has Context Sensitive Evaluation Basically the two contexts you should think of are: (1) EVAL WORD SEQUENCE This occurs in 2 places in POSIX shell: ls $x$y # simple command is a sequence of words for i in $x$y; do echo $i; done # for loop And 1 place in bash: a=( $x$y ) # array literal In these cases, the shell "wants" a sequence of strings, not a single one. So it does splitting. --- (2) EVAL WORD TO STRING But there are many other contexts where the shell does not "want" a sequence of strings. It wants a SINGLE string. And conversely, it actually JOINS arrays of strings, rather than splitting. Usually "$@" is an array / sequence of strings, while $@ or $* is a string, roughly speaking. But the shell doesn't want sequences of strings in MANY cases, e.g. a=$@ # I only want 1 string here, so I JOIN rather than splitting echo hi > "$@" # redirect arg (not all shells agree though!) case "$@" in ... esac # as you point out So the bottom line is that variables aren't really strings OR arrays of strings. Whatever the shell wants, it converts it to. And shells also DISAGREE on the specifics of those rules. POSIX shell has the array "$@", but arrays in general are not in POSIX. --- And even worse, think about this case: local x=$a Does it behave like an assignment, which wants a single string? Or does it behave like a simple command, which wants a sequence? You can look at it both ways. The bottom line is that assignment builtins are special and they don't follow the normal rules of simple commands. Shells have differed, but POSIX decided on this awhile ago. --- This is all of course mind numbing trivia that has no real reason for existing ... YSH fixes it, and it's now pure native C++, no more Python. YSH Doesn't Require Quoting Everywhere - https://www.oilshell.org/blog/2021/04/simple-word-eval.html https://www.oilshell.org/blog/2021/04/simple-word-eval.html (Oil was renamed to YSH since this blog post was written) Simple Word Evaluation in Unix Shell - https://www.oilshell.org/release/latest/doc/simple-word-eval.html https://www.oilshell.org/release/latest/doc/simple-word-eval... In YSH you can tell just by looking it's a single string or an array. ls $a # identical to ls "$a" ls @myarray # splice an array It never "molests" your variables. There's no auto-conversion, and you can upgrade to those rules with shopt --set ysh:upgrade

2y ago