6 ms·

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any ba

by koolba 5y ago

The first two lines of the script are already wrong; #!/bin/bash Should be: #!/usr/bin/env bash set -euo pipefail That’s table stakes for any bash script. With the first piece, exit on error, being critically important.

switch007 5y ago
Not using “env” isn’t “wrong”. It depends how many platforms they want to supply Bet yes pipefail and nounset should definitely be set.
- switch007 5y ago
  Support*
c0l0 5y ago
I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.
- koolba 5y ago
  Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.
  - eurasiantiger 5y ago
    No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.
    - longcommonname 5y ago
      If I'm writing a bash script I run manually and it's more than a handful of functions I agree. I'm there to babysit it and its probably complicated. If the bash script is ran on thousands of containers where its not possible to babysit. My number one job is to stop immediately when an error happens and surface that error to any monitoring system.
      - eurasiantiger 5y ago
        I agree about that distinction, but still, won’t that error need to be formatted? Is it safe to rely on logging picking up on the error or does the ”simple script” solution imply that there is monitoring for the script exit code?
senko 5y ago
Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin
- pkulak 5y ago
  I'd argue it's easier to find a distro without /usr/bin/env.
- zeec123 5y ago
  NixOS
- jethro_tell 5y ago
  As a system eng for almost 15 years I've spent a good number of years handling deployments, dependancies, environment evaluations and all the little minutiae that's required to run large complex distributed systems at scale. This is fine to ignore if it's your personal box and no one else is working on it. However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' prod setups where the system bash and the deployed environment may contain different bash. The different bash is the one that was tested in the test env with the automation. There may even be multiple different versions of an interpreter on the system with multiple app environments running. This was a pretty common way to package apps before easy access to containers and container managers. This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.
  - senko 5y ago
    I was a sysadmin on SunOS, HP/UX and BSD systems back in the day and have been maintaining various Linux-based systems in the past few decades as well. We're not talking about Google data center here, it's just someone's shell script.
  - scottlamb 5y ago
    > This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field. We don't have industry best practices for shell scripts, no matter what consultants / HN commenters with strong opinions say. You can see in this thread that folks disagree about what the best practices are. It's worth paying attention to folks' rationale for their opinions—I learned some caveats about "-e" from following the links here. But the talk about "table stakes" and "industry best practices" is (extended bleep). Those don't exist. Different projects/companies may have their own best practice guides that make sense in their environment. senko mentioned Google data centers as a place where it might make sense to be more rigorous. Google's guide says to use "#!/bin/bash". [1] If that doesn't work in your environment, fine, but that doesn't make them wrong. Shell is a surprisingly and unnecessarily difficult language to write correctly. To the extent there is a best practice on it, I think it's "use a better language for anything that might become large or important". The Google style guide I linked says more or less the same thing near the beginning. The subtleties discussed in the rest give you a taste of why... [1] https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html
chasil 5y ago
If I may add a few critiques... # Make sure no one else is using rsync pro_on=$(ps aux | grep -c rsync) A better way to do that is with the flock utility. ( flock -n 9 || exit 1 # Critical section-allow only one process. ...single thread shell script ) 9> ~/.empty_lock_file Note that the flock utility is specific to Linux, but POSIX mkdir() is atomic and could be more portable. "${SOURCES[@]}" POSIX shells do not support arrays. Iterating with read over a here document is more portable. minutes=$(($minutes - 1)) POSIX is specific that the $ prefix on a variable name can be omitted in an arithmetic expression. ECHO="/bin/echo" Many shell scripts never use echo, and this is a good idea. 'NEVER use echo like this. According to POSIX, echo has unspecified behavior if any of its arguments contain "\" or if its first argument is "-n".' http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html Perhaps use this instead, in a subshell to avoid stomping on variables: myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )
0xbadcafebee 5y ago
#!/usr/bin/env bash and set -u are always good ideas. There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted. When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like [ "${DEBUG:-0}" = "1" ] && set -x
- scottlamb 5y ago
  > Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted. Really? In this script's "ps aux | grep -c rsync" for example, if "ps" fails, you'll just get 0 without the grep failing. (Speaking of that line: chasil's completely right that it's much better to use "flock" than "ps" for locking...)
- mekster 5y ago
  > There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. You don't let go of -e for that. dont_mind_failure || true important_process add 'true' specifically if you must.
  - 0xbadcafebee 5y ago
    There are scripts where you don't want any failure to kill the script. You would have to wrap nearly every line in that kind of exception handling, and if you miss one, the script fails. It is much simpler to omit -e in that case
LoveGracePeace 5y ago
I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues. Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.
- LeoPanthera 5y ago
  > "#!/bin/bash" works fine It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.
  - deckard1 5y ago
    does it matter? MacOS is using some ancient ass 3.x bash from 2007. The location of bash is the least of your worries when it comes to portability. There's no guarantee any of the commands in your script (a) exist on the system or (b) work with the same options and flags that the script uses. I don't even know how many "netcat" commands I've seen in the wild. You could be running in some BusyBox or pared down Docker container. Blindly running a script that hasn't been deliberately crafted to work on your system is just asking for trouble.
    - mekster 5y ago
      You call 2007 ancient when bash has been around for over 30 years? What exactly has changed in the last 15 years?

by koolba 5y ago

switch007 5y ago

Not using “env” isn’t “wrong”. It depends how many platforms they want to supply Bet yes pipefail and nounset should definitely be set.

switch007 5y ago
Support*

c0l0 5y ago

I wish this misguided notion regarding "inofficial strict mode" would go away, as `set -e` is suboptimal (and can be very tedious to deal with on top of that): https://mywiki.wooledge.org/BashFAQ/105 https://mywiki.wooledge.org/BashFAQ/105 It makes sense to opt into errexit for select blocks/sections in scripts and under certain circumstances, but having it default-on is a recipe for quite a bit of head-scratching in the future.

koolba 5y ago
Using -e isn't an excuse to not understand how it works and when it doesn't. It's so that the default in most situation is to exit on failure as that's likely what you want to do. That leads to more terse scripts which hopefully are easier to write correctly and understand. I'd argue that opting in to exit-on-failure for stand alone commands is the right default.
- eurasiantiger 5y ago
  No, absolutely wrong. The ”right default” is to catch return codes and provide actionable, clear and consistent error messages through an error-exit function.
  - longcommonname 5y ago
    If I'm writing a bash script I run manually and it's more than a handful of functions I agree. I'm there to babysit it and its probably complicated. If the bash script is ran on thousands of containers where its not possible to babysit. My number one job is to stop immediately when an error happens and surface that error to any monitoring system.
    - eurasiantiger 5y ago
      I agree about that distinction, but still, won’t that error need to be formatted? Is it safe to rely on logging picking up on the error or does the ”simple script” solution imply that there is monitoring for the script exit code?

senko 5y ago

Agreed on fail on error, but the first one is needlessly pedantic. Find me a single Linux distro where bash, if installed, is not available in /bin

pkulak 5y ago
I'd argue it's easier to find a distro without /usr/bin/env.
zeec123 5y ago
NixOS
jethro_tell 5y ago
As a system eng for almost 15 years I've spent a good number of years handling deployments, dependancies, environment evaluations and all the little minutiae that's required to run large complex distributed systems at scale. This is fine to ignore if it's your personal box and no one else is working on it. However I've seen it happen quite frequently in systems that were designed with container like 'chroot-lite' prod setups where the system bash and the deployed environment may contain different bash. The different bash is the one that was tested in the test env with the automation. There may even be multiple different versions of an interpreter on the system with multiple app environments running. This was a pretty common way to package apps before easy access to containers and container managers. This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field.
- senko 5y ago
  I was a sysadmin on SunOS, HP/UX and BSD systems back in the day and have been maintaining various Linux-based systems in the past few decades as well. We're not talking about Google data center here, it's just someone's shell script.
- scottlamb 5y ago
  > This is why we have industry best practices, so that people who don't understand why something exists can just follow the best practices and we don't all have to be experts in things outside of our direct field. We don't have industry best practices for shell scripts, no matter what consultants / HN commenters with strong opinions say. You can see in this thread that folks disagree about what the best practices are. It's worth paying attention to folks' rationale for their opinions—I learned some caveats about "-e" from following the links here. But the talk about "table stakes" and "industry best practices" is (extended bleep). Those don't exist. Different projects/companies may have their own best practice guides that make sense in their environment. senko mentioned Google data centers as a place where it might make sense to be more rigorous. Google's guide says to use "#!/bin/bash". [1] If that doesn't work in your environment, fine, but that doesn't make them wrong. Shell is a surprisingly and unnecessarily difficult language to write correctly. To the extent there is a best practice on it, I think it's "use a better language for anything that might become large or important". The Google style guide I linked says more or less the same thing near the beginning. The subtleties discussed in the rest give you a taste of why... [1] https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html

chasil 5y ago

If I may add a few critiques... # Make sure no one else is using rsync pro_on=$(ps aux | grep -c rsync) A better way to do that is with the flock utility. ( flock -n 9 || exit 1 # Critical section-allow only one process. ...single thread shell script ) 9> ~/.empty_lock_file Note that the flock utility is specific to Linux, but POSIX mkdir() is atomic and could be more portable. "${SOURCES[@]}" POSIX shells do not support arrays. Iterating with read over a here document is more portable. minutes=$(($minutes - 1)) POSIX is specific that the $ prefix on a variable name can be omitted in an arithmetic expression. ECHO="/bin/echo" Many shell scripts never use echo, and this is a good idea. 'NEVER use echo like this. According to POSIX, echo has unspecified behavior if any of its arguments contain "\" or if its first argument is "-n".' http://www.etalabs.net/sh_tricks.html http://www.etalabs.net/sh_tricks.html Perhaps use this instead, in a subshell to avoid stomping on variables: myecho () ( z=''; for x; do printf "$z%s" "$x"; z=' '; done; )

0xbadcafebee 5y ago

#!/usr/bin/env bash and set -u are always good ideas. There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted. When your script does fail unexpectedly, you'll want to re-run it with at least tracing enabled, so the third line should be something like [ "${DEBUG:-0}" = "1" ] && set -x

scottlamb 5y ago
> Using pipefail makes it more likely your script will fail unexpectedly and without a known cause. You have to check PIPELINE to see which command in a string of pipes failed and then report on it. This is often pointless, because usually just checking the output of the last pipe will tell you whether you got what you wanted. Really? In this script's "ps aux | grep -c rsync" for example, if "ps" fails, you'll just get 0 without the grep failing. (Speaking of that line: chasil's completely right that it's much better to use "flock" than "ps" for locking...)
mekster 5y ago
> There are cases where you don't want -e enabled, such as when you want to make sure your script makes the best attempt to continue operating even through unknown failures. You don't let go of -e for that. dont_mind_failure || true important_process add 'true' specifically if you must.
- 0xbadcafebee 5y ago
  There are scripts where you don't want any failure to kill the script. You would have to wrap nearly every line in that kind of exception handling, and if you miss one, the script fails. It is much simpler to omit -e in that case

LoveGracePeace 5y ago

I've been doing my scripts for decades on various, numerous, Linux and UNIX systems, real and virtual machines and never used env shell nor have I seen "set -euo pipefail" and have had zero issues. Saying "The first two lines of the script are already wrong;" is wrong. Is that better? IDK, maybe. But "#!/bin/bash" works fine.

LeoPanthera 5y ago
> "#!/bin/bash" works fine It doesn't work at all on any BSD OS, which does not store bash in /bin - instead it is in /usr/local/bin Specifying "env bash" makes it work on any UNIX, since the location of env is a constant, unlike bash.
- deckard1 5y ago
  does it matter? MacOS is using some ancient ass 3.x bash from 2007. The location of bash is the least of your worries when it comes to portability. There's no guarantee any of the commands in your script (a) exist on the system or (b) work with the same options and flags that the script uses. I don't even know how many "netcat" commands I've seen in the wild. You could be running in some BusyBox or pared down Docker container. Blindly running a script that hasn't been deliberately crafted to work on your system is just asking for trouble.
  - mekster 5y ago
    You call 2007 ancient when bash has been around for over 30 years? What exactly has changed in the last 15 years?