14 ms·

Bash Pitfalls

288 points by aaron-santos 6y ago

mey 6y ago
https://web.archive.org/web/20200907092725/https://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20200907092725/https://mywiki.wo... Since the site is being hugged a little too much. I also really like https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- scns 6y ago
  Shellcheck and IdeaVim are the first plugins i install on Jetbrains IDEs
  - pletnes 6y ago
    Shellcheck is, in general, one of the most undervalued dev tools hiding in the bushes. Especially as everyone has to shell at least a little bit, in particular linux newbies.
- INTPenis 6y ago
  Seeing this wiki hugged to death by HN makes me realize that it would be perfect for migrating to some simple cloud hosting and using static markdown in git. All their maintainers are already technical enough to handle doing updates in git. And with certain static doc generators like mkdocs you can have an Edit link in each page that brings the user to a Gitlab repo with an editor and Preview.
  - pathseeker 6y ago
    Throwing everything at another company so you can blame your outages on them does not a better engineer make. This site is just doing something silly dynamically for something that should be static (e.g. reading from a database). A single modern computer can easily handle the HN hug of death serving static files.
ryangittins 6y ago
Looks like HN has hugged this one to death. https://web.archive.org/web/20200907174629/https://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20200907174629/https://mywiki.wo...
Ambol 6y ago
These type of things is why I always use a real programming language rather than shell scripting. Can't wrap my head around all the different pitfalls of treating random string outputs as input for other commands without first validating and parsing them into an actual data structure.
- as-j 6y ago
  Shell scripting has it’s place. In general it works fairly well and it’s an important skill to have. I used to have a report who had a similar train of thought, and ended up writing shell scripts in python which was way more fragile, unreadable and more complicated. The edge cases are there, but you don’t run into them all that often. Or if you do, you fix your scripts. Take the cp —- 2nd example. You generally don’t start file names with a -. It just makes Unix life annoying so you get pushed away from doing it. Keep scripts sane and straight forward, you can do really nice and powerful work knowing Unix tools. You can also shoot yourself in the foot some day when you create a file called ‘-rf’.
  - 3pt14159 6y ago
    Bash is like a spec ops soldier's knife. Sure it could be used to kill someone (Bash on Balls, anyone?), but it's mostly for doing boring stuff like opening MREs (getting around a filesystem) or removing a deep thorn from the skin (removing a stupid "$n/a" string in a one-use TSV). I wouldn't send a navy seal into battle without one, but it isn't an absolutely critical piece of gear.
- Spivak 6y ago
  The cost you pay for this is verbose and unergonomic interop when your problem domain is executing a bunch of commands. For all its warts shells shine at being extremely expressive. It’s usually not worth the effort to shove everything into a real programming language but write a few self-contained commands designed to be called from a shell and then use the shell as the glue.
- rantwasp 6y ago
  yeah no. you should start with bash and drop into a proper programming language when complexity warrants it
rashkov 6y ago
Is there a greater principle behind all of these pitfalls, a mental model that I can learn and use to avoid these kinds of mistakes? Or is it sort of like CSS where years and years of ad-hoc development make it an exercise in memorizing edge-cases?
- Spivak 6y ago
  Some off the top of my head. * On Linux file and directory names can be anything except NUL. Because of how much software already assumes they’re UTF-8 you can probably safely assume it too and not run into much trouble. But since bash basically just does some text substitutions and then calls eval you have to be defensive. * Know how bash parses front and back. Almost all the pitfalls are unexpected consequences of the parser not behaving how you expect. https://web.archive.org/web/20200124085320/http://mywiki.wooledge.org/BashParser https://web.archive.org/web/20200124085320/http://mywiki.woo...
  - mfontani 6y ago
    IIRC directory (and file) names can contain anything but slash and null, not just null.
  - bewuethr 6y ago
    Pretty much this, and as a subset of it: quoting, word splitting, the IFS variable and friends. Reading the BashGuide [1] cover to cover is an excellent base, I find. [1]: https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
  - doctoboggan 6y ago
    Do you know if the `--` trick mentioned in the article to stop option scanning works with all bash programs? I thought parsing was up to the program itself. Is it just a widely used convention?
    - scbrg 6y ago
      You are correct, option parsing is up to each program. The double dash is just a convention.
deleted 6y ago
[deleted]
kennywinker 6y ago
Bash pitfall #1: deciding to use bash
- jjgreen 6y ago
  So true, bash is a fine interactive shell, but for scripts the Bourne shell is almost as capable, portable as in POSIX and did not suffer shellshock.
nikisweeting 6y ago
I consider shellcheck absolutely essential if you're writing even a single line of Bash. I also start all my scripts with this "unofficial bash strict mode" and DIR= shortcut: #!/usr/bin/env bash ### Bash Environment Setup # http://redsymbol.net/articles/unofficial-bash-strict-mode/ # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html # set -o xtrace set -o errexit set -o errtrace set -o nounset set -o pipefail IFS=$'\n' DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" I have more tips/tricks here: https://github.com/pirate/bash-utils/blob/master/util/base.sh https://github.com/pirate/bash-utils/blob/master/util/base.s...
- rantwasp 6y ago
  set -euxo pipefail
  - ed25519FUUU 6y ago
    Every script. Good enough.
  - Rapzid 6y ago
    Ahh, errexit: unless the command that fails is part of an until or while loop, part of an if statement, part of a && or || list, or if the command's return status is being inverted using !. All hope is lost.
- rgrau 6y ago
  You've a lot of cool stuff in that repo! I hope you don't mind me 'stealing' some stuff for a kind of booklet I'm working on with several shell tips and tricks. https://github.com/kidd/scripting-field-guide/ https://github.com/kidd/scripting-field-guide/ PS: Any comments/PRs are super welcome there too
  - nikisweeting 6y ago
    No problem, it's all MIT licensed so it's not stealing haha. See my parent comment with my full reading list as well. If you want to link back to my repo as a cited source/credit that would be nice, but is not required :)
zests 6y ago
Bash is one of my favorite programming languages. Not for production. Writing significant production Bash that others have to use and maintain is more of a struggle than it is worth. But for personal use? Hitting APIs, organizing data, writing to files... Bash has been enjoyable because it requires much more creativity than almost any other language. At some point it's just you and the coreutils against the world.
- bor100003 6y ago
  I have the same feeling. In most cases you do the job without any external libs which is rarely the case with other programming languages.
phibz 6y ago
I wonder if there's a linter to check for these issues.
- boogies 6y ago
  Shellcheck? https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
znpy 6y ago
It's both interesting to read and meh. It's meh because like many things, you actually have to read the fine manual if you want stuff to actually work. It's interesting because it's always nice to have a refresher and/or reminder of various bash things. That being said... I've said this many times, I'll say it again: bash scripts are software, and thus must not be exempted from various programming best practices. The first thing that comes to my mind is input validation and error checking. Bash is not magical. It's one of those things that have to be mastered through practice... Just like many other programming languages.
amirkdv 6y ago
Years ago I spent a lot of time (arguably too much time) writing (ba)sh scripts and this document was a great resource! But my main lesson from that period was that bash pitfalls are readability/maintenance nightmares, confounded by the fact that most people don't know about most of the pitfalls. To make matters worse, you can't know which pitfalls are or aren't known to the original author / future maintainer of the script in front of you. But I'm not sure to what extent the language warts are inevitable consequences of what makes it so expressive and powerful. For example: unix pipes are amazing and the bash syntax for them is quite expressive. I can reproduce the same behavior in any of the more "sane" languages but I find that the equivalent code is a less readable procedural soup of string munging, control flow, and error control.
zerofrancisco 6y ago
Excellent caveats and suggestions to do good bash. Thank you! :)
1vuio0pswjnm7 6y ago
"13. cat file | sed s/foo/bar/ > file You cannot ..." With sed, in some cases, you can. One of the differences between Linux-provided sed and BSD-provided sed is the -a option. FreeBSD manpage: "-a The files listed as parameters for the "w" functions are created (or truncated) before any processing begins, by default. The -a option causes sed to delay opening each file until a command containing the related "w" function is applied to a line of input." So let's say you have a file cat > file sandfoo ^D BSD-provided sed cat file|sed -na 's/foo/bar/;H;$!d;g;w'file The replacement is made without any temp file. Note this example will add a blank line at the top. Why use sed? sed is part of BSD base system, the toolchain^1 and install images thus it can easily be utilised early in boot process. I am not aware that "sponge" is part of any base system. 1. NetBSD's toolchain can be reliably cross-compiled on Linux.
- koala_man 6y ago
  An easier way to do the same thing but for any tool is `cat file | sed 's/foo/bar/' | sponge file`
  - yjftsjthsd-h 6y ago
    Good old moreutils:)
- samoa42 6y ago
  what is wrong with the '-i' option? -i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if SUFFIX supplied)
  - 1vuio0pswjnm7 6y ago
    sed -i creates a temp file. Needs additional storage space the size of the file. This can pose a problem with large files on smaller computers with limited storage and memory. In the past, some BSD seds (other than FreeBSD) did not have the -i option. Also worth noting that "sponge", unlike sed, apparently reads the entire file into memory. For a large file, this would require enough RAM to fit the entire file.
baby 6y ago
To me this screams: don't use bash in production.
- jakeogh 6y ago
  Yea. Or test the heck out of it. Once it works, it tends to work well. It took me ~8 years (so far, I'm sure there is a bug somewhere) and considerable #bash advice to make a simple script that locks commands+args https://github.com/jakeogh/commandlock https://github.com/jakeogh/commandlock I also wrote a MDA wrapper for gnupg, and again, it took years to feel good about having my mail filter through it https://github.com/jakeogh/gpgmda https://github.com/jakeogh/gpgmda Now that py has pathlib, if it's more than a few lines, py with @click is just way better. All that is really missing is cleaner copy/move abstractions. There are also some features that are missing from all shells... like the ability of a script to know if expansion happened before it got $@... that internal state just isn't exposed.
bluedays 6y ago
Oh neat. I found a problem with a script I wrote today because of this. I wrote a script recently that takes several screenshots from the same folder, merges them using Image Magick and outputs the merged image into a PDF. Found out through the first tip that I shouldn't be using ls: https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29 https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_....
asicsp 6y ago
See also: * BashFAQ - https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * shellharden: safe ways to do things in bash - https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... * Security implications of forgetting to quote a variable in bash/POSIX shells - https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells https://unix.stackexchange.com/questions/171346/security-imp...
- CaptainZapp 6y ago
  Especially the first link is incredibly useful to me. Thanks a lot!
slavik81 6y ago
I was writing a script recently and I was going to use `cp -- "$src" "$dst"` but neither `man cp` nor `cp --help` mentioned `--` as an option. I'm not sure I'm using info right, but it doesn't seem to be mentioned in `info cp`, either. Is that feature described somewhere else, or is it just undocumented?
- JdeBP 6y ago
  The -- mechanism in general, not specific to the cp command albeit mostly limited to programs that use getopt/popt for their argument parsing, is described all over the place. I first encountered it in a book, written by Eric Foxley, entitled UNIX for Super-Users and published in 1985. * It's perennially asked about on Unix and Linux Stack Exchange, ranging from https://unix.stackexchange.com/q/11376/5132 https://unix.stackexchange.com/q/11376/5132 almost a decade ago to https://unix.stackexchange.com/q/570729/5132 https://unix.stackexchange.com/q/570729/5132 last year. * It's in a whole bunch of duplicate questions there, including (but not limited to) https://unix.stackexchange.com/questions/linked/1519?lq=1 https://unix.stackexchange.com/questions/linked/1519?lq=1 . * It's in the Single Unix Specification, and has been for a long time. (https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilconv.html#tag_009_002 https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilcon...) * It's in the getopt(3) manual page from the Linux man-pages project. It's in the getopt(1) manual page from the util-linux project. * It's in getopt(1) and getopt(3) in the NetBSD Manual. The same goes for the OpenBSD and FreeBSD manuals. * It's in Blum's and Bresnehan's 2015 Linux Command Line and Shell Scripting Bible, on page 377. * It's in Marcel Gagné's 2002 Linux System Administration: A User's Guide, on page 48. * It's in Kirk Waingrow's 1999 UNIX Hints & Hacks. * It was Q&A number 32 in part 2 of the Usenet comp.unix.shell FAQ document. (http://cfajohnson.com/shell/cus-faq-2.html#Q32 http://cfajohnson.com/shell/cus-faq-2.html#Q32) Specific to the cp command, or rather to the many cp commands that you could end up using, as this is not a "bash command": * No, the OpenBSD, FreeBSD, and NetBSD cp(1) manuals do not mention it. * Neither does the GNU CoreUtils cp(1) manual page, as you noted. It's instead in the GNU CoreUtils doco section on "common options". The actual CoreUtils cp(1) manual doesn't mention that there even are "common options". (https://manpages.debian.org/sid/coreutils/cp.1.en.html https://manpages.debian.org/sid/coreutils/cp.1.en.html) That's only mentioned, as an aside, in the info page. (https://gnu.org/software/coreutils/manual/html_node/cp-invocation.html#cp-invocation https://gnu.org/software/coreutils/manual/html_node/cp-invoc...) * For the cp command that is part of Rob Landley's toybox, this is covered in the introduction to the toybox(1) manual page, and there isn't really a distinct cp(1) manual page. (http://landley.net/toybox/help.html http://landley.net/toybox/help.html) * No, BusyBox doesn't similarly document this for its cp command, or anywhere on busybox(1). (https://busybox.net/downloads/BusyBox.html https://busybox.net/downloads/BusyBox.html) * It's tangentially in several GNU CoreUtils FAQ Q&As, inasmuch as CoreUtils cp is the same as CoreUtils rm in this regard. (https://gnu.org/software/coreutils/faq/coreutils-faq.html#How-do-I-remove-files-that-start-with-a-dash_003f https://gnu.org/software/coreutils/faq/coreutils-faq.html#Ho...) * It's tangentially in the Linux Documentation Project's Advanced Bash-Scripting Guide, again because cp is the same as rm here. (https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHREF https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHR...) * For an operating system where this is actually in the cp(1) manual itself, look to Solaris, and its successor Illumos, where you'll find it in the NOTES section. (https://illumos.org/man/1/cp https://illumos.org/man/1/cp) * Another operating system where this is actually in the cp(1) manual itself is AIX, actually in the FLAGS table alongside all of the other flags. (https://ibm.com/support/knowledgecenter/ssw_aix_72/c_commands/cp.html https://ibm.com/support/knowledgecenter/ssw_aix_72/c_command...)
chousuke 6y ago
One new pitfall I learned yesterday is that sourcing a file searches $PATH. I ran a tool that did ". config" on OpenBSD and it tried to source config(8) instead of its configuration in the local directory. Oops.
- JdeBP 6y ago
  On the subject of OpenBSD, sourcing configuration files, and security: Enjoy https://unix.stackexchange.com/a/433245/5132 https://unix.stackexchange.com/a/433245/5132 .
rurban 6y ago
I have almost no bash scripts. Whenever sh is not good enough and I have to switch to bash, I'd rather rewrite it in proper Perl. Bash really looks like a hack, and not all environments have bash. Everybody has a conforming sh.
inoffensivename 6y ago
It would be easier to list all the constructs in bash that are well-behaved and intuitive.
bonoboTP 6y ago
The biggest transformative learning experience for me in Unix command gluing was to understand argument vectors. That a command execution is not just one big string, but will at the point of calling the command, be transformed by Bash to a vector of strings, to be passed on to the execve system call. The point is, all those quotes etc. are syntax we use to tell the Bash I terpreter what the strings passed on should be. Think in terms of the string vector and not in terms of the one big string in front of your eyes that is a line of Bash code. The latter is a recipe for memorizing magic incantations and cargo cult sprinkling of quotes here and there. The horrible thing with Bash is that while normal programming languages make the difference crystal clear between the name of a function being called and its mandatorily quoted string args neatly separated with commas, Bash will come up with this split on the spot using arcane rules. Letting a string automatically fall apart into multiple argemts on the spot may have seemed convenient in the old days for forwarding invocations, but it seems clear to me that this was a design mistake. In the end if you want to have robust Bash code you have to keep track of string splitting and will almost never rely on the default behavior of letting Bash "helpfully" pre-digest and split things up for you. Much of Bash expertise is then about how to fight vigorously against the default behavior of silently chunking up your args. But fighting really strong means using e. g. arrays with ugly syntax, and other magic looking constructs. It shouldn't take so much alertness to just keep to sane behavior. The problem is, the interpreter is actively hostile to your endeavor and will explicitly try to screw with you.
simias 6y ago
Most of the pitfalls described in this doc are /bin/sh pitfalls, not bash per-se. On that note, let me rant: stop writing bash scripts people. If POSIX shell is not good enough, it's a good sign that you should move to a more expressive language with better error handling. Bash is a crappy compromise: it's not as portable as POSIX but it's also a very crappy scripting language. You might say I'm a pedant because "bash is everywhere nowadays anyway, your /bin/sh might even be bash in disguise" but I'll be the one having the last laugh the day you have to script something in some busybox or bash-less BSD environment. Knowing how to write portable shell scripts is a skill I value a lot personally, but that might be because I work a lot with embedded systems. Also if you assume that your target system will always have bash, there's a very good chance that it'll also have perl and/or python too. So no excuses.
- scipute68 6y ago
  This is basic nonsense. Design your scripts and provision your environments. _You_ can stop writing bash scripts. I find them eminently useful and I control/provision the environments they run in. If you are not in control then do what you are told. Most people would puke if I told them that a lot of csh is alive and well in sci-prod environments but that is the way it are. You _do_ tailor your environment bc that is the way *nix was designed. Controlling people and peoples habits is faang land.
LockAndLol 6y ago
I was half-expecting a page with the bash grammar. It's super easy to make mistakes in that language. I'm glad there are things like xonsh around trying to act as sane alternatives.
hi41 6y ago
What is the currently accepted best practice to write scripts? Is it to use Perl or Python?

288 points by aaron-santos 6y ago

mey 6y ago

https://web.archive.org/web/20200907092725/https://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20200907092725/https://mywiki.wo... Since the site is being hugged a little too much. I also really like https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck

scns 6y ago
Shellcheck and IdeaVim are the first plugins i install on Jetbrains IDEs
- pletnes 6y ago
  Shellcheck is, in general, one of the most undervalued dev tools hiding in the bushes. Especially as everyone has to shell at least a little bit, in particular linux newbies.
INTPenis 6y ago
Seeing this wiki hugged to death by HN makes me realize that it would be perfect for migrating to some simple cloud hosting and using static markdown in git. All their maintainers are already technical enough to handle doing updates in git. And with certain static doc generators like mkdocs you can have an Edit link in each page that brings the user to a Gitlab repo with an editor and Preview.
- pathseeker 6y ago
  Throwing everything at another company so you can blame your outages on them does not a better engineer make. This site is just doing something silly dynamically for something that should be static (e.g. reading from a database). A single modern computer can easily handle the HN hug of death serving static files.

ryangittins 6y ago

Looks like HN has hugged this one to death. https://web.archive.org/web/20200907174629/https://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20200907174629/https://mywiki.wo...

Ambol 6y ago

These type of things is why I always use a real programming language rather than shell scripting. Can't wrap my head around all the different pitfalls of treating random string outputs as input for other commands without first validating and parsing them into an actual data structure.

as-j 6y ago
Shell scripting has it’s place. In general it works fairly well and it’s an important skill to have. I used to have a report who had a similar train of thought, and ended up writing shell scripts in python which was way more fragile, unreadable and more complicated. The edge cases are there, but you don’t run into them all that often. Or if you do, you fix your scripts. Take the cp —- 2nd example. You generally don’t start file names with a -. It just makes Unix life annoying so you get pushed away from doing it. Keep scripts sane and straight forward, you can do really nice and powerful work knowing Unix tools. You can also shoot yourself in the foot some day when you create a file called ‘-rf’.
- 3pt14159 6y ago
  Bash is like a spec ops soldier's knife. Sure it could be used to kill someone (Bash on Balls, anyone?), but it's mostly for doing boring stuff like opening MREs (getting around a filesystem) or removing a deep thorn from the skin (removing a stupid "$n/a" string in a one-use TSV). I wouldn't send a navy seal into battle without one, but it isn't an absolutely critical piece of gear.
Spivak 6y ago
The cost you pay for this is verbose and unergonomic interop when your problem domain is executing a bunch of commands. For all its warts shells shine at being extremely expressive. It’s usually not worth the effort to shove everything into a real programming language but write a few self-contained commands designed to be called from a shell and then use the shell as the glue.
rantwasp 6y ago
yeah no. you should start with bash and drop into a proper programming language when complexity warrants it

rashkov 6y ago

Is there a greater principle behind all of these pitfalls, a mental model that I can learn and use to avoid these kinds of mistakes? Or is it sort of like CSS where years and years of ad-hoc development make it an exercise in memorizing edge-cases?

Spivak 6y ago
Some off the top of my head. * On Linux file and directory names can be anything except NUL. Because of how much software already assumes they’re UTF-8 you can probably safely assume it too and not run into much trouble. But since bash basically just does some text substitutions and then calls eval you have to be defensive. * Know how bash parses front and back. Almost all the pitfalls are unexpected consequences of the parser not behaving how you expect. https://web.archive.org/web/20200124085320/http://mywiki.wooledge.org/BashParser https://web.archive.org/web/20200124085320/http://mywiki.woo...
- mfontani 6y ago
  IIRC directory (and file) names can contain anything but slash and null, not just null.
- bewuethr 6y ago
  Pretty much this, and as a subset of it: quoting, word splitting, the IFS variable and friends. Reading the BashGuide [1] cover to cover is an excellent base, I find. [1]: https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide
- doctoboggan 6y ago
  Do you know if the `--` trick mentioned in the article to stop option scanning works with all bash programs? I thought parsing was up to the program itself. Is it just a widely used convention?
  - scbrg 6y ago
    You are correct, option parsing is up to each program. The double dash is just a convention.

deleted 6y ago

[deleted]

kennywinker 6y ago

Bash pitfall #1: deciding to use bash

jjgreen 6y ago
So true, bash is a fine interactive shell, but for scripts the Bourne shell is almost as capable, portable as in POSIX and did not suffer shellshock.

nikisweeting 6y ago

I consider shellcheck absolutely essential if you're writing even a single line of Bash. I also start all my scripts with this "unofficial bash strict mode" and DIR= shortcut: #!/usr/bin/env bash ### Bash Environment Setup # http://redsymbol.net/articles/unofficial-bash-strict-mode/ # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html # set -o xtrace set -o errexit set -o errtrace set -o nounset set -o pipefail IFS=$'\n' DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" I have more tips/tricks here: https://github.com/pirate/bash-utils/blob/master/util/base.sh https://github.com/pirate/bash-utils/blob/master/util/base.s...

rantwasp 6y ago
set -euxo pipefail
- ed25519FUUU 6y ago
  Every script. Good enough.
- Rapzid 6y ago
  Ahh, errexit: unless the command that fails is part of an until or while loop, part of an if statement, part of a && or || list, or if the command's return status is being inverted using !. All hope is lost.
rgrau 6y ago
You've a lot of cool stuff in that repo! I hope you don't mind me 'stealing' some stuff for a kind of booklet I'm working on with several shell tips and tricks. https://github.com/kidd/scripting-field-guide/ https://github.com/kidd/scripting-field-guide/ PS: Any comments/PRs are super welcome there too
- nikisweeting 6y ago
  No problem, it's all MIT licensed so it's not stealing haha. See my parent comment with my full reading list as well. If you want to link back to my repo as a cited source/credit that would be nice, but is not required :)

zests 6y ago

Bash is one of my favorite programming languages. Not for production. Writing significant production Bash that others have to use and maintain is more of a struggle than it is worth. But for personal use? Hitting APIs, organizing data, writing to files... Bash has been enjoyable because it requires much more creativity than almost any other language. At some point it's just you and the coreutils against the world.

bor100003 6y ago
I have the same feeling. In most cases you do the job without any external libs which is rarely the case with other programming languages.

phibz 6y ago

I wonder if there's a linter to check for these issues.

boogies 6y ago
Shellcheck? https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck

znpy 6y ago

It's both interesting to read and meh. It's meh because like many things, you actually have to read the fine manual if you want stuff to actually work. It's interesting because it's always nice to have a refresher and/or reminder of various bash things. That being said... I've said this many times, I'll say it again: bash scripts are software, and thus must not be exempted from various programming best practices. The first thing that comes to my mind is input validation and error checking. Bash is not magical. It's one of those things that have to be mastered through practice... Just like many other programming languages.

amirkdv 6y ago

Years ago I spent a lot of time (arguably too much time) writing (ba)sh scripts and this document was a great resource! But my main lesson from that period was that bash pitfalls are readability/maintenance nightmares, confounded by the fact that most people don't know about most of the pitfalls. To make matters worse, you can't know which pitfalls are or aren't known to the original author / future maintainer of the script in front of you. But I'm not sure to what extent the language warts are inevitable consequences of what makes it so expressive and powerful. For example: unix pipes are amazing and the bash syntax for them is quite expressive. I can reproduce the same behavior in any of the more "sane" languages but I find that the equivalent code is a less readable procedural soup of string munging, control flow, and error control.

zerofrancisco 6y ago

Excellent caveats and suggestions to do good bash. Thank you! :)

1vuio0pswjnm7 6y ago

"13. cat file | sed s/foo/bar/ > file You cannot ..." With sed, in some cases, you can. One of the differences between Linux-provided sed and BSD-provided sed is the -a option. FreeBSD manpage: "-a The files listed as parameters for the "w" functions are created (or truncated) before any processing begins, by default. The -a option causes sed to delay opening each file until a command containing the related "w" function is applied to a line of input." So let's say you have a file cat > file sandfoo ^D BSD-provided sed cat file|sed -na 's/foo/bar/;H;$!d;g;w'file The replacement is made without any temp file. Note this example will add a blank line at the top. Why use sed? sed is part of BSD base system, the toolchain^1 and install images thus it can easily be utilised early in boot process. I am not aware that "sponge" is part of any base system. 1. NetBSD's toolchain can be reliably cross-compiled on Linux.

koala_man 6y ago
An easier way to do the same thing but for any tool is `cat file | sed 's/foo/bar/' | sponge file`
- yjftsjthsd-h 6y ago
  Good old moreutils:)
samoa42 6y ago
what is wrong with the '-i' option? -i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if SUFFIX supplied)
- 1vuio0pswjnm7 6y ago
  sed -i creates a temp file. Needs additional storage space the size of the file. This can pose a problem with large files on smaller computers with limited storage and memory. In the past, some BSD seds (other than FreeBSD) did not have the -i option. Also worth noting that "sponge", unlike sed, apparently reads the entire file into memory. For a large file, this would require enough RAM to fit the entire file.

baby 6y ago

To me this screams: don't use bash in production.

jakeogh 6y ago
Yea. Or test the heck out of it. Once it works, it tends to work well. It took me ~8 years (so far, I'm sure there is a bug somewhere) and considerable #bash advice to make a simple script that locks commands+args https://github.com/jakeogh/commandlock https://github.com/jakeogh/commandlock I also wrote a MDA wrapper for gnupg, and again, it took years to feel good about having my mail filter through it https://github.com/jakeogh/gpgmda https://github.com/jakeogh/gpgmda Now that py has pathlib, if it's more than a few lines, py with @click is just way better. All that is really missing is cleaner copy/move abstractions. There are also some features that are missing from all shells... like the ability of a script to know if expansion happened before it got $@... that internal state just isn't exposed.

bluedays 6y ago

Oh neat. I found a problem with a script I wrote today because of this. I wrote a script recently that takes several screenshots from the same folder, merges them using Image Magick and outputs the merged image into a PDF. Found out through the first tip that I shouldn't be using ls: https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29 https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_....

asicsp 6y ago

See also: * BashFAQ - https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ * shellharden: safe ways to do things in bash - https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md https://github.com/anordal/shellharden/blob/master/how_to_do... * Security implications of forgetting to quote a variable in bash/POSIX shells - https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells https://unix.stackexchange.com/questions/171346/security-imp...

CaptainZapp 6y ago
Especially the first link is incredibly useful to me. Thanks a lot!

slavik81 6y ago

I was writing a script recently and I was going to use `cp -- "$src" "$dst"` but neither `man cp` nor `cp --help` mentioned `--` as an option. I'm not sure I'm using info right, but it doesn't seem to be mentioned in `info cp`, either. Is that feature described somewhere else, or is it just undocumented?

JdeBP 6y ago
The -- mechanism in general, not specific to the cp command albeit mostly limited to programs that use getopt/popt for their argument parsing, is described all over the place. I first encountered it in a book, written by Eric Foxley, entitled UNIX for Super-Users and published in 1985. * It's perennially asked about on Unix and Linux Stack Exchange, ranging from https://unix.stackexchange.com/q/11376/5132 https://unix.stackexchange.com/q/11376/5132 almost a decade ago to https://unix.stackexchange.com/q/570729/5132 https://unix.stackexchange.com/q/570729/5132 last year. * It's in a whole bunch of duplicate questions there, including (but not limited to) https://unix.stackexchange.com/questions/linked/1519?lq=1 https://unix.stackexchange.com/questions/linked/1519?lq=1 . * It's in the Single Unix Specification, and has been for a long time. (https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilconv.html#tag_009_002 https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilcon...) * It's in the getopt(3) manual page from the Linux man-pages project. It's in the getopt(1) manual page from the util-linux project. * It's in getopt(1) and getopt(3) in the NetBSD Manual. The same goes for the OpenBSD and FreeBSD manuals. * It's in Blum's and Bresnehan's 2015 Linux Command Line and Shell Scripting Bible, on page 377. * It's in Marcel Gagné's 2002 Linux System Administration: A User's Guide, on page 48. * It's in Kirk Waingrow's 1999 UNIX Hints & Hacks. * It was Q&A number 32 in part 2 of the Usenet comp.unix.shell FAQ document. (http://cfajohnson.com/shell/cus-faq-2.html#Q32 http://cfajohnson.com/shell/cus-faq-2.html#Q32) Specific to the cp command, or rather to the many cp commands that you could end up using, as this is not a "bash command": * No, the OpenBSD, FreeBSD, and NetBSD cp(1) manuals do not mention it. * Neither does the GNU CoreUtils cp(1) manual page, as you noted. It's instead in the GNU CoreUtils doco section on "common options". The actual CoreUtils cp(1) manual doesn't mention that there even are "common options". (https://manpages.debian.org/sid/coreutils/cp.1.en.html https://manpages.debian.org/sid/coreutils/cp.1.en.html) That's only mentioned, as an aside, in the info page. (https://gnu.org/software/coreutils/manual/html_node/cp-invocation.html#cp-invocation https://gnu.org/software/coreutils/manual/html_node/cp-invoc...) * For the cp command that is part of Rob Landley's toybox, this is covered in the introduction to the toybox(1) manual page, and there isn't really a distinct cp(1) manual page. (http://landley.net/toybox/help.html http://landley.net/toybox/help.html) * No, BusyBox doesn't similarly document this for its cp command, or anywhere on busybox(1). (https://busybox.net/downloads/BusyBox.html https://busybox.net/downloads/BusyBox.html) * It's tangentially in several GNU CoreUtils FAQ Q&As, inasmuch as CoreUtils cp is the same as CoreUtils rm in this regard. (https://gnu.org/software/coreutils/faq/coreutils-faq.html#How-do-I-remove-files-that-start-with-a-dash_003f https://gnu.org/software/coreutils/faq/coreutils-faq.html#Ho...) * It's tangentially in the Linux Documentation Project's Advanced Bash-Scripting Guide, again because cp is the same as rm here. (https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHREF https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHR...) * For an operating system where this is actually in the cp(1) manual itself, look to Solaris, and its successor Illumos, where you'll find it in the NOTES section. (https://illumos.org/man/1/cp https://illumos.org/man/1/cp) * Another operating system where this is actually in the cp(1) manual itself is AIX, actually in the FLAGS table alongside all of the other flags. (https://ibm.com/support/knowledgecenter/ssw_aix_72/c_commands/cp.html https://ibm.com/support/knowledgecenter/ssw_aix_72/c_command...)

chousuke 6y ago

One new pitfall I learned yesterday is that sourcing a file searches $PATH. I ran a tool that did ". config" on OpenBSD and it tried to source config(8) instead of its configuration in the local directory. Oops.

JdeBP 6y ago
On the subject of OpenBSD, sourcing configuration files, and security: Enjoy https://unix.stackexchange.com/a/433245/5132 https://unix.stackexchange.com/a/433245/5132 .

rurban 6y ago

I have almost no bash scripts. Whenever sh is not good enough and I have to switch to bash, I'd rather rewrite it in proper Perl. Bash really looks like a hack, and not all environments have bash. Everybody has a conforming sh.

inoffensivename 6y ago

It would be easier to list all the constructs in bash that are well-behaved and intuitive.

bonoboTP 6y ago

The biggest transformative learning experience for me in Unix command gluing was to understand argument vectors. That a command execution is not just one big string, but will at the point of calling the command, be transformed by Bash to a vector of strings, to be passed on to the execve system call. The point is, all those quotes etc. are syntax we use to tell the Bash I terpreter what the strings passed on should be. Think in terms of the string vector and not in terms of the one big string in front of your eyes that is a line of Bash code. The latter is a recipe for memorizing magic incantations and cargo cult sprinkling of quotes here and there. The horrible thing with Bash is that while normal programming languages make the difference crystal clear between the name of a function being called and its mandatorily quoted string args neatly separated with commas, Bash will come up with this split on the spot using arcane rules. Letting a string automatically fall apart into multiple argemts on the spot may have seemed convenient in the old days for forwarding invocations, but it seems clear to me that this was a design mistake. In the end if you want to have robust Bash code you have to keep track of string splitting and will almost never rely on the default behavior of letting Bash "helpfully" pre-digest and split things up for you. Much of Bash expertise is then about how to fight vigorously against the default behavior of silently chunking up your args. But fighting really strong means using e. g. arrays with ugly syntax, and other magic looking constructs. It shouldn't take so much alertness to just keep to sane behavior. The problem is, the interpreter is actively hostile to your endeavor and will explicitly try to screw with you.

simias 6y ago

Most of the pitfalls described in this doc are /bin/sh pitfalls, not bash per-se. On that note, let me rant: stop writing bash scripts people. If POSIX shell is not good enough, it's a good sign that you should move to a more expressive language with better error handling. Bash is a crappy compromise: it's not as portable as POSIX but it's also a very crappy scripting language. You might say I'm a pedant because "bash is everywhere nowadays anyway, your /bin/sh might even be bash in disguise" but I'll be the one having the last laugh the day you have to script something in some busybox or bash-less BSD environment. Knowing how to write portable shell scripts is a skill I value a lot personally, but that might be because I work a lot with embedded systems. Also if you assume that your target system will always have bash, there's a very good chance that it'll also have perl and/or python too. So no excuses.

scipute68 6y ago
This is basic nonsense. Design your scripts and provision your environments. _You_ can stop writing bash scripts. I find them eminently useful and I control/provision the environments they run in. If you are not in control then do what you are told. Most people would puke if I told them that a lot of csh is alive and well in sci-prod environments but that is the way it are. You _do_ tailor your environment bc that is the way *nix was designed. Controlling people and peoples habits is faang land.

LockAndLol 6y ago

I was half-expecting a page with the bash grammar. It's super easy to make mistakes in that language. I'm glad there are things like xonsh around trying to act as sane alternatives.

hi41 6y ago

What is the currently accepted best practice to write scripts? Is it to use Perl or Python?