6 ms·

Bash Pitfalls

114 points by monort 11y ago

joshbaptiste 11y ago
Everything I know of bash I learned from Freenode irc #bash channel who have a very active bot that always points to this wiki, so many scripts at work I see the dreaded for i in `ls`; do... Which only works due to the fact that we hardly have any files with spaces, newlines etc..
- frou_dh 11y ago
  There should be a modern filesystem that's catered to nerd tastes and simply prohibits whitespace in file names.
  - hyperpape 11y ago
    Whitespace really might be a bridge too far, but more modest restrictions have been proposed: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h...
  - JoshTriplett 11y ago
    Alternatively, allow everything in filenames, including whitespace and '/', but escape them on the filesystem using something like URL escaping (%20, %2F). No reason the filesystem names have to match the user-friendly names precisely, as long as a lossless bidirectional conversion exists.
    - frou_dh 11y ago
      Neat idea. If I understand correctly, it'd still be fair to say that those characters wouldn't be allowed in file names since encoding/decoding the friendly form would be opt-in work each and every userspace program couldn't be relied upon doing?
      - mitchty 11y ago
        Yep, the vfs layer in most unixes won't let you use / or \0 in a filename. And for good reasons.
jordigh 11y ago
And people think C++ is hard to do correctly...
- agumonkey 11y ago
  The world is masochist. I say paradise is too boring.
- pen2l 11y ago
  Sorry, what do you mean to imply here... that Bash is difficult?
  - kzhahou 11y ago
    Is it even up for debate, that bash is hard to get right?
    - pen2l 11y ago
      The funny thing is, I don't consider myself a very smart person or a programmer, but I've gotten bash down pretty well at this point. And I'm a person who has a looooot of trouble with C++ etc., most of my code is shit, if not just unusable. Though, I have been using linux command-line daily for 12 years now for sysadmining tasks and such. I know there's a lot of weird stuff to worry about, (especially as is noted in the 'bash pitfalls' guide), but I think with a little bit of practice (about 6 months' worth) and good instruction (I love greycat's guides), anyone can get a reasonable good handle of bash.
      - seanp2k2 11y ago
        Agree. Also, http://wiki.bash-hackers.org/doku.php http://wiki.bash-hackers.org/doku.php has tons of great info. Sad that TLDP gets all the Google ranking for most bash searches when wooledge and BHW typically have much more detailed and modern (e.g. using "${var}" and [[ instead of [ ) explanations. Lastly, if you're really stuck on a bash problem, head on over to #bash in Freenode and the awesome people there will very likely help! Greycat hangs out there.
monort 11y ago
If your bash script is not trivial, and target machines have python, it's probably better to use sh: https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
- iberieve 11y ago
  First time I've seen this and it looks amazing. Poorly named perhaps, though it is certainly a nice option to be aware of! Thank you for sharing. One question - In this py-sh system are there error control options like what bash provides? e.g.: set -o errexit set -o pipefail set -o nounset
  - monort 11y ago
    These options are set by default - non zero exit code or unknown variable will raise an exception. You can catch it if you don't want to exit.
- heavenlyhash 11y ago
  Love amoffat's sh. love. Love it so much I wrote similarly styled libraries for both - go: https://github.com/polydawn/gosh https://github.com/polydawn/gosh - java: https://github.com/polydawn/josh https://github.com/polydawn/josh I particularly like the golang one, because it results in smallish (~1mb) files that can crash land on any linux with no other bootstrap at all... which while troubling for replacing your entire library of shell scripts, is great for bootstrapping. (For replacing your entire library of shell scripts, I just use it with `go run` after my bootstrap has installed the go toolchain.)
- rando289 11y ago
  Similar ruby library. not as extensive since ruby has more shell-like features built in http://ruby-doc.org/stdlib-2.2.1/libdoc/shell/rdoc/Shell.html http://ruby-doc.org/stdlib-2.2.1/libdoc/shell/rdoc/Shell.htm...
freddref 11y ago
Is there a script to check for these pitfalls? (And offer example solutions?) Has anyone forked bash to remove or fix these pitfalls, while maintaining maximum "bashness"? edit: http://www.shellcheck.net/ http://www.shellcheck.net/
- e40 11y ago
  Has anyone forked bash to remove or fix these pitfalls, while maintaining maximum "bashness"? I'm not sure how bash could be changed to prevent user error. Many are not due to bad bash design.
vezzy-fnord 11y ago
See also the Inferno shell: http://debu.gs/entries/inferno-part-1-shell http://debu.gs/entries/inferno-part-1-shell I've been playing around with the werc framework, 9base, plan9port and other Plan 9-derived tooling and have found rc shell to be rather pleasant compared to Bourne and Korn dialects.
hyperpape 11y ago
Referencing the other thread on shell scripting currently on the front page (https://news.ycombinator.com/item?id=10068668 https://news.ycombinator.com/item?id=10068668), some of these examples show ways in which bash is actually quite verbose: # POSIX for i in *.mp3; do [ -e "$i" ] || continue some_command "$i" done # HYPOTHETICAL SYNTAX 1 map some_command *.mp3 # HYPOTHETICAL SYNTAX 2 some_command *.mp3 It's hard to imagine how to create simple syntax for operations like that while accomodating other syntactic requirements (strings without quotations, pipelines, etc), but I dream about a shell language that lets me do things like the above.
- rando289 11y ago
  The first command is distinguishing between broken links, and changing the -e to different letters, it can differentiate between 23 different attributes, and act differently depending on each. Your hypothetical examples do not do not have that ability. And, to do what your commands would imply, usually you really can do "some_command *.mp3", it's not hypothetical!
  - hyperpape 11y ago
    There are two points here. You say you can do "some_command *.mp3"...true if either a) your command is safe to run with garbage input or b) you know that you have >= one file and no whitespace in filenames. Both of these are often reasonable assumptions, but I want a general purpose solution that doesn't have more edge cases than it has to. The mental model of bash is "this text expands to this text like so, which means x haappens." The model I want is "I'm gonna do x to each of these files." As for the 23 commands, my attitude is that this is usually either ~19 too many, or infinitely too few. I acknowledge that maybe the way of doing things will start to make sense at some point, but it hasn't happened yet, and I find trying to climb the hill of understanding the shell to be a joyless experience.
    - a3n 11y ago
      > As for the 23 commands, my attitude is that this is usually either ~19 too many, or infinitely too few. As recently seen on HN: http://www.cs.yale.edu/homes/perlis-alan/quotes.html http://www.cs.yale.edu/homes/perlis-alan/quotes.html 11. If you have a procedure with ten parameters, you probably missed some.
rando289 11y ago
We've overwhelmed it. Instead: https://web.archive.org/web/20150811151807/http://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20150811151807/http://mywiki.woo...
proactivesvcs 11y ago
A really helpful guide, particularly for someone just starting out on Linux such as myself. Hopefully I will not get into bad habits to begin with :-) Having looked at my scripts I seem to have been pretty cautious already (Windows batch has already scarred me plenty), but I have shored up a few minor areas.
asa400 11y ago
Li Haoyi just gave a talk at Scala by the Bay entitled "Beyond Bash", which details the shell scripting environment he's been working on that is hosted in a reimplementation of the Scala REPL, called Ammonite. Slides for the talk: http://tinyurl.com/beyondbash http://tinyurl.com/beyondbash Docs: http://lihaoyi.github.io/Ammonite/ http://lihaoyi.github.io/Ammonite/ I wasn't at the talk, but I downloaded it and have been playing around with it. It's really fun! The path operations are all typed (ie, you can't combine relative and absolute paths in stupid ways), you get all of Scala to operate on files and the filesystem (if you know Scala, this is pretty huge), and it has a handy pipelining syntax that is effectively an extension of the shell `|` operator we all know and love: http://lihaoyi.github.io/Ammonite/#Extensions http://lihaoyi.github.io/Ammonite/#Extensions There are other niceties built in as well, like syntax highlighting and pretty printing, that gave me the impression that the author really cares about the UX of the software. It's not all academic/pure, in fact it appears to be the kind of pragmatic, practical thing that I wish Scala was known for. I highly recommend giving it a shot, especially if you already know Scala. I definitely will be giving it some time in the coming weeks.

114 points by monort 11y ago

Everything I know of bash I learned from Freenode irc #bash channel who have a very active bot that always points to this wiki, so many scripts at work I see the dreaded for i in `ls`; do... Which only works due to the fact that we hardly have any files with spaces, newlines etc..

frou_dh 11y ago
There should be a modern filesystem that's catered to nerd tastes and simply prohibits whitespace in file names.
- hyperpape 11y ago
  Whitespace really might be a bridge too far, but more modest restrictions have been proposed: http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h...
- JoshTriplett 11y ago
  Alternatively, allow everything in filenames, including whitespace and '/', but escape them on the filesystem using something like URL escaping (%20, %2F). No reason the filesystem names have to match the user-friendly names precisely, as long as a lossless bidirectional conversion exists.
  - frou_dh 11y ago
    Neat idea. If I understand correctly, it'd still be fair to say that those characters wouldn't be allowed in file names since encoding/decoding the friendly form would be opt-in work each and every userspace program couldn't be relied upon doing?
    - mitchty 11y ago
      Yep, the vfs layer in most unixes won't let you use / or \0 in a filename. And for good reasons.

jordigh 11y ago

And people think C++ is hard to do correctly...

agumonkey 11y ago
The world is masochist. I say paradise is too boring.
pen2l 11y ago
Sorry, what do you mean to imply here... that Bash is difficult?
- kzhahou 11y ago
  Is it even up for debate, that bash is hard to get right?
  - pen2l 11y ago
    The funny thing is, I don't consider myself a very smart person or a programmer, but I've gotten bash down pretty well at this point. And I'm a person who has a looooot of trouble with C++ etc., most of my code is shit, if not just unusable. Though, I have been using linux command-line daily for 12 years now for sysadmining tasks and such. I know there's a lot of weird stuff to worry about, (especially as is noted in the 'bash pitfalls' guide), but I think with a little bit of practice (about 6 months' worth) and good instruction (I love greycat's guides), anyone can get a reasonable good handle of bash.
    - seanp2k2 11y ago
      Agree. Also, http://wiki.bash-hackers.org/doku.php http://wiki.bash-hackers.org/doku.php has tons of great info. Sad that TLDP gets all the Google ranking for most bash searches when wooledge and BHW typically have much more detailed and modern (e.g. using "${var}" and [[ instead of [ ) explanations. Lastly, if you're really stuck on a bash problem, head on over to #bash in Freenode and the awesome people there will very likely help! Greycat hangs out there.

monort 11y ago

If your bash script is not trivial, and target machines have python, it's probably better to use sh: https://amoffat.github.io/sh/ https://amoffat.github.io/sh/

iberieve 11y ago
First time I've seen this and it looks amazing. Poorly named perhaps, though it is certainly a nice option to be aware of! Thank you for sharing. One question - In this py-sh system are there error control options like what bash provides? e.g.: set -o errexit set -o pipefail set -o nounset
- monort 11y ago
  These options are set by default - non zero exit code or unknown variable will raise an exception. You can catch it if you don't want to exit.
heavenlyhash 11y ago
Love amoffat's sh. love. Love it so much I wrote similarly styled libraries for both - go: https://github.com/polydawn/gosh https://github.com/polydawn/gosh - java: https://github.com/polydawn/josh https://github.com/polydawn/josh I particularly like the golang one, because it results in smallish (~1mb) files that can crash land on any linux with no other bootstrap at all... which while troubling for replacing your entire library of shell scripts, is great for bootstrapping. (For replacing your entire library of shell scripts, I just use it with `go run` after my bootstrap has installed the go toolchain.)
rando289 11y ago
Similar ruby library. not as extensive since ruby has more shell-like features built in http://ruby-doc.org/stdlib-2.2.1/libdoc/shell/rdoc/Shell.html http://ruby-doc.org/stdlib-2.2.1/libdoc/shell/rdoc/Shell.htm...

freddref 11y ago

Is there a script to check for these pitfalls? (And offer example solutions?) Has anyone forked bash to remove or fix these pitfalls, while maintaining maximum "bashness"? edit: http://www.shellcheck.net/ http://www.shellcheck.net/

e40 11y ago
Has anyone forked bash to remove or fix these pitfalls, while maintaining maximum "bashness"? I'm not sure how bash could be changed to prevent user error. Many are not due to bad bash design.

vezzy-fnord 11y ago

See also the Inferno shell: http://debu.gs/entries/inferno-part-1-shell http://debu.gs/entries/inferno-part-1-shell I've been playing around with the werc framework, 9base, plan9port and other Plan 9-derived tooling and have found rc shell to be rather pleasant compared to Bourne and Korn dialects.

hyperpape 11y ago

Referencing the other thread on shell scripting currently on the front page (https://news.ycombinator.com/item?id=10068668 https://news.ycombinator.com/item?id=10068668), some of these examples show ways in which bash is actually quite verbose: # POSIX for i in *.mp3; do [ -e "$i" ] || continue some_command "$i" done # HYPOTHETICAL SYNTAX 1 map some_command *.mp3 # HYPOTHETICAL SYNTAX 2 some_command *.mp3 It's hard to imagine how to create simple syntax for operations like that while accomodating other syntactic requirements (strings without quotations, pipelines, etc), but I dream about a shell language that lets me do things like the above.

rando289 11y ago
The first command is distinguishing between broken links, and changing the -e to different letters, it can differentiate between 23 different attributes, and act differently depending on each. Your hypothetical examples do not do not have that ability. And, to do what your commands would imply, usually you really can do "some_command *.mp3", it's not hypothetical!
- hyperpape 11y ago
  There are two points here. You say you can do "some_command *.mp3"...true if either a) your command is safe to run with garbage input or b) you know that you have >= one file and no whitespace in filenames. Both of these are often reasonable assumptions, but I want a general purpose solution that doesn't have more edge cases than it has to. The mental model of bash is "this text expands to this text like so, which means x haappens." The model I want is "I'm gonna do x to each of these files." As for the 23 commands, my attitude is that this is usually either ~19 too many, or infinitely too few. I acknowledge that maybe the way of doing things will start to make sense at some point, but it hasn't happened yet, and I find trying to climb the hill of understanding the shell to be a joyless experience.
  - a3n 11y ago
    > As for the 23 commands, my attitude is that this is usually either ~19 too many, or infinitely too few. As recently seen on HN: http://www.cs.yale.edu/homes/perlis-alan/quotes.html http://www.cs.yale.edu/homes/perlis-alan/quotes.html 11. If you have a procedure with ten parameters, you probably missed some.

rando289 11y ago

We've overwhelmed it. Instead: https://web.archive.org/web/20150811151807/http://mywiki.wooledge.org/BashPitfalls https://web.archive.org/web/20150811151807/http://mywiki.woo...

proactivesvcs 11y ago

A really helpful guide, particularly for someone just starting out on Linux such as myself. Hopefully I will not get into bad habits to begin with :-) Having looked at my scripts I seem to have been pretty cautious already (Windows batch has already scarred me plenty), but I have shored up a few minor areas.

asa400 11y ago

Li Haoyi just gave a talk at Scala by the Bay entitled "Beyond Bash", which details the shell scripting environment he's been working on that is hosted in a reimplementation of the Scala REPL, called Ammonite. Slides for the talk: http://tinyurl.com/beyondbash http://tinyurl.com/beyondbash Docs: http://lihaoyi.github.io/Ammonite/ http://lihaoyi.github.io/Ammonite/ I wasn't at the talk, but I downloaded it and have been playing around with it. It's really fun! The path operations are all typed (ie, you can't combine relative and absolute paths in stupid ways), you get all of Scala to operate on files and the filesystem (if you know Scala, this is pretty huge), and it has a handy pipelining syntax that is effectively an extension of the shell `|` operator we all know and love: http://lihaoyi.github.io/Ammonite/#Extensions http://lihaoyi.github.io/Ammonite/#Extensions There are other niceties built in as well, like syntax highlighting and pretty printing, that gave me the impression that the author really cares about the UX of the software. It's not all academic/pure, in fact it appears to be the kind of pragmatic, practical thing that I wish Scala was known for. I highly recommend giving it a shot, especially if you already know Scala. I definitely will be giving it some time in the coming weeks.