17 ms·
Safe ways to do things in bash
- nisa 9y agoIf you reached a point where you need to require bash and not a posix shell and need to enforce these rules just use python or lua if you can or some scheme or whatever else... it's not worth the wasted time hunting bash cruft. if you are on busybox with ash none of this is helping (except shellcheck which is great).
- ausjke 9y agoI still use bash a lot though, it's just handy, with lua and python you will have to deal with various of its packages sooner or later, for bash it is just one binary and you get 90% scripting tasks covered, zero dependencies.
- nisa 9y agoif you need arrays use mksh - https://www.mirbsd.org/mksh.htm https://www.mirbsd.org/mksh.htm - if you need complex scripting with small footprint and standard compliance avoid bash and use mksh.
- JoshTriplett 9y agoYou can count on bash being present on almost any Linux system, other than tiny embedded ones. You can't count on mksh being anywhere unless you install it. Doesn't matter if it's better or worse, it isn't omnipresent.
- ausjke 9y agoactually I do lots of embedded, busybox is about 2MB and bash is 1MB, unless the resource is extremely restricted I just install bash for scripting purposes. python is about 4.5MB and Lua is about 200K, both run on top of a shell anyways, and need some packages to be fully useful. To be fair adding 200KB Lua (no luarocks etc) on top of a shell is useful sometimes on embedded system, but I rarely need that.
- groovy2shoes 9y agoIn addition to bash itself being widespread, most of the common "bashisms" are also supported by zsh and most variants of ksh (and many systems actually use some variant of ksh disguised as /bin/sh). This is unsurprising given that bash itself was originally meant to be a clone of ksh. The places where they differ tend to cluster around uncommon/newer features (like coprocesses), certain syntactic quirks (but if you're aware of them they're usually easy to work around), and interactive features (which are mostly irrelevant for scripts). If you're writing simple scripts (or you're writing complex ones in a careful way), you can reasonably "target" bash and expect them to work on other common shells. If you're going to be distributing shell scripts, it's probably a good idea to test them on a handful of common shells anyway. If portability between shells isn't something you're worried about, it's exactly as reasonable to ask users to install bash as it is to ask them to install mksh or zsh (or even python or lua or whatever) anyway. If you're most comfortable with bash, just use bash! But for the love of god and all that is holy, please don't write csh scripts ;)
- mixedCase 9y agoSometimes you just want some 30-50 lines of piping a few commands and a couple of conditionals. Python (the language with the most community traction to replace bash for scripts) is a royal pain to use for this without libraries that are present in no default system, and even with those it often ends up being more verbose than it should.
- nisa 9y ago> Sometimes you just want some 30-50 lines of piping a few commands and a couple of conditionals. maybe I was not clear - nothing against shell-scripting but doing some weird dancing like in the article is imho useless, also depending on bash is a stupid idea imho. posix sh + shellcheck is all you need. if you can't solve your problem in posix sh rethink your code / approach and simplify until it will work.
- deleted 9y ago[deleted]
- thisacctforreal 9y agoI agree. I've used this page successfully as reference for portable syntax: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3... Some sections on features I use heavily: - Parameter Expansion (specifically :-, %, %%, #, ##) - Special Parameters (specifically "$@", $#, $?) - set --, this lets you set the $1, $2, etc. variables. I use this with "$@" for arrays, primarily for building command strings. Here's an small shell script demonstrating some of them: #!/bin/sh # err function err() { echo "$1" >&2; exit 1; } # init variables unset src unset dst dry_run=false # get arguments while [ $# -gt 0 ]; do case "$1" in -d|--dry-run) dry_run=true ;; --) shift; break ;; -*) err "unknown option: $1" ;; *) if [ -z "$src" ]; then src="$1" elif [ -z "$dst" ]; then dst="$1" else err "unexpected argument: $1" fi ;; esac shift done # sanity checks ## TODO: print usage if [ -z "$src" ]; then err "source not specified"; fi if [ ! -d "$src" ]; then err "source does not exist"; fi if [ ! -r "$src" ]; then err "cannot read source directory"; fi if [ -z "$dst" ]; then err "destination not specified"; fi if ! rsync --version >/dev/null 2>&1; then err "missing rsync(1)" fi # build rsync command set -- rsync -aq "$src" "$dst" # log command echo "copying $src to $dst" echo " $@" if ! $dry_run; then if "$@"; then echo "success" else # rsync will have printed an error message err "rsync exited with error code $?" fi else echo "dry run; not executing" fi Also be sure to read the man page for test (the [ command).
- nunez 9y agoBash is available on almost every system. The portability is desirable sometimes.
- h1d 9y agoHow many people need that portability? Most of the shell scripts are quick work that needs to run on small amount of environments.
- nodesocket 9y agoHighly recommend shellcheck[1]. There is a SublimeLinter plugin[2] that automatically checks your shell scripts as you code them. It generally makes best practice suggestions including quoting. [1] https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck [2] https://github.com/SublimeLinter/SublimeLinter-shellcheck https://github.com/SublimeLinter/SublimeLinter-shellcheck
- aequitas 9y agoNot only does it make suggestions, almost all of the 'error codes' have extensive documentation on why something is wrong and often multiple alternative solutions for each use case (eg: https://github.com/koalaman/shellcheck/wiki/SC2086 https://github.com/koalaman/shellcheck/wiki/SC2086). I learned more bash from Shellcheck than all tutorials and references combined.
- freedomben 9y agoSame. I was a bit cocky when I first tried out shell check because I had been doing bash for years. Shell check flagged something I'd been using for a while and after reading the docs I realized shell check's suggestio was much cleaner and just-as-safe way to do what I was doing. Really impressive piece of software.
- __blockcipher__ 9y agoSeconding this. Shellcheck is fucking amazing and whenever I visit the wiki page for a specific issue it's extremely descriptive and helps you understand why the flagged behavior is a problem. Some of the tips are obscure stuff that I never would have realized, like using `command -v` instead of `which` in my (arch) linux install script because command -v is more portable. And then you have the classic stuff like printf instead of echo -e, using pritnf string formatting, writing `\\n` instead of `\n`, double quoting variables etc
- jwilk 9y agoI'm all for "command -v", but do you know which OSes don't have "which" by default?
- jwilk 9y ago> POSIX mandates /bin/sh Nope. On the contrary, it says: Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh Source: http://pubs.opengroup.org/onlinepubs/009695399/utilities/sh.html http://pubs.opengroup.org/onlinepubs/009695399/utilities/sh.... A pedantically-compliant shell script should not have shebang at all.
- Twisol 9y agoWhat's wrong with `#!/usr/bin/env sh`?
- dwheeler 9y agoPOSIX doesn't include shebang (#!) at all. It should, but it currently doesn't.
- oblio 9y ago> currently doesn’t Yeah, we should be gentle with POSIX, it’s only 30 years young :)
- xyzzyz 9y agoNewest revision is from 2017, so they certainly had time to fix that, if they consider it a problem.
- jwilk 9y agoIt doesn't include /usr/bin/ either. The "env" utility could be somewhere else.
- _kst_ 9y agoI've worked on systems where env was installed as /bin/env and not as /usr/bin/env . (I think it was SunOS 4.) For that matter, under Termux on Android it's /data/data/com.termux/files/usr/bin/env (but termux has a hack to make normal shebangs work).
- IshKebab 9y agoFrom the author of "Russian roulette: how to probably not die", "Staying healthy with fast food" and "Self-immolation for dummies". If you need safety don't use Bash.
- waynecochran 9y agoAmen. Writing bash scripts (I have been programming in Unix since 1985) is an unnatural, fragile, and error prone endeavor. If is more than 5 lines long, I use Perl (which still far from ideal). When an article is almost solely about what not to do, that tells you something.
- koolba 9y agoFrom the article: > Should I use curly braces? Bad: some_command $arg1 $arg2 $arg3 Extra bad (cargo culting unnecessary braces): some_command ${arg1} ${arg2} ${arg3} Correct: some_command "${arg1}" "${arg2}" "${arg3}" Better: some_command "$arg1" "$arg2" "$arg3" > In the "extra bad" and "correct" examples, braces compete with quotes under the limits of tolerable verbosity. > Shellharden will rewrite all these variants into the "better" form. I prefer the "${bracey}" form for all variable usage. Yes it's marginally more verbose but it has the advantage of being consistent, easier on the eyes due to less overall quoting when part of full string interpolation[1], and cleaner diffs[2] as converting "${foo}" to "${foo}-bar" only leads to word-diff of "-bar". [1]: "${foo} bar baz" v.s. "$foo"" bar baz" [2]: You are source controlling your shell scripts right?
- saagarjha 9y ago> You are source controlling your shell scripts right? My personal heuristic for shell scripts are that if I care enough about them to put them under source control, they shouldn't be a shell script.
- mehrdadn 9y agoDo Makefiles fall under that category?
- stochastic_monk 9y agoChiming in for myself, no. Makefiles are more about dependency graphs and reproducible pipelines. I use them to make more sophisticated things than bash scripts, which are purely imperative. I very much dislike cmake, and I've not yet been impressed by any of the other many Make replacement candidates. How else would you accomplish what a Makefile would?
- GlennS 9y agoI think for C compilation, Tup is probably better than make. It also has a couple of small syntax improvements compared to make. In particular, it's much better with commands that loop over targets and dependencies. However, as someone who doesn't write much C, I went back to make. Makefiles seem to adapt really well to different workflows. A lot of my makefiles look like "download this publically available data, scp this secret data off a company server, mungle all the data with some scripts, shallow clone this branch of that repository, run the program in the repository a few times under different conditions, make a chart of the results". Tup really doesn't like steps like "clone this repository". Make isn't exactly great with keeping track of targets that output entire directories, but it does work. I also feel like, for me, make has become more powerful over time as I've learned other tools. Specifically: being able to just magic up any old environment using Nix and be confident that it won't break, mashing data with things like jq, cut, sed and awk, and quick database operations with SQLite.
- mehrdadn 9y agoA little heads-up: "$var" does what you think, but "$(cmd)" likely does not do what you think: - The former just gives you a string whose contents are identical to that of var. - The latter would do similar for the output of cmd, except that it strips away the trailing newline. This is often not an issue, but can be crucially important in some cases, and can catch you off-guard. The point I'm making here is that it's actually quite difficult to get a string that literally has the contents you want. The fact that *nix lets you put pretty much any characters in file names (even newlines) means that, just like in Windows, your scripts can actually fail even when you try to quote things "properly".
- dozzie 9y ago> The point I'm making here is that it's actually quite difficult to get a string that literally has the contents you want. It's not difficult, it's just tedious. foo=$(whatever). foo=${foo%.}
- mehrdadn 9y agoThat's not general POSIX, right? I seem to recall it's Bash-specific? (P.S. I think you forgot quotes?) The other problem (which I guess I accidentally brushed under the rug when I singled out "variables") is that having to do this actually means you need to put it in a variable. If you're nesting subshells, this gets pretty darn tedious, easy to forget about, and difficult to read pretty quickly... it seems you wouldn't consider that "difficulty" and think of it as "just tediousness", but I think if something is too easy to do incorrectly and too tedious to get right, that's also a kind of added difficulty.
- dozzie 9y ago> That's not general POSIX, right? I seem to recall it's Bash-specific? (P.S. I think you forgot quotes?) Wrong. (And no, I did not.) POSIX, or rather SUS (I've never had an access to POSIX), mandates ${foo%...} syntax and its three cousins. And assignment is not subject to word splitting for variable expansion.
- thomasjames 9y agoBackticks are very error prone $(cat foo.txt) is much more explicit and visually clearer for command substitution.
- schizoidboy 9y agoAnd this way also allows nesting.
- deleted 9y ago[deleted]
- h1d 9y agofish forces you to use braces. (Even without the dollar sign.)
- Aelius 9y agoInteresting stuff, but given bash's - Relative unportability - Poor noncompliant sh interpreter - Poor performance (can be 4x slower than POSIX sh shells like dash for certain tasks) I personally think bash is always the wrong choice. Use POSIX sh (or a real language). POSIX sh is 98% the same thing, there's no good reason to even use bash over sh in the vast majority of cases. It's just this blight that won't go away. Given that bash is mostly sh anyway, most of this writeup applies to sh, too. AFAIK the only thing bash-specific here is arrays.
- raggi 9y ago[[ alone is reason enough to prefer bash over sh on teams where you inevitably have non-experts making changes. Safe Bash is hard to teach, posix is far harder.
- rascul 9y agoI stick with bash to avoid implementation differences due to POSIX ambiguity [0]. I know that all my bash scripts will run on at least bash-4.0 but I have no idea what shell /bin/sh is going to be for any given system. I haven't written a single shell script where the performance difference matters, nor do I expect to. And there are a few nice bashisms besides arrays. I'm not sure what portability issues you're referring to. [0] https://stackoverflow.com/a/16376043 https://stackoverflow.com/a/16376043
- frou_dh 9y agoI agree with you, but there's one fly in the ointment. POSIX Shell does not support local variables! Writing functions without being able to have local variables? That's pretty damn gross. So I use the `local` keyword, and now my scripts aren't strictly POSIX Shell any more.
- skywhopper 9y agoMost of this is fine, but not all. In particular, command failure conditions are a huge source of bugs, but `set -euo pipefail` is not going to solve all your problems. `set -e` is just as likely to cause problems because it can cause scripts to silently fail. And pipefail can pass through errors that aren't relevant. These are great tools to have, but don't blindly invoke them as magic spells if you don't actually understand the implications.
- ta4354546444 9y agoBash strikes me as a bit of a mess, as in people threw the kitchen sink into it for 'portability'. Things like being able to open a socket e.g. using the /dev/tcp/<ip>/<port> stuff give me the willies a bit. I lean towards Ruby if it's going to be anything longer than a few lines, or requires anything but the simplest of logic/commands. Otherwise I was always told /bin/sh is likely to be the most portable, so tend to use that in the absence of any other good reason.
- BenjiWiebe 9y agoI've written two network/telnet game-bots in Bash. It's terribly hacky but fun in it's own way. Using the dev tcp trick and arrays and lots of bashisms.
- jonnycomputer 9y agoif it needs to be safe, use python. please don't down vote me
- tootie 9y agoSafe bash is what Perl was invented for. Don't downvote me.
- jonnycomputer 9y agosomeday i might even add it to my toolkit
- tannhaeuser 9y agoI think Python is made for other tasks than shell. Shell programs mainly consist of invocations for external command-line utilities that you can develop and generalize from interactive shell sessions and one-off automation scripts. Python OTOH is a general-purpose programming language.
- jonnycomputer 9y agoYes, Python is a general purpose language; however it has powerful libraries to safely interact with command line utilities. The main drawback is their verbosity. But you get a lot in the exchange. For small scripts of a few hundred lines, bash (or some equivalent) is the way to go. But anything more than that, you need something like python or perl (in my opinion, of course)
- ljm 9y ago> Gotcha: Errexit is ignored depending on caller context It proves the point that it's a gotcha but those examples seemed sensible to me. As far as I understand it, `set -e` doesn't turn every unchecked, non-zero exit code into an exception because there's no way of knowing whether the function or sub shell you're invoking was written by you or pulled in elsewhere, and as a result you don't know if a non-zero exit code is a legitimate, show-stopping failure. Those functions and subshells might as well be mini inline executables and in that context it makes sense to only check the final output. If that's horribly wrong and confusing, maybe there should be a move to make `set -e` the default so all error handling is explicit, but you've got other languages for that that don't involve throwing `|| true` onto the end of every unimportant command you run. I also realise that this doesn't make a case for Bash being intuitive. Precisely the opposite. But I suppose you have that with a shell where it's more important to be adaptive to the person behind the keyboard at the expense of the purity of the implementation. Especially considering the history of it all.
- kazinator 9y ago> Quoting inhibits both word splitting and wildcard expansion, for variables and command substitutions. The result of a variable substitution isn't subject to wilcard expansion, whether quoted or not. If your only reason to quote "$foo" is because you think foo expands to a globbing pattern, and no other reason is justified, you can drop the quotes.
- koala_man 9y agoIt is subject to wildcard expansion. You can verify this with var="/*"; echo $var
- Hello71 9y agowhat the balls are you talking about? $ touch bad; a='*'; echo $a bad I tested with shbot and this is the case in at least original bourne sh, ksh, mksh, dash, bash 1, and bash 4.4.
- kazinator 9y agoIndeed; see https://news.ycombinator.com/item?id=17070872 https://news.ycombinator.com/item?id=17070872 I'd erase my hasty comment if I could. I'm just praying it gets downvoted to invisibility now.
- chubot 9y agoI don't agree with the advice to use arrays and set -u together. Sometimes you just need to get something done, and being pedantic works against you. This advice only works in bash 4.4, and many common distros are on bash 4.3, like Ubuntu 16.02 LTS. (bash 4.4 was released September 2016.) Because of this bug, the section "how to begin a bash script" is version-specific and awkward IMO. If you need to process untrusted filenames, use arrays, but otherwise it's probably more trouble than it's worth. [1] I want my scripts to work on older versions of bash, and I think 'set -u' is important, so I mostly get by without arrays. It's not ideal, but shell is full of compromises. A workaround is to use ${a[@]+"${a[@]}"}, which avoids the bug in bash 4.3, but that seems too ugly to recommend. More details in this comment I wrote on the same article: https://lobste.rs/s/4jegyk/how_do_things_safely_bash#c_kmldw6 https://lobste.rs/s/4jegyk/how_do_things_safely_bash#c_kmldw... [1] Thirteen Incorrect Ways and Two Awkward Ways to Use Arrays http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html
- nunez 9y agoShellcheck made me a MUCH better Bash developer. Also, I prefer using [ condition ] for tests instead of the less-portable [[ cond ]] syntax despite the latter being more feature-rich. Didn’t see that one in there.
- nerdponx 9y agoStep 1 to write safer shell scripts: use a safer shell. Zsh gives the user much more control, has safer defaults, and is itself quite portable (even if Zsh scripts are not portable to other shells).
- Hello71 9y agounfortunately, zsh is installed on probably about 1% of Linux systems worldwide. maybe it can be installed almost anywhere, but the fact is it isn't, and you might as well use Python or something at that point.
- h1d 9y agoAnd how many on the system that you actually touch? How hard is it to (ask to) install a new shell? Obviously if you're distributing your script publicly no one would write that in zsh compatible way.
- BeetleB 9y agoI gave up long ago on trying to learn shell scripting and remembering it. Switching to xonsh for my shell made shell scripting easier. http://xon.sh/ http://xon.sh/
- p3llin0r3 9y agoRecently I've replaced most of my bash scripting with the python library invoke. It's written by the same people who wrote the python ssh scripting framework fabric. http://www.pyinvoke.org/ http://www.pyinvoke.org/ Works great!!
- TheGrassyKnoll 9y agoI'll have to try that one. I've done a few things with: Python process launching http://amoffat.github.com/sh http://amoffat.github.com/sh
- p3llin0r3 9y agoI worked with `sh` for a while. I strongly prefer invoke. It handles signal processing and TTY flawlessly. You can even get it to ssh into remote terminals and open a console and it works. It's also a CLI builder where you can chain commands together
- theamk 9y agoBe careful with sh library! It runs the programs under tty by default, so you get random effects like ascii color sequences and truncated git output. There is an option to change this, but it not default, and it is easy to forget it.
- codedokode 9y agoHow does one properly quote an argument for a command that can contain spaces? For example: parent_dir = "$(basename $dir)" How to quote $dir here? What if it contains spaces or other special characters? That's what I dislike about Bash. Also, always start your scripts with set -e This prevents script from running after error, without any messages though. Also, I always make mistakes when using [ and [[.
- zeroimpl 9y agoYou just add quotes around $dir. Bash understands the quotes properly/recursively. parent_dir = "$(basename "$dir")"
- codedokode 9y agoSo I finally found the answer. Thanks.
- 8xde0wcNwpslOw 9y agoBear in mind (pointed by the article as well) that variable assignment doesn't allow spaces around the equals sign, and that in this case the outer parentheses are unnecessary anyway.
- rossy 9y agoYou quote it like this: parent_dir="$(basename "$dir")" Quotes are syntactic in bash, so it understands that the outer pair of quotes are surrounding the command substitution and the inner pair are inside it. This will work regardless of any spaces or special characters in $dir.
- deleted 9y ago[deleted]
- koala_man 9y ago$(..) is its own context so you just put double quotes around any variables like you normally would: parent_dir="$(basename "$dir")" [ and [[ can indeed be tricky. You may find ShellCheck useful, since it recognizes common problems like [ a=b ], [-e .bashrc], [ 1 > 2 ], [ -n var ], [ false ] and several others.
- floatingatoll 9y agoThe advice to double-quote everything is an interesting way to circumvent "detailed knowledge may be required". I wish it noted that you can't quote regular expressions, though: $ cat - > foo #!/bin/bash a="BCD" [[ $a =~ .C. ]] && echo 1 [[ $a =~ ".C." ]] && echo 2 ^D $ bash foo 1 $
- Hello71 9y agothis is intentional, since you can store your regular expression in a variable: $ a=BCD; b=.C.; [[ $a =~ $b ]]; echo $?; [[ $a =~ "$b" ]]; echo $? otherwise, interpolating variables in regular expressions as text would require other syntax (more confusing). also, "cat -" is redundant, use "cat". this behavior is specified by POSIX.
- floatingatoll 9y agoI use “cat -“ so that my code makes more sense to people. I want STDIN declared somehow, and the dash is effective. Technically I shouldn’t have bothered with the cat at all in an HN code snippet, but it was a courtesy to provide a familiar environment for the block I wanted to convey. It worked so well that you linted it! I really appreciate the thought.
- shmerl 9y agoif [ ${#array[@]} -gt 0 ]; then a better way to write it: if (( ${#array[@]} > 0 )); then
- freedomben 9y agoI've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors and needle-in-hay-stack searches using only shell commands. 3. Shell is the universal language. 4. Bash isn't that bad once you get used to it (seriously. I'll grant you tho that arrays are still nasty ;-) ). 5. Bash is not that dangerous if you follow best practices. Don't be lazy! 6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing.
- digitalsushi 9y agoRe 6: I am an /extremely/ mediocre developer but I have crafted some real 99th percentile bash skills (within my company, not globally) and I completely self-sustain myself on just that. (You're wondering how many people write bash where I am, and the answer is somewhere between 8 and 15 thousand people). It's funny how many things that seem kind of incredible for a pure bash solution, looking back over the past 21 years doing this drivel. This year is the first I had to learn how to export all of the shell's variables and function definitions for a forked clone to run asynchronous background processing of the main thread. Bash! Bash of all things. Probably 10 seconds in any other language, but here is bash kind of hopping alongside. Callbacks, reflective programming, there's always a really awkward way in bash to do what's happening in the popular languages.
- koolba 9y ago> It's funny how many things that seem kind of incredible for a pure bash solution, ... My go to example of “Bash can do whaaat?!” is the source for xip.io. A custom DNS server written in a handful of lines of Bash! https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdns https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdn...
- 9y ago
- theamk 9y agoIf you get to the point when you need to write IFS=$'\v' read -d '' -ra a < <(printf '%s\v' "$s") it is a good sign that it's time to switch to some other language. For example in python, it will be just a = s.split("\v") (and yes, sometime you have legacy system, or writing initrd script. But how often does this happen? Any why does your initrd has full bash anyway, as opposed to dash or sh?)
- devhead 9y agonice write up, starred it for future hand outs to my friends. I do find that once my bash script is going over a hundred lines or so it's likely a good time to move to python. I love bash, it's great; at some point setting up a proper script (in bash) with arguments, options, logging and or validation you end up spending more time getting it to work than you do on solving the actual problem; enter your favorite programming language here.
- Xcelerate 9y agoI always think — when a programming/scripting language requires this much bizarre knowledge just to write basic code that performs basic tasks, perhaps it is time for that language to be retired. I really don't understand why bash still exists. I've switched to fish and am much happier with the change.
- tokyodude 9y agoI agree with you 100% It basically comes down to tradition, ubiquitousness, and stubbornness. Which is sad to me. We fix other stuff. If an exploit appears in the OS it gets patched. Yet here we have a system that basically defaults to bad behavior, un-secure. It should be redesigned/replaced with something that makes it hard to do it wrong. If not replacing then maybe adding a "safe" mode and slowly deprecating the non-safe way. Sure it may take years. So did moving from python 2 to 3 but it is happening. Maybe it will take the powers-that-be to start having their data stolen though shell exploits to realize if they made the shell hard to get wrong by default they'd be helping themselves as well?
- LeoPanthera 9y agoBecause it is ubiquitous. You can virtually guarantee that bash will be found on any arbitrary unix-like system.
- nickpsecurity 9y agoThat makes it a good idea for a compiler target for a better programming language. Not using it, though. ;)
- kazinator 9y agoSomeone thought it was a good idea to use a portable subset of the shell as a target for a worse programming language: layers of M4 macros. They didn't back away with anything like "not using it though". So now we have GNU Autotools.
- 9y ago
- grawprog 9y agoI find these days I avoid bash a much as possible. I find myself using D for things I would normally use shell scripts for. Bash just...I dunno...why? Why was it designed in such a stupid way....?
- wocram 9y agoIs this project ready to share? It's not published to crates.io, and doesn't have a license or Cargo.toml
- deleted 9y ago[deleted]
- xixixao 9y agoIf your everyday language is JavaScript, use shelljs. It’s great. You’ll get portable scripts in no time. Sure it’s “slow”, in a way that likely doesn’t matter because of what you’re calling from the script.
- meow_mix 9y agoWhy ever resort to shell scripts when we have languages like ruby / python for anything more complex than installs? Clarity is king people
- scbrg 9y agoBecause foo | awk '/bar/ {print $3}' is more clear than import subprocess foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE) for line in foo.stdout: if 'bar' in line: try: print(line.split()[2]) except IndexError: print('') Sometimes shell scripts are more clear. Especially for tasks that involve running lots of external commands.
- meow_mix 9y agoNobody doubts that pipes and languages like awk are great for one liners, but I think that's a little besides the point of this post, which is advocating for things like the use of bash arrays: ``` array=( a b ) array+=(c) if [ ${#array[@]} -gt 0 ]; then rm -- "${array[@]}" fi ``` ``` array = [a, b] array << c array.map { |i| `rm #{i}` } if array.length > 0 ``` There's also nobody stopping you from using text processing tools like awk and sed, or bash one liners in ruby/python either, but I think we should leave the logic and arrays for scripting languages, no?
- scbrg 9y agoYou're moving the goalposts! Previously it was "anything but install scripts". Now it's "logic and arrays". I actually think we agree with each other, but we express it differently. I never write bash scripts, I write POSIX shell, so all the array juggling of bash is something I never deal with. As you say, by the time you need arrays, you should have switched languages already. That said, I think there's a fairly large domain of problems - apart from install scripts - that are better solved with shell scripts, because of their clarity. Anything which relies on invokation of lots of other tools, and in particular problems that fit the pipe model (take output from this tool, extract interesting bits from it, and feed it to that tool, etc). And this I say as an otherwise almost slightly fanatical Pythonista :-)
- Annatar 9y ago" Should I use backticks? Command substitutions also come in this form: Correct: "`cmd`" Bad: `cmd` While it is possible to use this style correctly, it looks even more awkward in quotes and is less readable when nested. The consensus around this one is pretty clear: Avoid." This is how one can tell a rookie just learning to program in shell: usage of $() syntax is limited to Bourne family of shells which implement that particular POSIX specification aspect. Backticks, on the other hand, although they incur a performance penalty since they spawn a subshell, make one's code instantly portable across all UNIX-like operating systems and even across disparate shell families, as they work exactly the same in C-shells. (Whether one should program in a C-shell family is a different discussion.) The subshell performance penalty is negligible in 99% of the cases as this 1970's technology has tiny memory and processor overhead due to the fact that it's been developed on systems with small memory and a slow CPU, so it had been optimized for performance. Over my 30+ years of shell programming, I know of only one documented instance where the $() construct which doesn't spawn a subshell made a difference, and was the only time it was actually a valid requirement: https://www.joyent.com/blog/building-packages-at-scale https://www.joyent.com/blog/building-packages-at-scale but even then, the author ended up using dash, not bash. For maximum portability and closest adherence to POSIX, program in Korn shell, ksh93. (Modern versions of ksh implement ksh93 functionality.) Then you may safely use $() and be assured it will work in all Korn shells across different operating systems (even in ksh88). Otherwise, DON'T avoid using backticks, because you will be giving away portability for no good reason. Don't program in bash, but in original Bourne shell (sh) for maximum portability across different operating systems; don't assume that you can use bash constructs in Bourne shell (as /bin/sh on GNU/Linux tells bash to run in Bourne shell emulation mode, but that mode isn't implemented completely or correctly, since bash constructs are still accepted). Always test your shell code on a traditional UNIX like HP-UX or a Solaris derivative like SmartOS if possible, with a real Bourne shell.
- posix_me_less 9y ago> "This is how one can tell a rookie just learning to program in shell: usage of $() syntax is limited to Bourne family of shells which implement that particular POSIX specification aspect. Backticks, on the other hand, although they incur a performance penalty since they spawn a subshell, make one's code instantly portable across all UNIX-like operating systems and even across disparate shell families, as they work exactly the same in C-shells." The '$()' notation is a standard shell feature de facto. The fraction of people who care about their script working on all UNIX-like operating systems' default shells is very close to 0. The recommendation is fine - this notation is more readable and nestable. > "Always test your shell code on a traditional UNIX like HP-UX or a Solaris derivative like SmartOS if possible, with a real Bourne shell." Only if your script needs to be "original Bourne shell" compatibile. Which is almost never for most script writers.
- Groxx 9y agoOverwhelmingly a great resource, though I will nitpick at this one: >Globbing is easier to use correctly than find. For very simple purposes or small file trees, sure. Outside that: find is incredibly more powerful, and worth using in many cases. If nothing else, learning to use `find . -path ignore_this_path -prune -o *.ext -print` can change e.g. a Go project script from visibly-slow to instant. (e.g. the latest place I used this went from 1-5+ (hot vs cold) seconds to 5-50ms)
- JepZ 9y agoWhile I agree with the guide, there is one thing I was missing while writing bash scripts with 'set -e' and that was some kind of stack tracing. So I added a nice trap function to my personal bash script template. Be aware, that this version does not include all best practices described in the guide. #!/usr/bin/env bash #-------------------------------------------- # Default Bash Script Header set -eu trap stacktrace EXIT function stacktrace { if [ $? != 0 ]; then echo -e "\nThe command '$BASH_COMMAND' triggerd a stacktrace:" for i in $(seq 1 $((${#FUNCNAME[@]} - 2))); do j=$(($i+1)); echo -e "\t${BASH_SOURCE[$i]}: ${FUNCNAME[$i]}() called in ${BASH_SOURCE[$j]}:${BASH_LINENO[$i]}"; done fi } SCRIPT_DIR="$(dirname "$(readlink -f "$0")")" #--------------------------------------------
- fredley 9y agoA tip from me, based on a mistake I made yesterday: don't `source ~/.bash_history` instead of `~/.bash_profile`. (Luckily it entered vim relatively soon, from where I could kill the process).
- xyproto 9y agoAt least it's not recursive.
- tzahola 9y agoObligatory: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/
- elbear 9y agoI notice a pattern in all articles about BASH. Some people say it should die. Others praise its ubiquity and versatility at handling certain tasks. Of course, both sides are right. From my point of view, BASH will never disappear, because it's not a living thing that runs out of food or habitat and dies. That doesn't happen, unless there's a major revolution in computing that makes current paradigms obsolete, something on the scale of the disappearance of the dinosaurs. Until that happens, I welcome any projects that aim to decrease the amount of buggy BASH in the wild. I avoid it as much as possible, but if someone's going to use it, at least they'll have safety nets to reduce the possible damage. PS: I have a feeling my metaphors are all over the place, but I hope that doesn't detract from the message.
- ausjke 9y agoFor further portability you can convert bash shell to C and compile it into binaries. https://github.com/neurobin/shc https://github.com/neurobin/shc
- vinceguidry 9y agoReason number 650 why I love Ruby: it offers ridiculously gradual transitions from shell scripts. One time I built up a pretty weighty conditional-heavy script in Bash and didn't want to keep adding to it. So I simply made all the bash calls use backticks. Took me all of a few minutes to get option parsing working again. One day I'll learn the Ruby way of doing shell one-liners and that'll hopefully keep me out of man pages for that sort of thing forever.
- andreyv 9y ago> Furthermore, set -u must not be used in Bash 4.3 and earlier. This is a dangerous advice. set -u indeed requires more verbose array handling in Bash < 4.4, but it also catches code like this: rm -rf "${OOPS_UNDEFINED}/"
- yaccz 9y agoerrexit gives action at distance antipattern.