8 ms·
Advanced Bash-Scripting Guide (2014)
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- uhtred 4y agoCue all the "when your program gets longer than one line you should consider using python" comments.
- metadat 4y ago100% sane advice on the "should" front, but I don't always follow the shoulds.. Bash is just too damn fast to whip things up in (even at the risk of endless footguns). Build up a pipeline of commands, translate into script, done. It's powerful and can scale if common conventions and good practices are used. For easier debugging I always add a simple flag along the lines of: if $1 = -v then set -x; shift Obviously shellscripts are a terrible or highly suspect idea when production-grade transactional integrity is critical, but in many cases the swiftness of shell development outweighs the cons when the goal is to Get Things Done. The shell is a pretty sweet interface.
- cjvirtucio 4y agoit's so much easier to glue tools together with it. I'd end up writing more code if I had used python's subprocess package or ruby's IO package. as long as I don't need a complex data structure of some sort.
- js2 4y agoLast week I wrote a bash script that ended up being about 250 lines. It was mostly running other processes. For technical reasons, it needed to re-invoke itself a few times via sudo. When I was done I looked over it and thought: this is a nice piece of code right now. It's well documented. It's self contained. It passes my tests. I could maintain this code. But it made use of quite a few bash'isms: arrays and so forth. I wasn't confident in the ability of my colleagues to keep it in good state. So I rewrote it in Python. The Python version ended up being a little longer, but that was mostly due to formatting difference. The Python version in addition I could reformat with black and isort, I added type annotations so I could test it with mypy. (I had formatting the shell version with shfmt and checked it with shellcheck, but still.) The Python version is much easier to read, and I think will be more maintainable. I consider myself competent in both languages so it's a courtesy to my colleagues if nothing else to ship the Python version.
- disgruntledphd2 4y agoI used to think this way, and still do to a large extent. Unfortunately, we still need bash to build software (docker containers for example) so there's a limit to how much bash we can get rid of, unfortunately.
- js2 4y agoOf course, and I still use it in those places where it's required and relatively self contained, or frankly, if it just happens to be the best tool for the job which it sometimes is. But now that I can expect python3 to be available on the systems I care about, and with the fairly comfortable ergonomics of subprocess.run, I find myself using Python in places I would've previously used bash. A good heuristic is the moment a bash script needs to use curl + jq, it's time for python. Especially on macOS it's nice having things like plistlib so I can avoid `defaults` and `PlistBuddy`.
- aaaaaaaaaaab 4y agoYawn… Python sucks for building processing pipelines. How many lines of Python code would you need to implement this? paste -d \\n <(foo …) <(bar …) | while read -r f && read -r b; do baz “$f” “$b” done Mind you, `foo` and `bar` run in parallel.
- slt2021 4y agoreplace bash one-liner with python? No, thanks replace 200+ bash script with multiple subroutines in python? yes
- dmtroyer 4y agoas someone who is maintaining mountains of legacy bash, for all that is holy, yes. or Go. or Rust. anything.
- idispatch 4y agoMind you, before this goes to prod please add error handling and logging. What happens when bar fails? What about baz? Now, please rewrite this so your team mates can maintain and extend. Yawn, I’ll wait for the PR.
- aaaaaaaaaaab 4y agoGotcha, thanks for the feedback! We should also totally A/B test this! I'll make sure that this ties in with the quarterly OKRs, too! I hope I'll get promoted to E6 for that sweet 10% raise next cycle :fingers_crossed:
- d0mine 4y agoNothing stops you from calling bash from Python to combine bash pipelines (onelines) but if you must use Python for some reason, to run `baz` for each line in `foo` & `bar` output: from subprocess import Popen, PIPE, run with Popen('foo', stdout=PIPE) as foo, Popen('bar', stdout=PIPE) as bar: for f, b in zip(foo.stdout, bar.stdout): run(['baz', f, b]) (there is a slight difference in behaviour)
- bryanlarsen 4y agoIf your primary consideration is that it's generally available on a random developer's machine, what are the options? IOW out of the box on OSX and the major Linux distributions, and readily available on Windows. Python is pretty safe, but your code would have to work with both 2 & 3 and not pull in any external dependencies. And recommended practices have changed so drastically on how to install Python over the years that a random dev could have theirs in a fairly weird state. Perl would work, but hacked-together Perl has the similar drawbacks to hacked-together Bash, especially when it's me doing the coding. Ruby has problems similar to Python. Make & awk have problems similar to Perl & Bash. Therefore my latest rust package contains bash scripts.
- vaporup 4y agoTCL
- bryanlarsen 4y agoThat's not installed by default on Linux distributions that I'm aware of. Expect is an awesome tool though.
- kevin_thibedeau 4y agoIt's in the Python standard library. Tkinter is just a wrapper around Tcl.
- gpderetta 4y agoTo be fair you can do a lot in a bash one liner.
- dmtroyer 4y agodoesn’t mean you should, for the sake of the next person.
- vlunkr 4y agoMy rule is when you start needing functions or arrays you should consider using something else. Bash is really neat and easy to get started with, but maintaining a large codebase is terrible.
- dotancohen 4y agoThis is a good guide for Bash, but I wouldn't use the code as-is in production. For example, the following code is recommended for cleaning up a log file and leaving only the last 50 lines: tail -n $lines messages > mesg.temp mv mesg.temp messages That code has a clear race condition and will lose lines on a busy server.
- metadat 4y agoYes, use `mktemp', just as you would in Python, Go, Rust, C, or Java.
- metadat 4y agoActually, to protect against the clobber of the original file, you could use a flocker lock on the process to serialize the operations. Otherwise it will be unsafe.
- wswope 4y agoHow would you fix that race condition?
- formerly_proven 4y agoHave the writing process do it. Which kind of reminds me of that product running on a pseudo-distributed NoSQL database built out of text files with "record per line" structure and sort of a compacting garbage collector for these files to remove dead records. And there's just absolutely nothing that can go wrong with that.
- dotancohen 4y agoI would rotate the log file, and possibly store only the last log file.
- metadat 4y agoLots of good refreshers, the cat + heredoc combo is powerful but lots of cryptically subtle variations in behavior between: <<EOF (expands variables etc) <<'EOF' (doesn't expand variables etc) <<-EOF (trims leading tabs) Am I missing any more of the tricks?
- dotopotoro 4y agoTrimming version does not trim whitespace indentation. I guess point goes to “tabs” camp.
- metadat 4y agoCorrect, though you could pipe to sed and trim leading spaces with relative ease: sed 's/^ *//' <<EOF Which I'd prefer anyways because it's less cryptic. I try to stay away from the obscure, rarely used notations because I'll always have to look it up in StackOverflow anytime I come back to it. These can be confusingly sharp edges.
- 0xbadcafebee 4y agohttps://linux.die.net/man/1/bash https://linux.die.net/man/1/bash "Here Documents" and "Here Strings" sections.
- alganet 4y agoYou can do it in compound commands as well: while read -r LINE do printf %s "$LINE" done <<FOO foo bar baz FOO This is faster than using cat (one less path lookup, one less fork, one less exec, read is builtin), works on all shells. There is a <<-'EOF' as well (trims left tabs and does not expand vars). Another cool trick that only bash and zsh have is `printf -v NAME`, which prints to a variable instead of stdout. You can use it to avoid subshells (one less fork for each subshell avoided).
- 0xbadcafebee 4y agoThe best guide to using Bash is to read the manual (https://linux.die.net/man/1/bash https://linux.die.net/man/1/bash). Nobody who has ever used Bash has read the whole manual. It is actually good and useful, please read it. Yes, it is very big. But you will be using this shell for the rest of your life (even if you use Zsh, you will end up using other people's Bash scripts). Consider it an investment in your future. ...and now that I say that - try to restrict shell scripts to POSIX semantics (https://betterprogramming.pub/24-bashism-to-avoid-for-posix-compliant-shell-scripts-8e7c09e0f49a https://betterprogramming.pub/24-bashism-to-avoid-for-posix-...). POSIX is very simple and most shells can run it. If you need a Bashism, try Bash v3 first so it works on a Mac without Homebrew. Simple is better/portabler than complex. If you need something Bash doesn't provide, use any language that the majority of your team knows well. If you want to test a specific version of Bash, there's containers @ https://hub.docker.com/_/bash https://hub.docker.com/_/bash (Macs use v3.2.57). Test POSIX scripts with `bash --posix`. Definitely use Shellcheck.
- dijit 4y ago>Nobody who has ever used Bash has read the whole manual. Bold claim and patently untrue because I have read the whole manual. However: 1) Things you don't use often leak out of your brain at a rate that is absurd. 2) Different versions have different features and there's wide variations between versions on different platforms, you can't actually depend on a lot of bash stuff existing, so instead it's usually better to target POSIX sh
- jagged-chisel 4y agoI, too, experience Swiss-cheese-brain. I once thought I’d solve this by taking notes, organizing a journal, and writing ... Well, it was a manual. That had the same problem as other manuals - too long, not gonna read. Full circle.
- nocman 4y agoYour note taking, organizing and writing will have helped you retain a lot more info for a longer period of time. It wasn't wasted effort. Still, there is a such thing as "too much". Higher degrees of success are achieved by regularly evaluating where you are in the "too little" to "too much" spectrum.
- nixcraft 4y agoI also like, BASH Frequently Asked Questions https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ and linter https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck
- michaelsbradley 4y agoBash Hackers Wiki is also a nice resource: https://wiki.bash-hackers.org/ https://wiki.bash-hackers.org/
- alganet 4y agoI am on the portable shell team. It is hard to write for all shells, but totally worth it. If we all did it, reusing shell code would be easier. We never left the hell caused by the war between shells to provide interactive features. They were all feature creeped for terminal usage and the scripting API was left to rot. I get the people that say we should use python or something else instead. It won't work though if that language is not capable of replacing the shell _as a build dependency_. As long as we need the shell to build stuff, we're stuck to it. Dockerfiles, GitHub Actions YAMLs, all these tools accept shell input. This is shiny new software that could have picked any language for their proprietary DSLs, but the shell was chosen. What the shell needs is what happened to JavaScript around the 2000s. Different interpreters started to align, standards were written, reusable libraries started to pop up everywhere, and now it runs in our brains. Before all of that, JavaScript was merely a cute toy to animate pages.
- js2 4y agoThe one thing I really miss in portable shell code is arrays. There’s a place reserved in hell for whoever thought it was a good idea to design a build system around shell code in yaml files. I’m in the process of moving thousands of lines of such code into standalone scripts that then get called from the yaml files. Keep the yaml as minimal as possible. Then I can shellcheck the shell code or easily port bits to other languages as needed.
- alganet 4y agoYou can reimplement arrays using pure shell! I saw it first here: https://github.com/shellfire-dev/core/blob/master/variable/array.functions https://github.com/shellfire-dev/core/blob/master/variable/a... For many cases, this is even better than bash native arrays. You can pass these as arguments and preserve the structure, while the bash ones will autoexpand if you try that. I ended up cooking my own array implementation later, but it is a bit more unorthodox (I create extra vars with numeric pointers to handle the values). Let me know if you want to see it, it's not complete but it works. I'm not very fond of YAML either, I feel your pain :(
- ufo 4y agoSometimes I pause to think how we're held back by needing to make our shell scripts backwards compatible with Bash or POSIX, because that's what is installed by default. Shell scripting would be nicer if the shell could be less crufty. For example, get rid of all the footguns about expanding unquoted variables...
- arinlen 4y ago> Sometimes I pause to think how we're held back by needing to make our shell scripts backwards compatible with Bash or POSIX, because that's what is installed by default. I think you're missing the forest for the trees. Bash is installed by default because Bash is specified in an international standard dubbed the Portable Operating System Interface (POSIX). Your bash scripts are not backwards compatible. Your bash scripts are compatible, and portable, and standardized, because that's their point, and the whole point of POSIX. No one is held back by bash or POSIX. You are free to use anything that suits your fancy. Feel free to whip out scripts in Python or Perl or Ruby. Some people do, and last time I checked some operating systems like macOS shipped their interpreters by default. Odds are you are free to easily install those interpreters in those who don't.
- theonemind 4y agoPOSIX most definitely does not specify bash. People typically use the word "bashism" for POSIX-shell-incompatible bash behavior used in shell scripts. random high result from googling "bashism": https://betterprogramming.pub/24-bashism-to-avoid-for-posix-compliant-shell-scripts-8e7c09e0f49a https://betterprogramming.pub/24-bashism-to-avoid-for-posix-...
- Beltalowda 4y ago> No one is held back by bash or POSIX. You are free to use anything that suits your fancy. Feel free to whip out scripts in Python or Perl or Ruby. The point is that there are many improvements that could be made to bash which would retain the good properties but would get rid of the worst footguns and suckiness. Quite a number of these (though not all) have been available for decades in the form of zsh. Also, as pointed out already in another comment, bash is not in POSIX at all. POSIX is very much a "lowest common denominator"-kind of specification which moves at geological speeds. Even dash and busybox sh implement some non-POSIX features.
- redleader55 4y agoI'm getting a "404 No Found" on my mobile. Is this supposed to be META? Edit: it's a genuine question, I have no idea what everyone else is seeing.
- dejj 4y agoBackup here: http://web.archive.org/web/20220819180012/https://tldp.org/LDP/abs/html/ http://web.archive.org/web/20220819180012/https://tldp.org/L...
- stjohnswarts 4y agoDoesn't this get posted like every other month?
- AceJohnny2 4y agoChecking HN Search (the box at the bottom of the page), this link has only been posted 3 times ever, last time being 11 months ago (and before that, 4 years ago).
- dejj 4y agoDon't forget to shellcheck¹ your script before checking it in. Also available in VSCode, IntelliJ, and others. [1] https://www.shellcheck.net/ https://www.shellcheck.net/
- Cwizard 4y agoI’m getting a 404
- AceJohnny2 4y agoDo not follow this guide. It is outdated and IIRC in some places actually wrong. Much, much better is the Bash Hacker's Wiki: https://wiki.bash-hackers.org/ https://wiki.bash-hackers.org/ And Wooledge's Wiki: https://mywiki.wooledge.org/BashFAQ https://mywiki.wooledge.org/BashFAQ
- alganet 4y agoAnother simple trick that I almost forgot: You don't need to escape new lines if you end them with an operator: test 1 = 1 && test a = b || exit 122 Works with all logical operators, ampersand and pipe. By the way, the ampersand is often expressed with a new line after it, but it doesn't require it: this_in_background & this_in_foreground This is the same as writing: this_in_background & this_in_foreground
- hsbauauvhabzb 4y agoI’m sure I saw a quote from a bash developer saying that bash shouldn’t be considered for security, but dont recall where I saw it. I’d love to know a source if anyone knows where I can find it.
- Too 4y agoNo need to quote a security expert to realize that. A language that mixes code and data as the same variable type, with only a single missing quote needed to turn any variable into a full RCE. On top of that, add all the obscure syntax, stringly typed data for everything and with whatever measly error handling it actually does support being defaulted to off. Guess what you get.