26 ms·

How to do things safely in Bash (2018)

173 points by soheilpro 6y ago

pwdisswordfish8 6y ago
I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for each little string manipulation task. And if I ever need to ‘shell out’ to an external process, there's always the subprocess module that is much more robust than ‘set -e’ ever was. Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead.
- lanstin 6y ago
  All the deployment things are just wrappers to shell commands so if you want to help people debug user-data.sh, cron, ansible, docker files, or puppet, you need to understand shell. Also, it is better to have something in bash as code than not have it be fully automated. I draw the line at arrays tho. If you want arrays don’t do bash. But if you want to just run a bunch of commands on your compute, bash is there. One thing I have started doing for stuff that need to go via ssh is I just do a here doc for an embedded shell script that I scp to the remote and then just use ssh to invoke it. The quoting and pseudotty stuff just got to be too dumb to understand. Also, and maybe this is just for non-interactive scripts, I find everything from writing them to using them is a lot easier if you take pains to make them idempotent, so you can run them one or 100 times with the same effect.
- yeowMeng 6y ago
  > The [python] scripts are much faster... Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell. Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? I like python, but if I am in a rush it’s shell.
  - neolog 6y ago
    > Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? Are you asking why people make recommendations in general?
  - marcosdumay 6y ago
    Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.
  - kaliszad 6y ago
    Actually, the one thing that I as a systems engineer really like about Clojure is the threading macro (-> some-data some-function-applied-on-some-data some-otherfunction) or (->> ...) for inserting not as a second, but last argument. That is like a pipeline but usually more readable to me even though I am proficient with the shell (and I must admit even PowerShell, which really is more comparable to Perl with a bit of SQL and other flavours here and there than bash). Sysadmins/ system engineers might love Babashka: https://github.com/babashka/babashka https://github.com/babashka/babashka which is a large subset of Clojure + some frequently used libraries as a native GraalVM image. It is portable, has very fast startup and if the script becomes a larger program, you can easily switch to ClojureScript + Node.js (e.g. for still very fast startup) or Clojure (on the normal JVM) or perhaps build your own GraalVM image. You might also just open a REPL and run it as a single session but that is rather unique in the sysadmin/ systems engineer space, where most things are launched on schedule e.g. each 5 minutes by a script and in case the startup time is somewhat long, it might dominate the execution time. Btw. babashka seems to be about twice as fast to start on my Debian: time bb -e '(+ 1 1)' executes on average in about 11 ms vs time python3 -c 'print(1 + 1)' executes on average in about 23 ms
- IshKebab 6y ago
  100% agree. This article should just be "rewrite your script in Python". It's not the best scripting language ever, but it works fairly well and it's about 100 times better than Bash.
- stormbrew 6y ago
  This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been. I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?
  - blabitty 6y ago
    I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.
    - kubanczyk 6y ago
      Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.
- jrockway 6y ago
  Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". The shell is a pretty good REPL, and glue like seq, find, xargs, grep, etc. are very good. But honestly, I find myself reaching for purpose-built tools rather than ad-hoc pipelines more and more. fdfind does more of what I want on average than find; rg does more of what I want than 'find | xargs grep'. I also find myself using more and more structured data, and write more jq pipelines than I do bash pipelines. Finally, I am getting more and more frustrated at basic shell mechanics like history handling. I have 5 terminals open on average, and I don't understand why C-r in one won't find history in another (I know why, of course, but I don't like it). I'm getting pretty close to just writing my own shell and toolchain. I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't. They just hacked Unicode and remote RPCs into shell prompts so that pressing "enter" on an empty shell prompt takes 25 seconds to run. I'm not sure why people are so focused on the polish and not the core inadequacies, but they are, and I feel like I'm going to have to fix that myself in the next couple years...
  - chubot 6y ago
    Have you tried any of the shells here? https://github.com/oilshell/oil/wiki/Alternative-Shells https://github.com/oilshell/oil/wiki/Alternative-Shells Discussed recently: https://news.ycombinator.com/item?id=26121592 https://news.ycombinator.com/item?id=26121592 A lot of them are "fundamentally reimagining" shell -- actually I'd say MOST of them are; whether that's good or bad depends on the user's POV. Oil is reimagining shell, but also providing a graceful upgrade path. Out of all the shells I'd say it's most focused on the fundamental language and runtime, and less on the interactive issues (right now).
  - deleted 6y ago
    [deleted]
  - imiric 6y ago
    I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks. > I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls. > I find myself reaching for purpose-built tools rather than ad-hoc pipelines It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell. > I am getting more and more frustrated at basic shell mechanics like history handling That's mostly a solved problem in many shells. Zsh supports it natively[2]. I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost. [1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ [2]: https://nuclearsquid.com/writings/shared-history-in-zsh/ https://nuclearsquid.com/writings/shared-history-in-zsh/
- chubot 6y ago
  Shouldn't scripts over 100 lines be rewritten in Python or Ruby? http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh... tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write) Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Professional-Computng/dp/0131429019 https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe... Online for free: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/
- Supersaiyan_IV 6y ago
  At least shell follows POSIX standard which is more than one can say about Python. Have witnessed self-proclaimed "shell replacing" python scripts invoking commands, and never checking the exit code. Debugging hell.
- macromagnon 6y ago
  I like bash because it's easy to connect different data together. I can output a column from a sql query to a file and easily write a for loop that goes over the data and use it. I work mostly in payment related code, so most often I use bash to hot fix in production. Like if we need to refund certain people, I'll convert the refund api call to a curl call, get the data from sql and then just run a simple loop.
- gorgoiler 6y ago
  Bash is fine as long as everything works. Handling errors in bash is a fool’s errand and you’re quite right to use Python for resilience in the face of errors or malformed input. Sometimes you just gotta eat. Python is a sit-down meal. Bash is a can of beans. I’m happy eating both for dinner but it depends who (if anyone) I’m sharing my meal with.
- lenkite 6y ago
  Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!
antegamisou 6y ago
Here's a great resource for proper Bash syntax http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide
alblue 6y ago
If you’re interested in writing safe shell scripts then check out shellcheck: https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck If you’re interested I’ve written a git hook for it that runs a check when you git commit: https://github.com/alblue/scripts/blob/main/shellcheck-pre-commit https://github.com/alblue/scripts/blob/main/shellcheck-pre-c... You should also check out her Google shell script style guide: https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html
- woudsma 6y ago
  Shellcheck and the VS Code Shellcheck extension[1] are really good. I never write a bash script without it. Built-in autofixes and direct links to extensive documentation on each rule, with easy to grasp examples. It's just awesome. [1]: https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck https://marketplace.visualstudio.com/items?itemName=timonwon...
- pabs3 6y ago
  An earlier comment of mine about some more shell checking/linting/formatting tools: https://news.ycombinator.com/item?id=26510549 https://news.ycombinator.com/item?id=26510549
nas 6y ago
The fact that you have to use quoting nearly everywhere is a design flaw in the Borne shell. Some shells, like Plan 9's rc, for example, don't expand after variable substantiation. They have an operator to call if you want to explicitly force expansion. That's so much cleaner and less error prone.
- rustshellscript 6y ago
  zsh is also doing the variable substitution better than bash. FYI, I just released rust_cmd_lib 1.0 recently, which can do variable substitution without any quotes: https://github.com/rust-shell-script/rust_cmd_lib https://github.com/rust-shell-script/rust_cmd_lib
- chubot 6y ago
  Oil is Bourne compatible, but has a mode to opt you into the better behavior. Example: osh$ empty='' osh$ x='name with spaces.mp3' This is like Bourne shell: $ argv $empty $x ['name', 'with', 'spaces.mp3'] # omit empty and split $ argv "$empty" "$x" ['', 'name with spaces.mp3'] # unchanged Opt into better behavior, also available with bin/oil: $ shopt --set oil:basic $ argv $empty $x ['', 'name with spaces.mp3'] # no splitting/elision If you want to omit empty strings, you can use the maybe() function, which returns a 0 or 1 length array for SPLICING with @: $ argv @maybe(empty) "$x" ['name with spaces.mp3'] # omitted empty string Example of splicing arrays: $ array=("foo $x" '1 2') $ argv $empty @array ['', 'foo name with spaces.mp3', '1 2'] This is called "Simple Word Evaluation": https://www.oilshell.org/release/latest/doc/simple-word-eval.html https://www.oilshell.org/release/latest/doc/simple-word-eval... Feedback appreciated! (Interestingly zsh also doesn't split words, but it silently removes empty strings).
  - JNRowe 6y ago
    > Interestingly zsh also doesn't split words Should zsh users ever want that behavior they can enable it globally with the SH_WORD_SPLIT option, or more sensibly local to a given parameter with the = parameter expansion as in ${=var}. Also, it has the best comment in zsh's manpage: "SH_WORD_SPLIT [...] Note that this option has nothing to do with word splitting." $ x='name with spaces.mp3' $ print -l -- $=x # "print -l" displays one element per line name with spaces.mp3 > it silently removes empty strings You can keep the empty strings too if needed, but it requires using the @ expansion flag as in ${(@)arr}. It all becomes superbly readable, here I'll prove it: $ arr=($x '' 'old file') $ print -l -- "${(@D)arr:A:gs/old/new/}" ~/Desktop/name with spaces.mp3 ~/Desktop/new file In all seriousness, the zsh default handling feels right for interactive usage in this case. I'd love the strictness of oil, so that I simply don't need to remember these things. Not yet quite ready to give up on the zshexpn(1) goodies though.
hutrdvnj 6y ago
I think the best way to write things safely is to not write them in Bash in the first place. Better pick a strongly typed language of your choice.
- macintux 6y ago
  How many strongly typed languages are: + Installed everywhere + Even within 20% as concise as bash
  - deleted 6y ago
    [deleted]
  - effie 6y ago
    Indeed. Python does not win over bash in either category, and it is slower. I wish there was a super-fast shell-like scripting language everywhere that could serve as a general programming language. Like Perl, but simpler rules and readable syntax. But there is nothing. Who would have thought, I may have to learn Perl eventually.
    - drran 6y ago
      IMHO, we need more than one language in the shell: one to orchestrate launching of applications, plumbing, and processing of their inputs and outputs, another one for computations with integers, floats, complex values, arrays, matrices, another one for text processing and parsing, another one for argument parsing and default values, another one for error handling and recovery, another one for pattern matching and state machines, another one for object-oriented programming, another one for network programming, another one for record-oriented programming, another one for security and debugging, another one for container management, another one for system manipulation, upgrading, recovering, another one for documentation, and so on, with advanced way to share variables and data between these languages (environment variables on steroids). It's not possible to implement such monstrosity as one simple, portable, small, fast, and secure binary.
      - 6y ago
arielb1 6y ago
When calling shell from bash (to use pipes etc.), I found it useful to pass variables via the environment, like this: def safe_call(command, **keywds): return subprocess.check_call(f'set -euo pipefail; {command}', shell=True, env=keywds) safe_call('command1 -- "$bar" | command2 --baz="$baz" | command3 > "$output_file"', bar=bar, baz=baz, output_file=output_file)
- arielb1 6y ago
  The main difficulty I have with this method is that I don't have an easy way to pass a Python array to bash as an array of parameters.
montroser 6y ago
This all has its place, and I learned a good few things from the article. But also, in many situations when you may not need to deal with completely arbitrary input, it can be nice to shed all of the "correct" syntax and just optimize for readability.
maxrev17 6y ago
See the same debate ensue each time with bash - use python, they shout. Perhaps the overriding factor here is familiarity? Past looping and some basic flow control I'm guessing most people run out of being comfortable in bash and can get things done quicker in their fave Lang?
- kaba0 6y ago
  It’s not about being comfortable, but about the million ways your shell script will fail. A file has a space in the name? Good chance your script failed. No file founs? Same. Multiple files found after expansion? Same.
- Too 6y ago
  Please read http://mywiki.wooledge.org/BashPitfalls http://mywiki.wooledge.org/BashPitfalls. Now you can find such a page of footguns for almost any language out there. But this on a whole different level and affects even the most fundamental features. One can't even for-loop a list of files or use echo in the first intuitive way without running into a trap that potentially blows up your whole system. It's the complete opposite of "pit of success" Familiarity is of course an important factor, though I would actually claim the problem is the opposite, people reach towards bash - because of familiarity of what they type on the cli. Not realizing that scripting is a complete different world from interactive use where you can inspect, correct and manually adjust every action step by step, with same input each time. A language with only string as a data type (it has arrays and numbers but its so confusing and converts implicitly that you might as well ignore it, IFS anyone?), surprising quoting, data and input interpreted as code, globbing and expansions where you least expect it, global variable everything, no module-system, only error handling is either errexit or manually checking each command and every sub-pipe, functions make error-handling behave even weirder. It can't be made safe no matter how hard you try to familiarize yourself with it.
geocrasher 6y ago
I've been shell scripting intermittently for a long time and have written some bash scripts that have seen a lot of use. This document really showed me how bad some of my programming is. But then again, the first thing I tell people when demonstrating one of these programs: "I am not a programmer. I am a bash scripter. And, a bad one." The stuff still works. But I would feel a lot better if I were doing things the right way. I'll be reviewing this next time I write a bigger s/program/script/
- inetknght 6y ago
  Shellcheck helped me learn a lot about bash. https://www.shellcheck.net/ https://www.shellcheck.net/
jwilk 6y ago
A portable way to test if the variable is set: test -n "${VAR+x}"
xonix 6y ago
I personally have found that AWK is a surprisingly good alternative to shells for scripts bigger than average. Why? 1. Portability (AWK is a part of POSIX) 2. Very clean syntax 3. Powerful associative arrays 4. Super-powerful string manipulations facilities 5. Easy shell interoperability As an example here is a simplistic build tool [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/Win (via Git Bash). [1] https://github.com/xonixx/makesure/blob/main/makesure.awk https://github.com/xonixx/makesure/blob/main/makesure.awk [2] https://github.com/xonixx/makesure/actions/runs/702594092 https://github.com/xonixx/makesure/actions/runs/702594092
- drran 6y ago
  AWK is good, but Perl is better. Perl is good, but Python is cleaner. Python is good, but Go is better. Go is good, but Rust is faster. Rust is good, but shell is simpler. Shell is good, but AWK is better...
  - max_hammer 6y ago
    Perl is like swiss army knife for sysadmins.
nunez 6y ago
If you're gonna Bash, definitely use shellcheck for everything. If you're writing Bash in vim, I'd highly recommend installing Syntastic and enabling the checker below: ``` let g:syntastic_sh_checkers = ["shellcheck", "-e", "SC1090"] ```
- greggyb 6y ago
  Note that HN doesn't support markdown, but does have its own limited markup. Specifically, backticks are simply displayed, rather than marking a code snippet or block. You can indent by four spaces for a code block as below: This is a preformatted section preceded by four spaces in the comment editor.
belter 6y ago
Every time I think of writing some Bash I spend some time here: https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls and then ...I use something else.
- agumonkey 6y ago
  Yeah it's ironic that you have such a wonderful repository of knowledge only to realize that you shouldn't have that much corner cases and bag of tricks.
gdavisson 6y ago
I agree with almost everything in this guide, but it has a couple of recommendations that create new problems (while solving others): The nullglob option ('shopt -s nullglob') makes things like 'for f in .txt' work right when there are no matching files, but make other commands like 'grep somepattern .txt' change their behavior in ways that can cause serious trouble. grep (and many other commands), given no files as input, will read from standard input (up to the EOF); if it's running interactively and input hasn't been redirected, this causes the script to hang for no discernible reason. If input has been redirected, it steals input that was presumably meant to be read by some other command. In my opinion, the problems this causes are more serious than what it solves. The errexit option ('set -e' or 'shopt -s errexit') can both fail to exit when you expect/want it to (several such situations are described in the guide) and also exit when you don't expect/want it to. The guide mentions suppressing unexpected exits with '|| true', but it's not always obvious when this is needed, and it's not even consistent between versions of bash. There's a good parable about this (and some examples) at http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105 I really don't like trying to predict and work around unpredictable features like this; I'd much rather deal with explicit error handling, like 'commandThatMightFail || { echo "Aaaargh" >&2; exit 1; }'
dang 6y ago
Discussed at the time: Safe ways to do things in bash - https://news.ycombinator.com/item?id=17057596 https://news.ycombinator.com/item?id=17057596 - May 2018 (240 comments)
renewiltord 6y ago
Embarrassingly, instead of fixing the code, I fix the data for bash. I just make sure my input is well-formed haha. I don't think I have any files or folders with spaces in them on my computer. It may not be the way but it is a way.
- lpapez 6y ago
  I tend to do this as well. This is the way.
- deleted 6y ago
  [deleted]
tediousdemise 6y ago
PowerShell is cross platform now and it’s quite good. I would encourage people who’ve never tried it to give it a shot. Like 99% of other people who never used PoSh, I thought it was just a Windows shovelware replacement for Command Prompt. I was mistaken—the syntax is actually a lot simpler than bash, but it’s just as capable. One of the cooler features about PoSh is that you can leverage Visual Basic/C#/.NET from it, and you can script GUIs (like Python’s tkinter library).
- JNRowe 6y ago
  I'd probably like to see powershell usage increase too, but I think we may be in the minority. Every time it comes up I check to see if it is available in Debian and see that the packaging bug¹ has been open for years without anyone caring enough to move on it. The referenced upstream bug² points out that it isn't capital-F Free enough to distribute [yet], but you'd expect more comments in the bug if people were pushing(or a package for the non-free section). ¹ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834756 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834756 ² https://github.com/PowerShell/PowerShell/issues/5869 https://github.com/PowerShell/PowerShell/issues/5869
- haolez 6y ago
  But What-Is-Up-With-Those-Really-Long-Commands? :)
commandlinefan 6y ago
> Command substitution: > Good: "$(cmd)" > Bad: $(cmd) And I’m over here still using `cmd`...
ta20210403 6y ago
Just for the love of God don't use Perl.

173 points by soheilpro 6y ago

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for each little string manipulation task. And if I ever need to ‘shell out’ to an external process, there's always the subprocess module that is much more robust than ‘set -e’ ever was. Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead.

lanstin 6y ago
All the deployment things are just wrappers to shell commands so if you want to help people debug user-data.sh, cron, ansible, docker files, or puppet, you need to understand shell. Also, it is better to have something in bash as code than not have it be fully automated. I draw the line at arrays tho. If you want arrays don’t do bash. But if you want to just run a bunch of commands on your compute, bash is there. One thing I have started doing for stuff that need to go via ssh is I just do a here doc for an embedded shell script that I scp to the remote and then just use ssh to invoke it. The quoting and pseudotty stuff just got to be too dumb to understand. Also, and maybe this is just for non-interactive scripts, I find everything from writing them to using them is a lot easier if you take pains to make them idempotent, so you can run them one or 100 times with the same effect.
yeowMeng 6y ago
> The [python] scripts are much faster... Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell. Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? I like python, but if I am in a rush it’s shell.
- neolog 6y ago
  > Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? Are you asking why people make recommendations in general?
- marcosdumay 6y ago
  Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.
- kaliszad 6y ago
  Actually, the one thing that I as a systems engineer really like about Clojure is the threading macro (-> some-data some-function-applied-on-some-data some-otherfunction) or (->> ...) for inserting not as a second, but last argument. That is like a pipeline but usually more readable to me even though I am proficient with the shell (and I must admit even PowerShell, which really is more comparable to Perl with a bit of SQL and other flavours here and there than bash). Sysadmins/ system engineers might love Babashka: https://github.com/babashka/babashka https://github.com/babashka/babashka which is a large subset of Clojure + some frequently used libraries as a native GraalVM image. It is portable, has very fast startup and if the script becomes a larger program, you can easily switch to ClojureScript + Node.js (e.g. for still very fast startup) or Clojure (on the normal JVM) or perhaps build your own GraalVM image. You might also just open a REPL and run it as a single session but that is rather unique in the sysadmin/ systems engineer space, where most things are launched on schedule e.g. each 5 minutes by a script and in case the startup time is somewhat long, it might dominate the execution time. Btw. babashka seems to be about twice as fast to start on my Debian: time bb -e '(+ 1 1)' executes on average in about 11 ms vs time python3 -c 'print(1 + 1)' executes on average in about 23 ms
IshKebab 6y ago
100% agree. This article should just be "rewrite your script in Python". It's not the best scripting language ever, but it works fairly well and it's about 100 times better than Bash.
stormbrew 6y ago
This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been. I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?
- blabitty 6y ago
  I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.
  - kubanczyk 6y ago
    Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.
jrockway 6y ago
Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". The shell is a pretty good REPL, and glue like seq, find, xargs, grep, etc. are very good. But honestly, I find myself reaching for purpose-built tools rather than ad-hoc pipelines more and more. fdfind does more of what I want on average than find; rg does more of what I want than 'find | xargs grep'. I also find myself using more and more structured data, and write more jq pipelines than I do bash pipelines. Finally, I am getting more and more frustrated at basic shell mechanics like history handling. I have 5 terminals open on average, and I don't understand why C-r in one won't find history in another (I know why, of course, but I don't like it). I'm getting pretty close to just writing my own shell and toolchain. I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't. They just hacked Unicode and remote RPCs into shell prompts so that pressing "enter" on an empty shell prompt takes 25 seconds to run. I'm not sure why people are so focused on the polish and not the core inadequacies, but they are, and I feel like I'm going to have to fix that myself in the next couple years...
- chubot 6y ago
  Have you tried any of the shells here? https://github.com/oilshell/oil/wiki/Alternative-Shells https://github.com/oilshell/oil/wiki/Alternative-Shells Discussed recently: https://news.ycombinator.com/item?id=26121592 https://news.ycombinator.com/item?id=26121592 A lot of them are "fundamentally reimagining" shell -- actually I'd say MOST of them are; whether that's good or bad depends on the user's POV. Oil is reimagining shell, but also providing a graceful upgrade path. Out of all the shells I'd say it's most focused on the fundamental language and runtime, and less on the interactive issues (right now).
- deleted 6y ago
  [deleted]
- imiric 6y ago
  I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks. > I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls. > I find myself reaching for purpose-built tools rather than ad-hoc pipelines It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell. > I am getting more and more frustrated at basic shell mechanics like history handling That's mostly a solved problem in many shells. Zsh supports it natively[2]. I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost. [1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ [2]: https://nuclearsquid.com/writings/shared-history-in-zsh/ https://nuclearsquid.com/writings/shared-history-in-zsh/
chubot 6y ago
Shouldn't scripts over 100 lines be rewritten in Python or Ruby? http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh... tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write) Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Professional-Computng/dp/0131429019 https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe... Online for free: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/
Supersaiyan_IV 6y ago
At least shell follows POSIX standard which is more than one can say about Python. Have witnessed self-proclaimed "shell replacing" python scripts invoking commands, and never checking the exit code. Debugging hell.
macromagnon 6y ago
I like bash because it's easy to connect different data together. I can output a column from a sql query to a file and easily write a for loop that goes over the data and use it. I work mostly in payment related code, so most often I use bash to hot fix in production. Like if we need to refund certain people, I'll convert the refund api call to a curl call, get the data from sql and then just run a simple loop.
gorgoiler 6y ago
Bash is fine as long as everything works. Handling errors in bash is a fool’s errand and you’re quite right to use Python for resilience in the face of errors or malformed input. Sometimes you just gotta eat. Python is a sit-down meal. Bash is a can of beans. I’m happy eating both for dinner but it depends who (if anyone) I’m sharing my meal with.
lenkite 6y ago
Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!

antegamisou 6y ago

Here's a great resource for proper Bash syntax http://mywiki.wooledge.org/BashGuide http://mywiki.wooledge.org/BashGuide

alblue 6y ago

If you’re interested in writing safe shell scripts then check out shellcheck: https://github.com/koalaman/shellcheck https://github.com/koalaman/shellcheck If you’re interested I’ve written a git hook for it that runs a check when you git commit: https://github.com/alblue/scripts/blob/main/shellcheck-pre-commit https://github.com/alblue/scripts/blob/main/shellcheck-pre-c... You should also check out her Google shell script style guide: https://google.github.io/styleguide/shellguide.html https://google.github.io/styleguide/shellguide.html

woudsma 6y ago
Shellcheck and the VS Code Shellcheck extension[1] are really good. I never write a bash script without it. Built-in autofixes and direct links to extensive documentation on each rule, with easy to grasp examples. It's just awesome. [1]: https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck https://marketplace.visualstudio.com/items?itemName=timonwon...
pabs3 6y ago
An earlier comment of mine about some more shell checking/linting/formatting tools: https://news.ycombinator.com/item?id=26510549 https://news.ycombinator.com/item?id=26510549

nas 6y ago

The fact that you have to use quoting nearly everywhere is a design flaw in the Borne shell. Some shells, like Plan 9's rc, for example, don't expand after variable substantiation. They have an operator to call if you want to explicitly force expansion. That's so much cleaner and less error prone.

rustshellscript 6y ago
zsh is also doing the variable substitution better than bash. FYI, I just released rust_cmd_lib 1.0 recently, which can do variable substitution without any quotes: https://github.com/rust-shell-script/rust_cmd_lib https://github.com/rust-shell-script/rust_cmd_lib
chubot 6y ago
Oil is Bourne compatible, but has a mode to opt you into the better behavior. Example: osh$ empty='' osh$ x='name with spaces.mp3' This is like Bourne shell: $ argv $empty $x ['name', 'with', 'spaces.mp3'] # omit empty and split $ argv "$empty" "$x" ['', 'name with spaces.mp3'] # unchanged Opt into better behavior, also available with bin/oil: $ shopt --set oil:basic $ argv $empty $x ['', 'name with spaces.mp3'] # no splitting/elision If you want to omit empty strings, you can use the maybe() function, which returns a 0 or 1 length array for SPLICING with @: $ argv @maybe(empty) "$x" ['name with spaces.mp3'] # omitted empty string Example of splicing arrays: $ array=("foo $x" '1 2') $ argv $empty @array ['', 'foo name with spaces.mp3', '1 2'] This is called "Simple Word Evaluation": https://www.oilshell.org/release/latest/doc/simple-word-eval.html https://www.oilshell.org/release/latest/doc/simple-word-eval... Feedback appreciated! (Interestingly zsh also doesn't split words, but it silently removes empty strings).
- JNRowe 6y ago
  > Interestingly zsh also doesn't split words Should zsh users ever want that behavior they can enable it globally with the SH_WORD_SPLIT option, or more sensibly local to a given parameter with the = parameter expansion as in ${=var}. Also, it has the best comment in zsh's manpage: "SH_WORD_SPLIT [...] Note that this option has nothing to do with word splitting." $ x='name with spaces.mp3' $ print -l -- $=x # "print -l" displays one element per line name with spaces.mp3 > it silently removes empty strings You can keep the empty strings too if needed, but it requires using the @ expansion flag as in ${(@)arr}. It all becomes superbly readable, here I'll prove it: $ arr=($x '' 'old file') $ print -l -- "${(@D)arr:A:gs/old/new/}" ~/Desktop/name with spaces.mp3 ~/Desktop/new file In all seriousness, the zsh default handling feels right for interactive usage in this case. I'd love the strictness of oil, so that I simply don't need to remember these things. Not yet quite ready to give up on the zshexpn(1) goodies though.

hutrdvnj 6y ago

I think the best way to write things safely is to not write them in Bash in the first place. Better pick a strongly typed language of your choice.

macintux 6y ago
How many strongly typed languages are: + Installed everywhere + Even within 20% as concise as bash
- deleted 6y ago
  [deleted]
- effie 6y ago
  Indeed. Python does not win over bash in either category, and it is slower. I wish there was a super-fast shell-like scripting language everywhere that could serve as a general programming language. Like Perl, but simpler rules and readable syntax. But there is nothing. Who would have thought, I may have to learn Perl eventually.
  - drran 6y ago
    IMHO, we need more than one language in the shell: one to orchestrate launching of applications, plumbing, and processing of their inputs and outputs, another one for computations with integers, floats, complex values, arrays, matrices, another one for text processing and parsing, another one for argument parsing and default values, another one for error handling and recovery, another one for pattern matching and state machines, another one for object-oriented programming, another one for network programming, another one for record-oriented programming, another one for security and debugging, another one for container management, another one for system manipulation, upgrading, recovering, another one for documentation, and so on, with advanced way to share variables and data between these languages (environment variables on steroids). It's not possible to implement such monstrosity as one simple, portable, small, fast, and secure binary.
    - 6y ago

arielb1 6y ago

When calling shell from bash (to use pipes etc.), I found it useful to pass variables via the environment, like this: def safe_call(command, **keywds): return subprocess.check_call(f'set -euo pipefail; {command}', shell=True, env=keywds) safe_call('command1 -- "$bar" | command2 --baz="$baz" | command3 > "$output_file"', bar=bar, baz=baz, output_file=output_file)

arielb1 6y ago
The main difficulty I have with this method is that I don't have an easy way to pass a Python array to bash as an array of parameters.

montroser 6y ago

This all has its place, and I learned a good few things from the article. But also, in many situations when you may not need to deal with completely arbitrary input, it can be nice to shed all of the "correct" syntax and just optimize for readability.

maxrev17 6y ago

See the same debate ensue each time with bash - use python, they shout. Perhaps the overriding factor here is familiarity? Past looping and some basic flow control I'm guessing most people run out of being comfortable in bash and can get things done quicker in their fave Lang?

kaba0 6y ago
It’s not about being comfortable, but about the million ways your shell script will fail. A file has a space in the name? Good chance your script failed. No file founs? Same. Multiple files found after expansion? Same.
Too 6y ago
Please read http://mywiki.wooledge.org/BashPitfalls http://mywiki.wooledge.org/BashPitfalls. Now you can find such a page of footguns for almost any language out there. But this on a whole different level and affects even the most fundamental features. One can't even for-loop a list of files or use echo in the first intuitive way without running into a trap that potentially blows up your whole system. It's the complete opposite of "pit of success" Familiarity is of course an important factor, though I would actually claim the problem is the opposite, people reach towards bash - because of familiarity of what they type on the cli. Not realizing that scripting is a complete different world from interactive use where you can inspect, correct and manually adjust every action step by step, with same input each time. A language with only string as a data type (it has arrays and numbers but its so confusing and converts implicitly that you might as well ignore it, IFS anyone?), surprising quoting, data and input interpreted as code, globbing and expansions where you least expect it, global variable everything, no module-system, only error handling is either errexit or manually checking each command and every sub-pipe, functions make error-handling behave even weirder. It can't be made safe no matter how hard you try to familiarize yourself with it.

geocrasher 6y ago

I've been shell scripting intermittently for a long time and have written some bash scripts that have seen a lot of use. This document really showed me how bad some of my programming is. But then again, the first thing I tell people when demonstrating one of these programs: "I am not a programmer. I am a bash scripter. And, a bad one." The stuff still works. But I would feel a lot better if I were doing things the right way. I'll be reviewing this next time I write a bigger s/program/script/

inetknght 6y ago
Shellcheck helped me learn a lot about bash. https://www.shellcheck.net/ https://www.shellcheck.net/

jwilk 6y ago

A portable way to test if the variable is set: test -n "${VAR+x}"

xonix 6y ago

I personally have found that AWK is a surprisingly good alternative to shells for scripts bigger than average. Why? 1. Portability (AWK is a part of POSIX) 2. Very clean syntax 3. Powerful associative arrays 4. Super-powerful string manipulations facilities 5. Easy shell interoperability As an example here is a simplistic build tool [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/Win (via Git Bash). [1] https://github.com/xonixx/makesure/blob/main/makesure.awk https://github.com/xonixx/makesure/blob/main/makesure.awk [2] https://github.com/xonixx/makesure/actions/runs/702594092 https://github.com/xonixx/makesure/actions/runs/702594092

drran 6y ago
AWK is good, but Perl is better. Perl is good, but Python is cleaner. Python is good, but Go is better. Go is good, but Rust is faster. Rust is good, but shell is simpler. Shell is good, but AWK is better...
- max_hammer 6y ago
  Perl is like swiss army knife for sysadmins.

nunez 6y ago

If you're gonna Bash, definitely use shellcheck for everything. If you're writing Bash in vim, I'd highly recommend installing Syntastic and enabling the checker below: ``` let g:syntastic_sh_checkers = ["shellcheck", "-e", "SC1090"] ```

greggyb 6y ago
Note that HN doesn't support markdown, but does have its own limited markup. Specifically, backticks are simply displayed, rather than marking a code snippet or block. You can indent by four spaces for a code block as below: This is a preformatted section preceded by four spaces in the comment editor.

belter 6y ago

Every time I think of writing some Bash I spend some time here: https://mywiki.wooledge.org/BashPitfalls https://mywiki.wooledge.org/BashPitfalls and then ...I use something else.

agumonkey 6y ago
Yeah it's ironic that you have such a wonderful repository of knowledge only to realize that you shouldn't have that much corner cases and bag of tricks.

gdavisson 6y ago

I agree with almost everything in this guide, but it has a couple of recommendations that create new problems (while solving others): The nullglob option ('shopt -s nullglob') makes things like 'for f in .txt' work right when there are no matching files, but make other commands like 'grep somepattern .txt' change their behavior in ways that can cause serious trouble. grep (and many other commands), given no files as input, will read from standard input (up to the EOF); if it's running interactively and input hasn't been redirected, this causes the script to hang for no discernible reason. If input has been redirected, it steals input that was presumably meant to be read by some other command. In my opinion, the problems this causes are more serious than what it solves. The errexit option ('set -e' or 'shopt -s errexit') can both fail to exit when you expect/want it to (several such situations are described in the guide) and also exit when you don't expect/want it to. The guide mentions suppressing unexpected exits with '|| true', but it's not always obvious when this is needed, and it's not even consistent between versions of bash. There's a good parable about this (and some examples) at http://mywiki.wooledge.org/BashFAQ/105 http://mywiki.wooledge.org/BashFAQ/105 I really don't like trying to predict and work around unpredictable features like this; I'd much rather deal with explicit error handling, like 'commandThatMightFail || { echo "Aaaargh" >&2; exit 1; }'

dang 6y ago

Discussed at the time: Safe ways to do things in bash - https://news.ycombinator.com/item?id=17057596 https://news.ycombinator.com/item?id=17057596 - May 2018 (240 comments)

renewiltord 6y ago

Embarrassingly, instead of fixing the code, I fix the data for bash. I just make sure my input is well-formed haha. I don't think I have any files or folders with spaces in them on my computer. It may not be the way but it is a way.

lpapez 6y ago
I tend to do this as well. This is the way.
deleted 6y ago
[deleted]

tediousdemise 6y ago

PowerShell is cross platform now and it’s quite good. I would encourage people who’ve never tried it to give it a shot. Like 99% of other people who never used PoSh, I thought it was just a Windows shovelware replacement for Command Prompt. I was mistaken—the syntax is actually a lot simpler than bash, but it’s just as capable. One of the cooler features about PoSh is that you can leverage Visual Basic/C#/.NET from it, and you can script GUIs (like Python’s tkinter library).

JNRowe 6y ago
I'd probably like to see powershell usage increase too, but I think we may be in the minority. Every time it comes up I check to see if it is available in Debian and see that the packaging bug¹ has been open for years without anyone caring enough to move on it. The referenced upstream bug² points out that it isn't capital-F Free enough to distribute [yet], but you'd expect more comments in the bug if people were pushing(or a package for the non-free section). ¹ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834756 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834756 ² https://github.com/PowerShell/PowerShell/issues/5869 https://github.com/PowerShell/PowerShell/issues/5869
haolez 6y ago
But What-Is-Up-With-Those-Really-Long-Commands? :)

commandlinefan 6y ago

> Command substitution: > Good: "$(cmd)" > Bad: $(cmd) And I’m over here still using `cmd`...

ta20210403 6y ago

Just for the love of God don't use Perl.