3 ms·
Bash is definitely painful to write - I write shell scripts a lot and I enjoy it (in a masochistic kind of way), and I wholeheartedly agree with most of the cri
by paskozdilar 4y ago
Bash is definitely painful to write - I write shell scripts a lot and I enjoy it (in a masochistic kind of way), and I wholeheartedly agree with most of the criticisms you've written.
However, the greatest strength of Bash is its ubiquity - it is available on almost every modern Unix-like environment - and when combined with standard POSIX tools it can provide a-little-less-painful
environment for writing reliable shell scripts.
In order to prove it, I will do my best to provide some kind of solution to each of the problems you have mentioned. Note that they are in no way perfect solutions, and all of them are just hacks that work around Bash's inherent clumsiness.
---
> 100% reliable argument passing
Wrapping a variable in double-quotes does this:
var="1 2 3 4"
for x in "$var"; do echo $x; done
For double-parsing (e.g. SSH commands) you can wrap the entire expression in single-quotes to maintain the original string until the second shell:
ssh user@host 'var="1 2 3 4"; for x in "$var"; do echo $x; done'
> 100% reliable file iteration
The "find" command does exactly this, and it works for whatever weird characters in the filename you have.
find ./directory/ -type f -exec SOME_COMMAND {} \;
> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.
Do not evaluate arguments directly in shell, use xargs to feed the arguments as standard input. E.g.:
find ./directory/ -type f | xargs SOME_COMMAND
> Excellent path parsing. Such as filename, basename, canonicalization, finding the file extension and "find the relative path between A and B".
dirname /foo/bar
basename /foo/bar
realpath ./symlink/to/foo/bar
basename /foo/bar | sed 's/.*\.//'
realpath --relative-to ./dirA/fileA ./dirB/fileB
> Good error handling and reporting
This is a tough one. Good error handling/reporting in Bash is extremely hard and requires discipline, but it's possible.
To exit immediately as soon as a command exits with non-zero exitcode:
set -e
However, this does not play nice with temporary resources, so I personally prefer hand-written error checks:
SOME_COMMAND || { echo "Some Command has failed" && exit 1; }
Error reporting depends on a scenario, but I generally have a pattern of doing checks like:
if [ "$var" != "expected" ]
then echo "Error: var different from expected" && exit 1
fi
And it works fine most of the time.
> Easy capture of stdout and stderr, at the same time. Either together or individually, as needed.
To capture both stdout and stderr in a same stream:
SOME_COMMAND |& SOME_FILTER
Capturing stdout and stderr in different streams is tough. The easiest way I can think of is to
use a pipe and job control mechanisms:
mkfifo pipe
SOME_COMMAND 2>pipe | FILTER_1 &
< pipe FILTER_2 &
wait
rm pipe
* Excellent process management. We're in 2022, FFS. We have 128 core CPUs. A modern shell scripting language should make it trivial to do something like: take these 50000 files, and feed them all through imagemagick, using every core available, while being able to report progress, record each failure, and abort the entire thing if needed.
While bash is truly terrible when it comes to concurrency, I find GNU Parallel to be pretty satisfying for most concurrent shell-scripting:
cat FILE.txt | parallel --timeout=30 SOME_COMMAND {} \|\| echo "Some Command failed with argument: {}"
For POSIX purity, xargs also can be used with "-L 1" argument that parses a single line per command iteration. For parallelism, there is also a "--max-procs" argument.
* Excellent error reporting. I don't want things failing with "Command failed, aborted". I want things to fail with "Command 'git checkout https:// https://....' exited with return code 3, and here's for good measure the stdout and stderr even if I redirected them somewhere".
I find that adding "set -x" to the top of your shell, which prints each command with "+" prefix as it is expanded and executed, very useful for error tracking.
* Give me helpers for common situations. Eg, "Recurse through this directory, while ignoring .git and vim backup files". Read this file into an array, splitting by newline, in a single line of code. It's tiresome to implement that kind of thing in every script I write. At the very least it should be simple and comfortable.
There are helpers out there for most common situations - it's just that they are implemented as CLI tools, and not officially part of the shell. And in the scenario where you can't expect the availability of those unofficial tools, you can always write your own small library of commonly used Bash functions, and just copy-paste them into your script whenever you need them. It's ugly, but still possible.
- Izkata 4y ago> > No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long. > Do not evaluate arguments directly in shell, use xargs to feed the arguments as standard input. E.g.: > find ./directory/ -type f | xargs SOME_COMMAND That'll have problems with characters like spaces, you either have to use the -print0 construct or just stick with find: find ./directory/ -type f -exec SOME_COMMAND {} + The + means "stack up as many results as you can, up to the length limit".
- paskozdilar 4y agoNice, I didn't know about that. Thanks for sharing!