21 ms·

Pure Bash Bible

746 points by ausjke 7y ago

0x0aff374668 7y ago
Very useful. I hate having to pull in external processes to do something simple in BASH that the manpage doesn't cover. Expecting PERL to be on every machine (or the right version of SED.. posix? gnu?) is not reliable in my line of work.
no-dr-onboard 7y ago
Oh cool. Same fellow who made pywal.
yread 7y ago
for f in *; do is cute but what about those idiots who put spaces in the filenames?
buraequete 7y ago
I expected a tool on bash where you can access the pure "Bible" hallelujah.sh
- sodaplayer 7y ago
  On that note, does anyone have a favorite cli Bible reading program?
  - buraequete 7y ago
    Hallelujah brother https://github.com/BibleJS/BibleApp https://github.com/BibleJS/BibleApp
  - Yajirobe 7y ago
    Why is this getting downvoted?
    - amdavidson 7y ago
      Because it is super offtopic. I would also downvote "does anyone have a favorite CLI recipe manager?"
  - unhammer 7y ago
    `apt search bible` gives lots of hits, though my favourite edition is unfortunately only available on tumblr: https://kingjamesprogramming.tumblr.com/ https://kingjamesprogramming.tumblr.com/
heuxbzjnz 7y ago
no mention of /dev/tcp?! yes, it looks like a device node in /dev, but it's really a pure bashism for opening tcp connections to arbitrary hosts and ports
- unixhero 7y ago
  Whoa... News to me! Thanks.
- dyanaraps 7y ago
  I've been meaning to get around to it. I hadn't messed around with /dev/tcp prior to writing this bible. I've since implemented a very bare-bones and very featureless IRC client using /dev/tcp and bash. https://github.com/dylanaraps/birch https://github.com/dylanaraps/birch I will get around to it eventually. The one hurdle I want to get over before writing a piece about it is the handling of binary data using bash. This is something a little tricky to do with bash but it'd allow for a 'wget'/'curl' like program without the use of anything external to the shell (no HTTPS of course). I want to really understand the feature before I write about it though in the meantime I could just write a reference to the syntax/basic usage. :)
- anaphor 7y ago
  https://www.linuxjournal.com/content/more-using-bashs-built-devtcp-file-tcpip https://www.linuxjournal.com/content/more-using-bashs-built-... has a decent explanation. I came here to say the exact same thing. This is my favourite thing most people don't know exists in Bash.
- dredmorbius 7y ago
  Keep in mind that major distros (Debian, possibly derivatives) disabled tcp/udp service names in 2000: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65172 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65172
jigglesniggle 7y ago
While this is interesting I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh. Run shellcheck on some of your own code, or the code of even a simple project to see how hard it is to really use bash. Why I think people gravitate towards it is because languages such as python add too much pomp to launching a shell process. A language like perl is usually easier to use but everyone hates it now.
- masklinn 7y ago
  > I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh I would heartily agree. If your shell script grows beyond half a dozen commands or so, you're probably better off rewriting it in just about anything. Python, ruby, go (gorun), rust (cargo-script), or whatever else, doesn't really matter as long as it's not shell.
  - dvfjsdhgfv 7y ago
    Or D via rdmd: https://dlang.org/rdmd.html https://dlang.org/rdmd.html
- danielecook 7y ago
  I don’t think it’s pomp. Once I learned Unix pipes and the tools for manipulating data (sed awk cut etc), it just became much faster and easier than writing python scripts that do the same thing. You can literally connect the output of one process to the input of another with a single character. It’s much more complex in python. It also tends to be very portable.
  - jigglesniggle 7y ago
    I think you're reading my comment the wrong way: I meant to say that doing e.g. piping in Python is a lot of pointless work (pomp), as you agree. This is perhaps one big benefit but not one that is exclusive to a sh-like language. Instead I would like to see a language with strong flow control or metaprogramming capabilities take on processes as a first class citizen. Perl is probably the closest but still has some warts related to redirection. The best pattern I have seen is encapsulating business logic into Python or Go and then if really necessary piping it to another script. But, often, if you do this you can just keep the piping internal using data structures. Unix-pattern facilities work very well for interactive use, which tends to be simple, exploratory, and trial-and-error. But a project's build script may not be simple.
deleted 7y ago
[deleted]
NegativeLatency 7y ago
Kinda cool, I don’t write or read much bash and tend to stick to sh compatible stuff. There are some neat tricks in here but they don’t seem very readable compared to perl/awk/sed.
- hagreet 7y ago
  Yeah, I also don't look at things like ``` trim_string() { # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]}"}" : "${_%"${_##[![:space:]]}"}" printf '%s\n' "$_" } ``` and think: "I should use bash more". Bash is nice for making simple things simple but for complicated things it's just shitty. I used to think that this is due to the complicated quoting rules which make the simple things simple but tcl does a much better job at that. In either case I prefer the clean rules of a Python or Perl for anything larger.
nxpnsv 7y ago
Neat. Sometimes bash is the right choice, and then a guide like this is a great complement to google / stackexchange.
tannhaeuser 7y ago
Didn't realize at first that "pure" refers to features available in bash without calling out to external processes, when I would've thought purism in this context should refer to avoiding bashisms and writing portable (ksh, POSIX shell) scripts.
- dyanaraps 7y ago
  I understand your concerns about bash features and POSIX compatibility. The bash bible was written specifically to document the shell extensions bash implements. My focus for the past few months has been writing a Linux distribution (and its package manager/tooling) in POSIX sh. I've learned a lot of tricks and I'm very tempted to write a second "bible" with snippets that are supported in all POSIX shells. (I created the bash bible).
  - acqq 7y ago
    > I've learned a lot of tricks and I'm very tempted to write a second "bible" with snippets that are supported in all POSIX shells. That could be potentially even more interesting than a bash-specific one (as it is harder to get it right -- bash can be figured out out of the single reference, anything "portable" has many dependencies).
    - dyanaraps 7y ago
      Off the top of my head, a few notable things I've learned: - Safely working with "string lists" (list="el el el el"). - Filtering out duplicate items. - Reversing the list. - etc. - Using `case` to do sub-string matching (using globbing). - Using `set -- el el el` to create an "array" (only one array at a time!). - `read -r` is still powerful in POSIX `sh` for getting data out of files. `while read -r` even more so. - POSIX `sh` has `set -e` and friends so you can exit on errors etc. - Ternary operators still exist for arithmetic (`$(($# > 0 ? 1 : 0))`). - Each POSIX `sh` shell has a set of quirks you need to account for. What works in one POSIX `sh` shell may not in another. (I found a set of differences between `ash`/`dash` in my testing). POSIX `sh` is a very simple language compared to `bash` and all of its extensions so there won't be as many snippets but there's some gold to be found.
al_form2000 7y ago
Very nice recipes. I'd avoid shadowing actual executables with bash function names (see head in the book) as a harbinger of great grief.
ethanpil 7y ago
Would love to see this in a ZSH equivalency... Especially with the impending move to ZSH in MacOS.
avyeed_desa 7y ago
I posted a link to the author's linux distribution [1] yesterday, that relies heavily on BASH scripts[2] [1] https://getkiss.org/ https://getkiss.org/ [2] https://news.ycombinator.com/item?id=21002928 https://news.ycombinator.com/item?id=21002928
dyanaraps 7y ago
Hello, I'm the author of the Pure Bash Bible. Happy to answer any questions you may have. Here's an example of what bash is capable of: https://github.com/dylanaraps/fff/ https://github.com/dylanaraps/fff/ (a TUI file manager written in bash)!
- hnarn 7y ago
  This might be an odd/off-topic question, but in Telegram this article has an auto-fetched thumbnail of a cat smoking a cigarette and a text similar to 'heavy metal music playing', I'm just curious where this picture is from, if you have any idea? I checked the README for the repo, pictures of the contributors etc. but I'm unable to figure out where it's coming from.
  - dyanaraps 7y ago
    That's a very very very old GitHub avatar of mine, I wonder why Telegram hasn't pulled a later one.
    - hnarn 7y ago
      Curious, thanks for the answer!
    - tckmn 7y ago
      Telegram also shows my old Github avatar - it seems to cache them extremely aggressively.
    - folex 7y ago
      You can use https://telegram.me/webpagebot https://telegram.me/webpagebot to refresh avatar cache
  - yahnu 7y ago
    The site[1] where the video is actually hosted has the image you described as the avatar of the uploader. Could it be the reason? 1. https://asciinema.org/a/qvNlrFrGB3xKZXb6GkremjZNp https://asciinema.org/a/qvNlrFrGB3xKZXb6GkremjZNp
deviantfero 7y ago
Great read dylan!
- dyanaraps 7y ago
  Thanks! Fancy seeing you here. :)
tutfbhuf 7y ago
How about performance? Is it fast as rust or slow as python?
faragon 7y ago
Related, pure Bash L77 data compression/decompression, crc32, hex enc/dec, base64 enc/dex, binary head/cut, in a single 13KB file: https://github.com/faragon/lzb https://github.com/faragon/lzb
- dyanaraps 7y ago
  This is really neat, thanks for sharing.
thih9 7y ago
I'm only using bash occasionally, this was useful. In particular the "obsolete syntax" section, I wasn't aware of it. https://github.com/dylanaraps/pure-bash-bible#obsolete-syntax https://github.com/dylanaraps/pure-bash-bible#obsolete-synta...
croo 7y ago
I write a bash script or two every month so I thought I'm okay. But then came along the very first example: trim_string() { # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" } Ok, so the : is somehow a temporary variable... Then there is a variable starting at $ and you lost me :D Can someone break down that line for me? What the hell is going on here? : "${1#"${1%%[![:space:]]*}"}"
- vghnb 7y ago
  : is the null command. Kind of like /bin/true. So : followed by anything exists simply to perform expansion on something. If you fully understand what that means, then you will understand this: $_ is simply “whatever the last argument to the previous command expanded to”. So : is the previous command and the ${1… expanded value is now ${_…. Because $_ is used in the expansion of itself, it is the same value, because the command has not yet completed, which would (re) set $_. So use of a thing ($_) in expanding the same thing ($_) is perfectly fine until after that (null) command (:) runs. You see that the final $_ is used standalone. Hope this helps.
- lervag 7y ago
  I think the idea is that the `:` builtin allows expansion of arguments without actually doing anything else. However, the temporary variable $_ is filled with the content of the expression. That is, after the first : "${1#"${1%%[![:space:]]*}"}" The $_ temporary variable contains the result of removing the leading spaces. In the next line, the spaces at the end are removed from the temporary variable with the "${_%..." syntax. You can test this in your own shell by e.g. doing: : $PATH echo $_
- emmelaich 7y ago
  I don't like this function very much but here's a few notes... : is a "do nothing" command -- but the line is still evaluated %% means to replace leading chars that match pattern ## means replace trailing chars I don't know why they're using $_; thats the variable containing the interpreter name, i.e. "/bin/bash" [edit - also the name of the previous command!] I can't be bothered analyzing it any further :-)
deleted 7y ago
[deleted]
jbrnh 7y ago
I don't quite get the recommendation to always use env bash over #!/bin/bash? If I use the full path, it is to get just that - the system's Bash. If it is missing or overruled in $PATH then I most likely don't want the script to run in the first place.
- YawningAngel 7y ago
  'System bash' isn't a universal or clear concept. For example, if you're using Modern OS X, you likely have bash via brew or some other userspace package manager but no system bash. Presumably you (or at least, most people) would still like your scripts to run in this case.
  - jbrnh 7y ago
    But if I write the script for a bunch of RHEL servers then OS X and brew are irrelevant, and the full path is better (IMHO). It's the 'always use env..' I object to.
    - moviuro 7y ago
      It's RHEL... today. `/usr/bin/env` is a POSIX standard. Maybe one day, RHEL will put bash in /usr/local/bin. Or maybe you'll switch to FreeBSD one day, and suddenly everything goes boom.
      - jbrnh 7y ago
        env and sh are both POSIX but AFAIK the path is not specified for either of them. If POSIX has an opinion on how you should start a script I would be happy for a link? Bash in RHEL is in /usr/bin and /bin as /bin in symlinked to /usr/bin. I think it is equally unlikely that RHEL (Debian, SLES..) will will move either /bin/bash or /usr/bin/env as it would break a million scripts out there. If we should migrate to FreeBSD while, for some reason, reusing linux oriented bash scripts, changing the path to /usr/local/bin/ would be the least of my headaches. I agree that 'env' can make good sense if you don't know who/where/when your script is used. For internal projects, I don't really see the advantage.
Jenz 7y ago
Looking at the very first example: trim_string(){ # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" } This reads horrible. I see no reason to prefer this over programs like sed, bash is after all a shell, intended firstly for running external programs/commands.
- collyw 7y ago
  > I see no reason to prefer this over programs like sed, bash is after all a shell, intended firstly for running external commands. Are there systems that don't come with sed installed? (Some docker containers I have logged into don't seem to have less).
  - hnlmorg 7y ago
    I've never known `sed` not be installed but I have been caught out by different implementations of sed before.
  - Jenz 7y ago
    Well, less is only intended for interactive use AFAIK, in contrast to sed which is very much scriptable; and part of the POSIX standard [1] [1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...
- Tepix 7y ago
  > This reads horrible. Just use the shell function by its descriptive name... The reasons are explained in the foreword: Calling an external process in bash is expensive and excessive use will cause a noticeable slowdown. Scripts and programs written using built-in methods (where applicable) will be faster, require fewer dependencies and afford a better understanding of the language itself.
  - james_s_tayler 7y ago
    The slow down caused to any developer that has to try and read that is probably orders of magnitude more worth optimizing for than how fast a bath script runs. I'll take the grep/sed/awk version.
jmnicolas 7y ago
I have a personal dislike for regexes and non human readable code, it gives maintenance headache. It's why I avoid shell scripts as much as possible. The first example is not human readable, if the name of the function is a lie, I have no idea what this piece of code do : trim_string() { : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" }
- 0x0aff374668 7y ago
  "maintenance headache." Then stop maintaining code from languages you don't understand. This is frustrating. I've seen solutions on the comments that call PYTHON! Are you effing kidding me? PYTHON ?? Granted, BASH docs aren't particularly succinct, but shell scripts are an absolute necessity in the OS world. It's pretty simple: If you don't understand shellcode, don't maintain an OS, or rather, don't expect to be accommodated for lack of knowledge for something that's been standard for 30+ years.
- Too 7y ago
  I only got to the first example and had a hard time deciding if the article was satire or not. Let's use bash instead of a real programming language! It is super convenient! To do super fundamental things like trimming strings you just have to implement your own function with 40+ non-ASCII characters in a row! That example makes regexps seem like human readable and god knows if it is even correct or works portably across different versions of bash. To be fair the article assumes you are first in bash and want to avoid launching subprocesses but for any production scripting that is the wrong hypothesis to begin with. A better approach would be to not start inside bash at all. Just do your scripting in a higher level language where basic ABC stuff like string, list, number and error handling are already there for you. Even if that trim_string function and everything else from the article would be provided in some bash-bible-std-lib it wouldn't even come close to what's available in say Python for example, and you still have to wrestle the syntax and other obscurities like -a (or -e) meaning file exists. While there are some good examples in there if you are stuck in bash, this article was more of a 100 reasons not to use bash to me.
- serhart 7y ago
  Just because you seem to be unfamiliar with bash doesn’t mean it’s unreadable. Almost any language will look cryptic if you don’t know it. That function is mostly just parameter expansion and very common in most bash scripts. I bet if you read the manual you would easily be able to figure it out. You just have to learn the language.
mistrial9 7y ago
I have inherited about 4K LOC Bash, which mostly works as advertised. No one wants to touch it! suggestions ?
- stjohnswarts 7y ago
  Leave it alone if you don't need to modify it. If you start needing to modify it a lot start rewriting it in ruby, go, or python at that point. Use set -x at the top of the file to get an idea on what's going on.
- tjpnz 7y ago
  Assuming it's all in one file I would move related pieces of functionality into separate files and and then source them when necessary. Should make things more manageable for people only wanting to make small changes.
  - mistrial9 7y ago
    no, its 27 directories and 947 files, with a plugin architecture (pipeline)
    - mistrial9 7y ago
      my mistake, the project is closer to 40,000 LOC, now that I count it...
- andrewl-hn 7y ago
  Skim https://learnxinyminutes.com/docs/bash/ https://learnxinyminutes.com/docs/bash/ Then open the code and just go through it. Check everything you find unfamiliar with that cheat sheet. If it doesn't cover something in your code, write it down to research separately later.
tjpnz 7y ago
Can anyone recommend something similar for Make?
vinceguidry 7y ago
I have moved to Crystal Lang for the times when i need the speed that ruby can't offer. I could have leveled up my bash skills, and was indeed using bash for that porpoise, but crystal is easier i think.
bwang008 7y ago
As someone that writes bash scripts regularly for my job, one of the big struggles I've had is remembering what I or someone else wrote when I look back 6+ months later haha. I had the same issue with reading other people's perl as well. I think the great and terrible thing about both languages is that there is literally a million different ways to skin the cat / write a regex.
taviso 7y ago
This seems like a good time to mention my (ridiculous) project, a ctypes module for bash. https://github.com/taviso/ctypes.sh/wiki https://github.com/taviso/ctypes.sh/wiki There are some little demos here: https://github.com/taviso/ctypes.sh/tree/master/test https://github.com/taviso/ctypes.sh/tree/master/test I even ported the GTK+3 Hello World to bash as a demo: https://github.com/taviso/ctypes.sh/blob/master/test/gtk.sh https://github.com/taviso/ctypes.sh/blob/master/test/gtk.sh
- bokchoi 7y ago
  This is so awesome. I need to play with this.
- donpdonp 7y ago
  "ctypes.sh is a bash plugin that provides a foreign function interface directly in your shell."
- singron 7y ago
  I found ctypes.sh to be legitimately useful for managing resources in nix-shell. E.g. flock a lock file and set CLOEXEC so that subprocesses don't hold the lock open after the shell exits. E.g. Use memfd_create to create a temp file and write a key. Then pass /proc/$$/fd/$FD to programs that need the key as a file. When the shell exits, the file can no longer be opened. You can do similar things with traps, but they aren't guaranteed to execute, whereas these OS primitives will always be cleaned up.
- valenciarose 7y ago
  This is MARVELOUSLY ridiculous and I love it. Thanks for sharing your work.
ape4 7y ago
Looks more like a "cookbook"
Sir_Cmpwn 7y ago
Please limit your shell scripting to POSIX sh for the sake of broad compatibility with current and future operating systems. There's a quick reference here: https://shellhaters.org/ https://shellhaters.org/ If you find yourself frustrated by the lack of bash extensions, your program is probably complex enough that you probably shouldn't be writing a shell script.
iandinwoodie 7y ago
Dylan, I was wondering if you could share your process for going from chapter text files to a full ebook? This looks like a really approachable way to writing a book. Is the conversion something scripted or is it a more involved process?
eatbitseveryday 7y ago
The title made me think you can query The Bible in bash...
- dredmorbius 7y ago
  Not bash, but CLI: https://packages.debian.org/buster/bible-kjv https://packages.debian.org/buster/bible-kjv
sigzero 7y ago
Kudos for using shellcheck. I use it all the time.

746 points by ausjke 7y ago

Very useful. I hate having to pull in external processes to do something simple in BASH that the manpage doesn't cover. Expecting PERL to be on every machine (or the right version of SED.. posix? gnu?) is not reliable in my line of work.

no-dr-onboard 7y ago

Oh cool. Same fellow who made pywal.

yread 7y ago

for f in *; do is cute but what about those idiots who put spaces in the filenames?

buraequete 7y ago

I expected a tool on bash where you can access the pure "Bible" hallelujah.sh

sodaplayer 7y ago
On that note, does anyone have a favorite cli Bible reading program?
- buraequete 7y ago
  Hallelujah brother https://github.com/BibleJS/BibleApp https://github.com/BibleJS/BibleApp
- Yajirobe 7y ago
  Why is this getting downvoted?
  - amdavidson 7y ago
    Because it is super offtopic. I would also downvote "does anyone have a favorite CLI recipe manager?"
- unhammer 7y ago
  `apt search bible` gives lots of hits, though my favourite edition is unfortunately only available on tumblr: https://kingjamesprogramming.tumblr.com/ https://kingjamesprogramming.tumblr.com/

heuxbzjnz 7y ago

no mention of /dev/tcp?! yes, it looks like a device node in /dev, but it's really a pure bashism for opening tcp connections to arbitrary hosts and ports

unixhero 7y ago
Whoa... News to me! Thanks.
dyanaraps 7y ago
I've been meaning to get around to it. I hadn't messed around with /dev/tcp prior to writing this bible. I've since implemented a very bare-bones and very featureless IRC client using /dev/tcp and bash. https://github.com/dylanaraps/birch https://github.com/dylanaraps/birch I will get around to it eventually. The one hurdle I want to get over before writing a piece about it is the handling of binary data using bash. This is something a little tricky to do with bash but it'd allow for a 'wget'/'curl' like program without the use of anything external to the shell (no HTTPS of course). I want to really understand the feature before I write about it though in the meantime I could just write a reference to the syntax/basic usage. :)
anaphor 7y ago
https://www.linuxjournal.com/content/more-using-bashs-built-devtcp-file-tcpip https://www.linuxjournal.com/content/more-using-bashs-built-... has a decent explanation. I came here to say the exact same thing. This is my favourite thing most people don't know exists in Bash.
dredmorbius 7y ago
Keep in mind that major distros (Debian, possibly derivatives) disabled tcp/udp service names in 2000: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65172 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=65172

jigglesniggle 7y ago

While this is interesting I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh. Run shellcheck on some of your own code, or the code of even a simple project to see how hard it is to really use bash. Why I think people gravitate towards it is because languages such as python add too much pomp to launching a shell process. A language like perl is usually easier to use but everyone hates it now.

masklinn 7y ago
> I see doing anything but launching programs with simple text-substituted arguments as too much for bash or sh I would heartily agree. If your shell script grows beyond half a dozen commands or so, you're probably better off rewriting it in just about anything. Python, ruby, go (gorun), rust (cargo-script), or whatever else, doesn't really matter as long as it's not shell.
- dvfjsdhgfv 7y ago
  Or D via rdmd: https://dlang.org/rdmd.html https://dlang.org/rdmd.html
danielecook 7y ago
I don’t think it’s pomp. Once I learned Unix pipes and the tools for manipulating data (sed awk cut etc), it just became much faster and easier than writing python scripts that do the same thing. You can literally connect the output of one process to the input of another with a single character. It’s much more complex in python. It also tends to be very portable.
- jigglesniggle 7y ago
  I think you're reading my comment the wrong way: I meant to say that doing e.g. piping in Python is a lot of pointless work (pomp), as you agree. This is perhaps one big benefit but not one that is exclusive to a sh-like language. Instead I would like to see a language with strong flow control or metaprogramming capabilities take on processes as a first class citizen. Perl is probably the closest but still has some warts related to redirection. The best pattern I have seen is encapsulating business logic into Python or Go and then if really necessary piping it to another script. But, often, if you do this you can just keep the piping internal using data structures. Unix-pattern facilities work very well for interactive use, which tends to be simple, exploratory, and trial-and-error. But a project's build script may not be simple.

deleted 7y ago

[deleted]

NegativeLatency 7y ago

Kinda cool, I don’t write or read much bash and tend to stick to sh compatible stuff. There are some neat tricks in here but they don’t seem very readable compared to perl/awk/sed.

hagreet 7y ago
Yeah, I also don't look at things like ``` trim_string() { # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]}"}" : "${_%"${_##[![:space:]]}"}" printf '%s\n' "$_" } ``` and think: "I should use bash more". Bash is nice for making simple things simple but for complicated things it's just shitty. I used to think that this is due to the complicated quoting rules which make the simple things simple but tcl does a much better job at that. In either case I prefer the clean rules of a Python or Perl for anything larger.

nxpnsv 7y ago

Neat. Sometimes bash is the right choice, and then a guide like this is a great complement to google / stackexchange.

tannhaeuser 7y ago

Didn't realize at first that "pure" refers to features available in bash without calling out to external processes, when I would've thought purism in this context should refer to avoiding bashisms and writing portable (ksh, POSIX shell) scripts.

dyanaraps 7y ago
I understand your concerns about bash features and POSIX compatibility. The bash bible was written specifically to document the shell extensions bash implements. My focus for the past few months has been writing a Linux distribution (and its package manager/tooling) in POSIX sh. I've learned a lot of tricks and I'm very tempted to write a second "bible" with snippets that are supported in all POSIX shells. (I created the bash bible).
- acqq 7y ago
  > I've learned a lot of tricks and I'm very tempted to write a second "bible" with snippets that are supported in all POSIX shells. That could be potentially even more interesting than a bash-specific one (as it is harder to get it right -- bash can be figured out out of the single reference, anything "portable" has many dependencies).
  - dyanaraps 7y ago
    Off the top of my head, a few notable things I've learned: - Safely working with "string lists" (list="el el el el"). - Filtering out duplicate items. - Reversing the list. - etc. - Using `case` to do sub-string matching (using globbing). - Using `set -- el el el` to create an "array" (only one array at a time!). - `read -r` is still powerful in POSIX `sh` for getting data out of files. `while read -r` even more so. - POSIX `sh` has `set -e` and friends so you can exit on errors etc. - Ternary operators still exist for arithmetic (`$(($# > 0 ? 1 : 0))`). - Each POSIX `sh` shell has a set of quirks you need to account for. What works in one POSIX `sh` shell may not in another. (I found a set of differences between `ash`/`dash` in my testing). POSIX `sh` is a very simple language compared to `bash` and all of its extensions so there won't be as many snippets but there's some gold to be found.

al_form2000 7y ago

Very nice recipes. I'd avoid shadowing actual executables with bash function names (see head in the book) as a harbinger of great grief.

ethanpil 7y ago

Would love to see this in a ZSH equivalency... Especially with the impending move to ZSH in MacOS.

avyeed_desa 7y ago

I posted a link to the author's linux distribution [1] yesterday, that relies heavily on BASH scripts[2] [1] https://getkiss.org/ https://getkiss.org/ [2] https://news.ycombinator.com/item?id=21002928 https://news.ycombinator.com/item?id=21002928

dyanaraps 7y ago

Hello, I'm the author of the Pure Bash Bible. Happy to answer any questions you may have. Here's an example of what bash is capable of: https://github.com/dylanaraps/fff/ https://github.com/dylanaraps/fff/ (a TUI file manager written in bash)!

hnarn 7y ago
This might be an odd/off-topic question, but in Telegram this article has an auto-fetched thumbnail of a cat smoking a cigarette and a text similar to 'heavy metal music playing', I'm just curious where this picture is from, if you have any idea? I checked the README for the repo, pictures of the contributors etc. but I'm unable to figure out where it's coming from.
- dyanaraps 7y ago
  That's a very very very old GitHub avatar of mine, I wonder why Telegram hasn't pulled a later one.
  - hnarn 7y ago
    Curious, thanks for the answer!
  - tckmn 7y ago
    Telegram also shows my old Github avatar - it seems to cache them extremely aggressively.
  - folex 7y ago
    You can use https://telegram.me/webpagebot https://telegram.me/webpagebot to refresh avatar cache
- yahnu 7y ago
  The site[1] where the video is actually hosted has the image you described as the avatar of the uploader. Could it be the reason? 1. https://asciinema.org/a/qvNlrFrGB3xKZXb6GkremjZNp https://asciinema.org/a/qvNlrFrGB3xKZXb6GkremjZNp

deviantfero 7y ago

Great read dylan!

dyanaraps 7y ago
Thanks! Fancy seeing you here. :)

tutfbhuf 7y ago

How about performance? Is it fast as rust or slow as python?

faragon 7y ago

Related, pure Bash L77 data compression/decompression, crc32, hex enc/dec, base64 enc/dex, binary head/cut, in a single 13KB file: https://github.com/faragon/lzb https://github.com/faragon/lzb

dyanaraps 7y ago
This is really neat, thanks for sharing.

thih9 7y ago

I'm only using bash occasionally, this was useful. In particular the "obsolete syntax" section, I wasn't aware of it. https://github.com/dylanaraps/pure-bash-bible#obsolete-syntax https://github.com/dylanaraps/pure-bash-bible#obsolete-synta...

croo 7y ago

I write a bash script or two every month so I thought I'm okay. But then came along the very first example: trim_string() { # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" } Ok, so the : is somehow a temporary variable... Then there is a variable starting at $ and you lost me :D Can someone break down that line for me? What the hell is going on here? : "${1#"${1%%[![:space:]]*}"}"

vghnb 7y ago
: is the null command. Kind of like /bin/true. So : followed by anything exists simply to perform expansion on something. If you fully understand what that means, then you will understand this: $_ is simply “whatever the last argument to the previous command expanded to”. So : is the previous command and the ${1… expanded value is now ${_…. Because $_ is used in the expansion of itself, it is the same value, because the command has not yet completed, which would (re) set $_. So use of a thing ($_) in expanding the same thing ($_) is perfectly fine until after that (null) command (:) runs. You see that the final $_ is used standalone. Hope this helps.
lervag 7y ago
I think the idea is that the `:` builtin allows expansion of arguments without actually doing anything else. However, the temporary variable $_ is filled with the content of the expression. That is, after the first : "${1#"${1%%[![:space:]]*}"}" The $_ temporary variable contains the result of removing the leading spaces. In the next line, the spaces at the end are removed from the temporary variable with the "${_%..." syntax. You can test this in your own shell by e.g. doing: : $PATH echo $_
emmelaich 7y ago
I don't like this function very much but here's a few notes... : is a "do nothing" command -- but the line is still evaluated %% means to replace leading chars that match pattern ## means replace trailing chars I don't know why they're using $_; thats the variable containing the interpreter name, i.e. "/bin/bash" [edit - also the name of the previous command!] I can't be bothered analyzing it any further :-)

deleted 7y ago

[deleted]

jbrnh 7y ago

I don't quite get the recommendation to always use env bash over #!/bin/bash? If I use the full path, it is to get just that - the system's Bash. If it is missing or overruled in $PATH then I most likely don't want the script to run in the first place.

YawningAngel 7y ago
'System bash' isn't a universal or clear concept. For example, if you're using Modern OS X, you likely have bash via brew or some other userspace package manager but no system bash. Presumably you (or at least, most people) would still like your scripts to run in this case.
- jbrnh 7y ago
  But if I write the script for a bunch of RHEL servers then OS X and brew are irrelevant, and the full path is better (IMHO). It's the 'always use env..' I object to.
  - moviuro 7y ago
    It's RHEL... today. `/usr/bin/env` is a POSIX standard. Maybe one day, RHEL will put bash in /usr/local/bin. Or maybe you'll switch to FreeBSD one day, and suddenly everything goes boom.
    - jbrnh 7y ago
      env and sh are both POSIX but AFAIK the path is not specified for either of them. If POSIX has an opinion on how you should start a script I would be happy for a link? Bash in RHEL is in /usr/bin and /bin as /bin in symlinked to /usr/bin. I think it is equally unlikely that RHEL (Debian, SLES..) will will move either /bin/bash or /usr/bin/env as it would break a million scripts out there. If we should migrate to FreeBSD while, for some reason, reusing linux oriented bash scripts, changing the path to /usr/local/bin/ would be the least of my headaches. I agree that 'env' can make good sense if you don't know who/where/when your script is used. For internal projects, I don't really see the advantage.

Jenz 7y ago

Looking at the very first example: trim_string(){ # Usage: trim_string " example string " : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" } This reads horrible. I see no reason to prefer this over programs like sed, bash is after all a shell, intended firstly for running external programs/commands.

collyw 7y ago
> I see no reason to prefer this over programs like sed, bash is after all a shell, intended firstly for running external commands. Are there systems that don't come with sed installed? (Some docker containers I have logged into don't seem to have less).
- hnlmorg 7y ago
  I've never known `sed` not be installed but I have been caught out by different implementations of sed before.
- Jenz 7y ago
  Well, less is only intended for interactive use AFAIK, in contrast to sed which is very much scriptable; and part of the POSIX standard [1] [1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...
Tepix 7y ago
> This reads horrible. Just use the shell function by its descriptive name... The reasons are explained in the foreword: Calling an external process in bash is expensive and excessive use will cause a noticeable slowdown. Scripts and programs written using built-in methods (where applicable) will be faster, require fewer dependencies and afford a better understanding of the language itself.
- james_s_tayler 7y ago
  The slow down caused to any developer that has to try and read that is probably orders of magnitude more worth optimizing for than how fast a bath script runs. I'll take the grep/sed/awk version.

jmnicolas 7y ago

I have a personal dislike for regexes and non human readable code, it gives maintenance headache. It's why I avoid shell scripts as much as possible. The first example is not human readable, if the name of the function is a lie, I have no idea what this piece of code do : trim_string() { : "${1#"${1%%[![:space:]]*}"}" : "${_%"${_##*[![:space:]]}"}" printf '%s\n' "$_" }

0x0aff374668 7y ago
"maintenance headache." Then stop maintaining code from languages you don't understand. This is frustrating. I've seen solutions on the comments that call PYTHON! Are you effing kidding me? PYTHON ?? Granted, BASH docs aren't particularly succinct, but shell scripts are an absolute necessity in the OS world. It's pretty simple: If you don't understand shellcode, don't maintain an OS, or rather, don't expect to be accommodated for lack of knowledge for something that's been standard for 30+ years.
Too 7y ago
I only got to the first example and had a hard time deciding if the article was satire or not. Let's use bash instead of a real programming language! It is super convenient! To do super fundamental things like trimming strings you just have to implement your own function with 40+ non-ASCII characters in a row! That example makes regexps seem like human readable and god knows if it is even correct or works portably across different versions of bash. To be fair the article assumes you are first in bash and want to avoid launching subprocesses but for any production scripting that is the wrong hypothesis to begin with. A better approach would be to not start inside bash at all. Just do your scripting in a higher level language where basic ABC stuff like string, list, number and error handling are already there for you. Even if that trim_string function and everything else from the article would be provided in some bash-bible-std-lib it wouldn't even come close to what's available in say Python for example, and you still have to wrestle the syntax and other obscurities like -a (or -e) meaning file exists. While there are some good examples in there if you are stuck in bash, this article was more of a 100 reasons not to use bash to me.
serhart 7y ago
Just because you seem to be unfamiliar with bash doesn’t mean it’s unreadable. Almost any language will look cryptic if you don’t know it. That function is mostly just parameter expansion and very common in most bash scripts. I bet if you read the manual you would easily be able to figure it out. You just have to learn the language.

mistrial9 7y ago

I have inherited about 4K LOC Bash, which mostly works as advertised. No one wants to touch it! suggestions ?

stjohnswarts 7y ago
Leave it alone if you don't need to modify it. If you start needing to modify it a lot start rewriting it in ruby, go, or python at that point. Use set -x at the top of the file to get an idea on what's going on.
tjpnz 7y ago
Assuming it's all in one file I would move related pieces of functionality into separate files and and then source them when necessary. Should make things more manageable for people only wanting to make small changes.
- mistrial9 7y ago
  no, its 27 directories and 947 files, with a plugin architecture (pipeline)
  - mistrial9 7y ago
    my mistake, the project is closer to 40,000 LOC, now that I count it...
andrewl-hn 7y ago
Skim https://learnxinyminutes.com/docs/bash/ https://learnxinyminutes.com/docs/bash/ Then open the code and just go through it. Check everything you find unfamiliar with that cheat sheet. If it doesn't cover something in your code, write it down to research separately later.

tjpnz 7y ago

Can anyone recommend something similar for Make?

vinceguidry 7y ago

I have moved to Crystal Lang for the times when i need the speed that ruby can't offer. I could have leveled up my bash skills, and was indeed using bash for that porpoise, but crystal is easier i think.

bwang008 7y ago

As someone that writes bash scripts regularly for my job, one of the big struggles I've had is remembering what I or someone else wrote when I look back 6+ months later haha. I had the same issue with reading other people's perl as well. I think the great and terrible thing about both languages is that there is literally a million different ways to skin the cat / write a regex.

taviso 7y ago

This seems like a good time to mention my (ridiculous) project, a ctypes module for bash. https://github.com/taviso/ctypes.sh/wiki https://github.com/taviso/ctypes.sh/wiki There are some little demos here: https://github.com/taviso/ctypes.sh/tree/master/test https://github.com/taviso/ctypes.sh/tree/master/test I even ported the GTK+3 Hello World to bash as a demo: https://github.com/taviso/ctypes.sh/blob/master/test/gtk.sh https://github.com/taviso/ctypes.sh/blob/master/test/gtk.sh

bokchoi 7y ago
This is so awesome. I need to play with this.
donpdonp 7y ago
"ctypes.sh is a bash plugin that provides a foreign function interface directly in your shell."
singron 7y ago
I found ctypes.sh to be legitimately useful for managing resources in nix-shell. E.g. flock a lock file and set CLOEXEC so that subprocesses don't hold the lock open after the shell exits. E.g. Use memfd_create to create a temp file and write a key. Then pass /proc/$$/fd/$FD to programs that need the key as a file. When the shell exits, the file can no longer be opened. You can do similar things with traps, but they aren't guaranteed to execute, whereas these OS primitives will always be cleaned up.
valenciarose 7y ago
This is MARVELOUSLY ridiculous and I love it. Thanks for sharing your work.

ape4 7y ago

Looks more like a "cookbook"

Sir_Cmpwn 7y ago

Please limit your shell scripting to POSIX sh for the sake of broad compatibility with current and future operating systems. There's a quick reference here: https://shellhaters.org/ https://shellhaters.org/ If you find yourself frustrated by the lack of bash extensions, your program is probably complex enough that you probably shouldn't be writing a shell script.

iandinwoodie 7y ago

Dylan, I was wondering if you could share your process for going from chapter text files to a full ebook? This looks like a really approachable way to writing a book. Is the conversion something scripted or is it a more involved process?

eatbitseveryday 7y ago

The title made me think you can query The Bible in bash...

dredmorbius 7y ago
Not bash, but CLI: https://packages.debian.org/buster/bible-kjv https://packages.debian.org/buster/bible-kjv

sigzero 7y ago

Kudos for using shellcheck. I use it all the time.