10 ms·

Surprising Bash Variables

200 points by zwischenzug 7y ago

teddyh 7y ago
As much as people use the shell, it’s regrettable how few have actually read the manual. Anything you use for extended periods of time on a regular basis is worth sitting down and reading the manual for.
- nerdponx 7y ago
  With a manual as labyrinthine and disorganized as Bash's, in my mind they are forgiven for not being able to struggle through it. It makes even the Zsh manual seem delightfully lucid by comparison.
  - dredmorbius 7y ago
    zsh accomplishes that by partitioning. Overall it's over 3x the size of the bash manpage; https://news.ycombinator.com/item?id=19892608 https://news.ycombinator.com/item?id=19892608
- zwischenzug 7y ago
  I think the bash manual page (and others) intimidates people with its concise and (to the outsider) jargon. I often describe my bash book as an attempt to carefully demystify the man page so that you can approach it with confidence.
  - teddyh 7y ago
    Instead of some proprietary book, I would suggest people read the classic Advanced Bash-Scripting Guide: http://tldp.org/guides.html#abs http://tldp.org/guides.html#abs Beginners might prefer the Bash Guide for Beginners: http://tldp.org/guides.html#bbg http://tldp.org/guides.html#bbg The Bash manual page is concise because it is a Unix manual page. Unix manual pages are supposed to be concise, because they are meant to be reference documents, not tutorials. In the GNU project, this is what the Info documentation is for. However, the bash project has elected not to maintain such a document, so the above documents from TLDP (The Linux Documentation Project) are the next best thing.
    - dredmorbius 7y ago
      That said, drop everything and get you some Jerry Peek; https://www.worldcat.org/title/from-bash-to-zshell-conquering-the-command-line/oclc/939074559&referer=brief_results https://www.worldcat.org/title/from-bash-to-zshell-conquerin... https://www.worldcat.org/title/learning-the-unix-operating-system-a-concise-guide-for-the-new-user/oclc/847931076&referer=brief_results https://www.worldcat.org/title/learning-the-unix-operating-s...
  - Sharlin 7y ago
    The bash man page is simply way too long, and IMHO not organized very clearly. And man pages don't even have a table of contents, never mind hyperlinks. The info version is naturally better in those respects, though.
- nickjj 7y ago
  I think that's a testament to how successful shell scripting is. Somehow you can be really productive and write useful tools without really ever diving deep into the docs. I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash. You can't do that with most programming languages.
  - DonHopkins 7y ago
    Some of us actually have to USE shell scripts written by other people who like to brag about only looking at the doc a couple of times, who just copy and paste things they found in outdated pages on google into hodge-podges of frankenscripts, and who then run them as root at regular intervals on unattended servers. It may be care free, fun, and effortless for you, but try being on the receiving end of that behavior some time. RTFM! And if the manual sucks (like the Bash manual), then learn to use another language. Instead of learning to write shell scripts better, people should learn NOT to write shell scripts, and learn a decent scripting language instead. You will find that most decent scripting languages like Python are actually a hell of a lot easier to learn and program and maintain than shell scripts.
    - mirkules 7y ago
      Just to add, I generally prefer shell scripts over Python because it is supported in all my use-cases out-of-the-box. However, I recently discovered the power of Python’s Fabric scripts, and holy cow... the ability to run functions on multiple computers at the same time based on roles is amazing (even if the “join” syntax is cumbersome as hell). I think in general, simple system things should be written as scripts. The moment you realize “this is going to take a LOT of scripting” it’s time to look at Python because chances are there’s a module for that.
      - nickjj 7y ago
        If you like writing things that you can execute on multiple machines take a look at Ansible. It's a Python based configuration management tool where a majority of what you write is YAML and Jinja 2 templates. But on that note, a majority of my current shell scripts are being used on my dev box to improve my day to day. Things like this: https://github.com/nickjj/invoice https://github.com/nickjj/invoice
- mistrial9 7y ago
  "the manual" is a form of communication for humans. Something nix has been de-prioritizing for thirty years! By experience, the nix culture has been "if you do not understand this, you should not be here" and a low-level hostility towards questions and learning, comes from some science gestalt in the 1800's German or something. *nix is friendly, it is just selective in who its friends are" .. is a dot-signature from that era A manual is a reading experience, a working reference system, and relies on the visual context it is presented in. There are vast differences in the quantitative and qualitative contents of manuals, on the same content! The BASH man page is .. improving ? Specifically compare an alphabetized list of every option, including "change the preferred shortcut" and "let the sea-water in, thereby killing everyone if you are underwater" .. are right next to each other .. to modern manuals in say, the Python culture. Specifically compare say, huge amounts of text supplied for some obscure option that is not at all needed, to a terse one-liner for something crucial that is used everyday. The parent comment cheerfully "blames the victim" with implied guilt for "not reading the manual" as if more time and effort on the part of the user, is the answer to the communication problem posed with such a dense and subtle realm. Last thing in this rant is, that a small interesting article on something specific is an antidote to the largely unsolvable challenge here, so good on that.
  - blattimwind 7y ago
    > a low-level hostility towards questions and learning, comes from some science gestalt in the 1800's German An oddly specific comparison.
  - detaro 7y ago
    I don't think you can generalize that. E.g. some of the BSDs pride themselves in great documentation. The shell manuals are not just alphabetical lists, but sort-of guides to their programming languages. etc.
  - nwatson 7y ago
    > The BASH man page is .. improving? When I had to start dealing with bash a lot about 5 years ago I consulted "Greg's Bash Wiki" at https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and it pretty comprehensively treats best practices for writing bash scripts. I often go back to consult the guide.
- nwellnhof 7y ago
  If you want to read the manual, start with the POSIX shell documentation: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799.2018edition... It's quite compact and helps to write portable shell scripts.
- shrimp_emoji 7y ago
  Bash is painful and awful, IMO. The last thing I want to do (if I were even capable of comprehending the man pages) is really learn it and internalize its awfulness (same with, say, Vim); I want to be able to do what I want to with some syntax quickly glued on from StackOverflow. Actually, I don't want to do that either. What I really want to do is use a Python script instead. :3
  - Pawamoy 7y ago
    But using Python is very painful as well for shell use-case. Do you really want to open files, subprocesses, and create fifo manually instead of typing "cmd1 | cmd2 > output"? Then you have Xonsh, but in my opinion, switching between the two contexts, shell and python, at each part of a command was too difficult. I was totally lost even at writing very basic commands.
- thatoneuser 7y ago
  It'd also be worth writing a human readable man page...
- dredmorbius 7y ago
  The bash manpage ... runs longish. 110 pages as 'man bash | pr". I do read it, frequently, though it's hard to assimilate in total andI constantly find new or unexpected things (after 20+ years). Checking zsh by comparison: 6 pages, though a long "SEE ALSO" list with 10 zsh-specific entries, totalling 374 pages in all: for zsh in zsh zshbuiltins zshcompwid zshcompsys zshcompctl \ zshexpn zshmisc zshmodules zshoptions zshparam zshzle; do\ man $zsh | pr | grep 'Page *[0-9][0-9]*$' | tail -1; \ done | awk '{tot+=$NF}; END{print tot}'
  - teddyh 7y ago
    The bash manual page is 78 pages as rendered to proper PostScript using man -t.
    - dredmorbius 7y ago
      I specified both result and methodology. Yes, a different method gives a different result. Using pr allows rapid verification of numerous manpages as in the zsh manpages example shown, for validation. I'm aware that generating postscript is possible, and results in fewer pages. But that doesn't readily permit comparisons with simple shell tools. (The output is decidedly pretty though.)
chrisweekly 7y ago
Cool list, couple or three were new to me. Thanks for posting!
yakattak 7y ago
HISTTIMEFORMAT is something I’ve wanted for so long but never knew existed. I should probably read man pages more!
- nickjj 7y ago
  On the flip side, I added this to my dot files quite a while ago but I didn't originally find out about it from the docs. I ran into a situation where I thought "wow it would have been useful if I knew when this command was run" while going through a massive amount of history on a foreign server. So then I Googled for "add timestamp to bash history" and immediately found it.
nerdponx 7y ago
This is similar to the problems seen when the dot folder is not included in the more familiar PATH variable FWIW I don't think I've ever seen this done in any system anywhere. My usual expectation is that local executables are run as "./foo".
- gizmo686 7y ago
  Windows does this.
noisy_boy 7y ago
Interesting list (not very surprising to me personally though saw $REPLY mentioned anywhere after a long time). One that surprised me recently (in a how-come-I-didn't-know-about-it-for-so-long way) was $PIPESTATUS e.g. if you are running a pipeline "cmd1 | cmd 2" and would like to know the return status of cmd1, you can use ${PIPESTATUS[0]} to get that. Useful, e.g. when you are tee-ing the output etc.
- teddyh 7y ago
  I would recommend against using $REPLY, for two related reasons: 1. It’s confusing to do a plain “read” and later have a magic $REPLY variable show up, seemingly from nowhere. It has shades of Perl, and I mean nothing positive with that statement. 2. Names are good. By setting a name to the thing which is read, you make the later code easier to read. It’s easier to understand “echo $filename” than “echo $REPLY”, so if you’re reading filenames, do “read filename”. (Or, rather, you probably want to do “read -r filename”, but that’s a different subject.)
  - daveFNbuck 7y ago
    I don't do a lot of bash programming, but I imagine using $REPLY also increases the danger of having the value overwritten before you use it as your code changes.
    - mar77i 7y ago
      Now that you mentioned it, I never hit the case where I would read two different times in the same function/subshell scope and would never have considered this a problem. But maybe I'm just preempting this halfway subconciously applying meticulous scoping?
  - mar77i 7y ago
    There's an actual difference in behavior between`read name` and `read`: one strips leading and trailing whitespace, the other does not. I know it because I once needed to strip whitespace and only found out through IRC; the difference is only somewhat hinted at in the manual.
DonHopkins 7y ago
>4) SHLVL: [...] This can be very useful in scripts where you’re not sure whether you should exit or not, or keeping track of where you are in a nest of scripts. If you're writing recursive shell scripts where you're not sure whether you should exit or not, you shouldn't be writing shell scripts. Leave it to bash to go out of its way to make it easier to write scripts that break encapsulation and implement spooky mysterious action-at-a-distance by implicitly depending on how they were run, and purposefully behave differently whether they're invoked from a command line or another script, and are so confused about what they should do that they have to make guesses about whether or not to exit.
- tech-hacker69 7y ago
  nope I think that's actually incorrect
- Pawamoy 7y ago
  What about a simple "die" function that echoes a message and return an error code? Could be used both in scripts and in the interactive shell. It would use "exit" when SHLVL is more than 1, and "return" otherwise.
  - DonHopkins 7y ago
    It's not about the best way to behave differently at runtime and sometimes call exit depending on how the script was invoked, it's about the idea that a scripting language would SUPPORT and ENCOURAGE writing that kind of script, with a special magic built-in variable and a section in the already-complex documentation encouraging its use. Scripts should not behave differently depending on if they were invoked from other scripts, and modular reusable libraries should never take it upon themselves to call "exit" when there's an error. The way to control the flow of recursive functions is by passing explicit parameters (like the recursion depth, or a data structure to walk, or an error callback), not reflecting on the depth of the runtime stack or looking at how the function was called. If bash is too weak to handle errors, or exceptions, or recursion, or passing functional callback parameters properly, or even file names with spaces in them, then use a better scripting language, don't just throw up your hands and exit. https://stackoverflow.com/questions/14199689/how-can-i-handle-exit-calls-in-3rd-party-library-code https://stackoverflow.com/questions/14199689/how-can-i-handl... https://github.com/texane/stlink/issues/634 https://github.com/texane/stlink/issues/634
ivanbakel 7y ago
> This is similar to the problems seen when the dot folder is not included in the more familiar PATH variable. Is this implying that you should put `.` on PATH? I've always heard that to be a security problem. What's interesting is that so many of these variables are incredibly tersely named. What's the reasoning behind not using a full name for them to make Bash actually readable? Surely not the minor performance scrape?
- johannes1234321 7y ago
  Yes, with `.` in the PATH you have potential security issues: If your working directory happens to be world-writable (i.e. while doing cleanup in /tmp) an attacker could put evil replacements into your PATH. "cd /tmp; sudo rm some_file" might execute /tmp/sudo and do really evil stuff. Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation.
  - tetha 7y ago
    > Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation. All systems are secure and sufficiently hardened until a botnet demonstrates otherwise, aren't they? And a botnet is the friendly hostile takeover because it'll just deploy a bot or a crypto miner and might even patch your system. Just don't put . on your PATH. And if you do, append it - <PATH=$PATH:.>. This way, /usr/bin/foo will win against ./foo.
  - dahart 7y ago
    Having a dot at the end of the path also trivially mitigates that particular scenario while still providing the convenience of being able to run an executable without having to always type dot-slash. I guess the reasons not to put the dot at the end are mainly about common typo attacks, but I’m partial to dot at the end.
nishparadox 7y ago
PROMPT_COMMAND ( is one of the powerful ones. I have modified my own to log the commands into timestamped based log files.
- noisy_boy 7y ago
  Setting it to show git branch information is another common use.
- DonaldPShimoda 7y ago
  I use it for printing the full cwd in my prompt (instead of the usual one abridged with tilde), as well as git branch info.
- Pawamoy 7y ago
  I use it in combination with a trap on DEBUG signal to compute information about each interactive command (start/stop timestamps, type, working dir, return code, and more) and save it in a big log file. I can then transform the data into pretty charts! It's also used by the `z` command line tool to maintain its directory "frecency" list.
mef 7y ago
@zwischenzug in case you’re not aware, visiting your site on mobile sometimes forwards on to a spammy third party ad site.
- zwischenzug 7y ago
  Thanks, have reported to Wordpress.
robinhouston 7y ago
If you set $CDPATH, then for goodness’ sake don’t export it. It changes the behaviour of the cd command to make it output the absolute path of the directory changed to, which breaks a common shell-scripting pattern for converting relative directory paths to absolute paths, viz: absolute=$(cd "$relative" && pwd) Conversely, if you’re writing a bash script and it needs to be robust against people who do export CDPATH, you can do it like this instead: absolute=$(CDPATH=. cd "$relative")
- jsjohnst 7y ago
  Or just learn the right method to do that and use `readlink` or `realpath` or similar solution.
  - robinhouston 7y ago
    That’s a very Linux-centric view. Neither of these works on BSD. It’s probably fine if you’re writing scripts that only need to run on Linux, but I wouldn’t call it the “right method”.
    - jsjohnst 7y ago
      Readlink is available on FreeBSD, OpenBSD, and NetBSD. https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1 https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1 So how is that a Linux-centric view again?
      - robinhouston 7y ago
        Sure, readlink exists. And it lets you read symlinks. But it can’t be used to convert relative to absolute directory paths, which I thought was the use you were proposing.
        jsjohnst 7y ago
        From the man page: > if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link. This is the case for *BSD, GNU/Linux, and Irix. If macOS would prioritize being consistent with the other Nix OSs, it would be supported too, but alas Apple doesn’t.
maxxxxx 7y ago
My problem with bash is that things are not very discoverable. There is so much cool stuff you find out even after ten years of using it daily. I wonder if there is a way to have all its features in a less obscure language.
- lucb1e 7y ago
  Could that be a property of command line things in general? Is that why GUIs are so popular with non-professionals that don't want to take the time to learn to use a system properly? There is so much in terms of data manipulation that you can do with basic tools, but people will open op Excel to find a graphical button they need instead of using, I don't know, maybe `man -K` or `apt search` to find the right tool. For years I've been trying to figure out what exactly the difference is. Terminals are seen as difficult and outdated, yet the more nerdy you get, the more likely it is that you end up using (and the more time you will spend in) a terminal. There has to be a reason terminals haven't replaced GUIs altogether, and GUIs haven't replaced terminals altogether, despite the near-complete overlap in things you can do with them. The only thing that really needs a mouse/touchscreen is when you have something spatial like photo editing, but conversely, the extreme end of the power user tools almost always use a terminal and can't really work with a GUI.
  - maxxxxx 7y ago
    Possible. I would say that the shell is even worse than most command line tools. I have started using PowerShell in my Mac and I find it easier to find stuff there then bash. As soon as I don’t use bash for a month I forget all the little obscure tricks.
gumby 7y ago
WARNING: this page has a dangerous typo. do not put '.' in your path. "This is similar to the confusion I felt when I realised the dot folder was not included in my more familiar PATH variable… but you should [not] do that in the PATH variable because you can get tricked into running a ‘fake’ command from some downloaded code."
- fiddlerwoaroof 7y ago
  Today, the threat model for a personal computer makes this a lot less important: to exploit this someone needs to be able to create a file on your computer and set its mode to +x: but if someone has that kind of access to one’s personal computer, it’s already game over.
  - tsbinz 7y ago
    you mean like you checking out some repository from github or you unzipping a file that you downloaded or ...?
    - fiddlerwoaroof 7y ago
      Mount those directories noexec
      - marcosdumay 7y ago
        Most people want to execute things on their homedir.
        fiddlerwoaroof 7y ago
        I usually mount a separate file system on ~/Downloads
        DonHopkins 7y ago
        You really are missing the point. https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf Unix-Haters Handbook Preface Things Are Going to Get Lot Worse Before Things Get Worse “I liken starting one’s computing career with Unix, say as an undergraduate, to being born in East Africa. It is intolerably hot, your body is covered with lice and flies, you are malnourished and you suffer from numerous curable diseases. But, as far as young East Africans can tell, this is simply the natural condition and they live within it. By the time they find out differently, it is too late. They already think that the writing of shell scripts is a natural act.” -Ken Pier, Xerox PARC
gavinpc 7y ago
"There are dark corners in the Bourne shell, and people use all of them." https://books.google.com/books?id=KJQRAwAAQBAJ&pg=PA36&lpg=PA36#v=onepage&q&f=false https://books.google.com/books?id=KJQRAwAAQBAJ&pg=PA36&lpg=P...
- teddyh 7y ago
  Why on Earth would you use a Google Books link for something in the TLDP? Here is the equivalent canonical link: http://tldp.org/LDP/abs/html/exit-status.html http://tldp.org/LDP/abs/html/exit-status.html (HTML) http://tldp.org/LDP/abs/abs-guide.pdf#page=57 http://tldp.org/LDP/abs/abs-guide.pdf#page=57 (PDF)
  - gavinpc 7y ago
    Thanks, I did search a bit and at any rate couldn't find an actual source for the quote itself.
    - teddyh 7y ago
      A simple search gave me the apparent original source for the quote; an article titled Bash — The GNU Shell, first published in the Linux Journal in 1994: https://www.linuxjournal.com/article/2800 https://www.linuxjournal.com/article/2800 The latest revision of that text seems to be available here: https://tiswww.case.edu/php/chet/bash/article.pdf https://tiswww.case.edu/php/chet/bash/article.pdf
foobarian 7y ago
Surprised to see that $RANDOM was not there. Who would have known!
andrepd 7y ago
Overall, I feel like "Surprising" is the word that best describes "Bash".
ianamartin 7y ago
Bash seems like a language written by people who never intend to do the same thing twice.
tech-hacker69 7y ago
ssuhhhhhhh
tech-hacker69 7y ago
suhhhhhhh
dorfsmay 7y ago
A lot of those variables come straight from Korn shell and are well documented in the Bolsky and Korn book. One very useful one not mentioned on this page, possibly because considered well-known: $RANDOM http://tldp.org/LDP/abs/html/randomvar.html http://tldp.org/LDP/abs/html/randomvar.html
Pawamoy 7y ago
I have to say that the LINENO variable does not have the expected behavior in process substitutions, especially when using multi-line commands (with backslashes at the end, or using multi-line strings). LINENO will use the last line number of a multi-line "$(...)" command, offsetting all the commands' line numbers within the substitution. Since multi-line strings are concatenated by Bash internally, LINENO will only increase by one for those. You end up with line numbers greater than the number of lines in your file, or overlapping with empty lines or comments. In this matter, Zsh's LINENO is way better as it will have the expected value (it will always be the line where the command was started, not an offset).
deleted 7y ago
[deleted]
zouhair 7y ago
There is also BASH_REMATCH[0] used in conjunction with the regex comparison "=~" to capture subpatern Here is a function to convert HH:MM:SS to seconds: toseconds() { # https://stackoverflow.com/a/12986612/1469043 Time="$1" while :; do if [[ $Time =~ ^([0-9]{2}):([0-9]{2}):([0-9]{2})$ ]]; then if [[ ! BASH_REMATCH[3] < 60 ]]\ && [[ ! BASH_REMATCH[2] < 60 ]]\ && [[ ! BASH_REMATCH[1] < 24 ]]; then break fi fi read -p "Wrong format. Please use the HH:MM:SS format: " -e Time done echo "$Time" | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }' } [0]: https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html https://www.gnu.org/software/bash/manual/html_node/Bash-Vari...
nibbula 7y ago
I find it surprising that the fellow that coined the word POSIX, agreed to censoring the POSIX_ME_HARDER variable. It's also surprising that the first version of bash committed source code suicide via a bug in globbing, which makes it's name seem more poignant, and was why there was a file named "-i" in it's source tree for many years.

200 points by zwischenzug 7y ago

As much as people use the shell, it’s regrettable how few have actually read the manual. Anything you use for extended periods of time on a regular basis is worth sitting down and reading the manual for.

nerdponx 7y ago
With a manual as labyrinthine and disorganized as Bash's, in my mind they are forgiven for not being able to struggle through it. It makes even the Zsh manual seem delightfully lucid by comparison.
- dredmorbius 7y ago
  zsh accomplishes that by partitioning. Overall it's over 3x the size of the bash manpage; https://news.ycombinator.com/item?id=19892608 https://news.ycombinator.com/item?id=19892608
zwischenzug 7y ago
I think the bash manual page (and others) intimidates people with its concise and (to the outsider) jargon. I often describe my bash book as an attempt to carefully demystify the man page so that you can approach it with confidence.
- teddyh 7y ago
  Instead of some proprietary book, I would suggest people read the classic Advanced Bash-Scripting Guide: http://tldp.org/guides.html#abs http://tldp.org/guides.html#abs Beginners might prefer the Bash Guide for Beginners: http://tldp.org/guides.html#bbg http://tldp.org/guides.html#bbg The Bash manual page is concise because it is a Unix manual page. Unix manual pages are supposed to be concise, because they are meant to be reference documents, not tutorials. In the GNU project, this is what the Info documentation is for. However, the bash project has elected not to maintain such a document, so the above documents from TLDP (The Linux Documentation Project) are the next best thing.
  - dredmorbius 7y ago
    That said, drop everything and get you some Jerry Peek; https://www.worldcat.org/title/from-bash-to-zshell-conquering-the-command-line/oclc/939074559&referer=brief_results https://www.worldcat.org/title/from-bash-to-zshell-conquerin... https://www.worldcat.org/title/learning-the-unix-operating-system-a-concise-guide-for-the-new-user/oclc/847931076&referer=brief_results https://www.worldcat.org/title/learning-the-unix-operating-s...
- Sharlin 7y ago
  The bash man page is simply way too long, and IMHO not organized very clearly. And man pages don't even have a table of contents, never mind hyperlinks. The info version is naturally better in those respects, though.
nickjj 7y ago
I think that's a testament to how successful shell scripting is. Somehow you can be really productive and write useful tools without really ever diving deep into the docs. I've written literally thousands of lines of shell scripts (maybe even 10k+) across dozens of scripts over the years and I've only really looked at the docs a couple of times. Of course I'm Googling various syntax things that are humanly impossible to remember, but I never once really sat there and combed through all of the docs or even read a book on Bash. You can't do that with most programming languages.
- DonHopkins 7y ago
  Some of us actually have to USE shell scripts written by other people who like to brag about only looking at the doc a couple of times, who just copy and paste things they found in outdated pages on google into hodge-podges of frankenscripts, and who then run them as root at regular intervals on unattended servers. It may be care free, fun, and effortless for you, but try being on the receiving end of that behavior some time. RTFM! And if the manual sucks (like the Bash manual), then learn to use another language. Instead of learning to write shell scripts better, people should learn NOT to write shell scripts, and learn a decent scripting language instead. You will find that most decent scripting languages like Python are actually a hell of a lot easier to learn and program and maintain than shell scripts.
  - mirkules 7y ago
    Just to add, I generally prefer shell scripts over Python because it is supported in all my use-cases out-of-the-box. However, I recently discovered the power of Python’s Fabric scripts, and holy cow... the ability to run functions on multiple computers at the same time based on roles is amazing (even if the “join” syntax is cumbersome as hell). I think in general, simple system things should be written as scripts. The moment you realize “this is going to take a LOT of scripting” it’s time to look at Python because chances are there’s a module for that.
    - nickjj 7y ago
      If you like writing things that you can execute on multiple machines take a look at Ansible. It's a Python based configuration management tool where a majority of what you write is YAML and Jinja 2 templates. But on that note, a majority of my current shell scripts are being used on my dev box to improve my day to day. Things like this: https://github.com/nickjj/invoice https://github.com/nickjj/invoice
mistrial9 7y ago
"the manual" is a form of communication for humans. Something nix has been de-prioritizing for thirty years! By experience, the nix culture has been "if you do not understand this, you should not be here" and a low-level hostility towards questions and learning, comes from some science gestalt in the 1800's German or something. *nix is friendly, it is just selective in who its friends are" .. is a dot-signature from that era A manual is a reading experience, a working reference system, and relies on the visual context it is presented in. There are vast differences in the quantitative and qualitative contents of manuals, on the same content! The BASH man page is .. improving ? Specifically compare an alphabetized list of every option, including "change the preferred shortcut" and "let the sea-water in, thereby killing everyone if you are underwater" .. are right next to each other .. to modern manuals in say, the Python culture. Specifically compare say, huge amounts of text supplied for some obscure option that is not at all needed, to a terse one-liner for something crucial that is used everyday. The parent comment cheerfully "blames the victim" with implied guilt for "not reading the manual" as if more time and effort on the part of the user, is the answer to the communication problem posed with such a dense and subtle realm. Last thing in this rant is, that a small interesting article on something specific is an antidote to the largely unsolvable challenge here, so good on that.
- blattimwind 7y ago
  > a low-level hostility towards questions and learning, comes from some science gestalt in the 1800's German An oddly specific comparison.
- detaro 7y ago
  I don't think you can generalize that. E.g. some of the BSDs pride themselves in great documentation. The shell manuals are not just alphabetical lists, but sort-of guides to their programming languages. etc.
- nwatson 7y ago
  > The BASH man page is .. improving? When I had to start dealing with bash a lot about 5 years ago I consulted "Greg's Bash Wiki" at https://mywiki.wooledge.org/BashGuide https://mywiki.wooledge.org/BashGuide and it pretty comprehensively treats best practices for writing bash scripts. I often go back to consult the guide.
nwellnhof 7y ago
If you want to read the manual, start with the POSIX shell documentation: https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799.2018edition... It's quite compact and helps to write portable shell scripts.
shrimp_emoji 7y ago
Bash is painful and awful, IMO. The last thing I want to do (if I were even capable of comprehending the man pages) is really learn it and internalize its awfulness (same with, say, Vim); I want to be able to do what I want to with some syntax quickly glued on from StackOverflow. Actually, I don't want to do that either. What I really want to do is use a Python script instead. :3
- Pawamoy 7y ago
  But using Python is very painful as well for shell use-case. Do you really want to open files, subprocesses, and create fifo manually instead of typing "cmd1 | cmd2 > output"? Then you have Xonsh, but in my opinion, switching between the two contexts, shell and python, at each part of a command was too difficult. I was totally lost even at writing very basic commands.
thatoneuser 7y ago
It'd also be worth writing a human readable man page...
dredmorbius 7y ago
The bash manpage ... runs longish. 110 pages as 'man bash | pr". I do read it, frequently, though it's hard to assimilate in total andI constantly find new or unexpected things (after 20+ years). Checking zsh by comparison: 6 pages, though a long "SEE ALSO" list with 10 zsh-specific entries, totalling 374 pages in all: for zsh in zsh zshbuiltins zshcompwid zshcompsys zshcompctl \ zshexpn zshmisc zshmodules zshoptions zshparam zshzle; do\ man $zsh | pr | grep 'Page *[0-9][0-9]*$' | tail -1; \ done | awk '{tot+=$NF}; END{print tot}'
- teddyh 7y ago
  The bash manual page is 78 pages as rendered to proper PostScript using man -t.
  - dredmorbius 7y ago
    I specified both result and methodology. Yes, a different method gives a different result. Using pr allows rapid verification of numerous manpages as in the zsh manpages example shown, for validation. I'm aware that generating postscript is possible, and results in fewer pages. But that doesn't readily permit comparisons with simple shell tools. (The output is decidedly pretty though.)

chrisweekly 7y ago

Cool list, couple or three were new to me. Thanks for posting!

yakattak 7y ago

HISTTIMEFORMAT is something I’ve wanted for so long but never knew existed. I should probably read man pages more!

nickjj 7y ago
On the flip side, I added this to my dot files quite a while ago but I didn't originally find out about it from the docs. I ran into a situation where I thought "wow it would have been useful if I knew when this command was run" while going through a massive amount of history on a foreign server. So then I Googled for "add timestamp to bash history" and immediately found it.

nerdponx 7y ago

This is similar to the problems seen when the dot folder is not included in the more familiar PATH variable FWIW I don't think I've ever seen this done in any system anywhere. My usual expectation is that local executables are run as "./foo".

gizmo686 7y ago
Windows does this.

noisy_boy 7y ago

Interesting list (not very surprising to me personally though saw $REPLY mentioned anywhere after a long time). One that surprised me recently (in a how-come-I-didn't-know-about-it-for-so-long way) was $PIPESTATUS e.g. if you are running a pipeline "cmd1 | cmd 2" and would like to know the return status of cmd1, you can use ${PIPESTATUS[0]} to get that. Useful, e.g. when you are tee-ing the output etc.

teddyh 7y ago
I would recommend against using $REPLY, for two related reasons: 1. It’s confusing to do a plain “read” and later have a magic $REPLY variable show up, seemingly from nowhere. It has shades of Perl, and I mean nothing positive with that statement. 2. Names are good. By setting a name to the thing which is read, you make the later code easier to read. It’s easier to understand “echo $filename” than “echo $REPLY”, so if you’re reading filenames, do “read filename”. (Or, rather, you probably want to do “read -r filename”, but that’s a different subject.)
- daveFNbuck 7y ago
  I don't do a lot of bash programming, but I imagine using $REPLY also increases the danger of having the value overwritten before you use it as your code changes.
  - mar77i 7y ago
    Now that you mentioned it, I never hit the case where I would read two different times in the same function/subshell scope and would never have considered this a problem. But maybe I'm just preempting this halfway subconciously applying meticulous scoping?
- mar77i 7y ago
  There's an actual difference in behavior between`read name` and `read`: one strips leading and trailing whitespace, the other does not. I know it because I once needed to strip whitespace and only found out through IRC; the difference is only somewhat hinted at in the manual.

DonHopkins 7y ago

>4) SHLVL: [...] This can be very useful in scripts where you’re not sure whether you should exit or not, or keeping track of where you are in a nest of scripts. If you're writing recursive shell scripts where you're not sure whether you should exit or not, you shouldn't be writing shell scripts. Leave it to bash to go out of its way to make it easier to write scripts that break encapsulation and implement spooky mysterious action-at-a-distance by implicitly depending on how they were run, and purposefully behave differently whether they're invoked from a command line or another script, and are so confused about what they should do that they have to make guesses about whether or not to exit.

tech-hacker69 7y ago
nope I think that's actually incorrect
Pawamoy 7y ago
What about a simple "die" function that echoes a message and return an error code? Could be used both in scripts and in the interactive shell. It would use "exit" when SHLVL is more than 1, and "return" otherwise.
- DonHopkins 7y ago
  It's not about the best way to behave differently at runtime and sometimes call exit depending on how the script was invoked, it's about the idea that a scripting language would SUPPORT and ENCOURAGE writing that kind of script, with a special magic built-in variable and a section in the already-complex documentation encouraging its use. Scripts should not behave differently depending on if they were invoked from other scripts, and modular reusable libraries should never take it upon themselves to call "exit" when there's an error. The way to control the flow of recursive functions is by passing explicit parameters (like the recursion depth, or a data structure to walk, or an error callback), not reflecting on the depth of the runtime stack or looking at how the function was called. If bash is too weak to handle errors, or exceptions, or recursion, or passing functional callback parameters properly, or even file names with spaces in them, then use a better scripting language, don't just throw up your hands and exit. https://stackoverflow.com/questions/14199689/how-can-i-handle-exit-calls-in-3rd-party-library-code https://stackoverflow.com/questions/14199689/how-can-i-handl... https://github.com/texane/stlink/issues/634 https://github.com/texane/stlink/issues/634

ivanbakel 7y ago

> This is similar to the problems seen when the dot folder is not included in the more familiar PATH variable. Is this implying that you should put `.` on PATH? I've always heard that to be a security problem. What's interesting is that so many of these variables are incredibly tersely named. What's the reasoning behind not using a full name for them to make Bash actually readable? Surely not the minor performance scrape?

johannes1234321 7y ago
Yes, with `.` in the PATH you have potential security issues: If your working directory happens to be world-writable (i.e. while doing cleanup in /tmp) an attacker could put evil replacements into your PATH. "cd /tmp; sudo rm some_file" might execute /tmp/sudo and do really evil stuff. Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation.
- tetha 7y ago
  > Sure, on an otherwise 100% secure system accessed only by trusted users there is no problem, but not having . in PATH is a trivial mitigation. All systems are secure and sufficiently hardened until a botnet demonstrates otherwise, aren't they? And a botnet is the friendly hostile takeover because it'll just deploy a bot or a crypto miner and might even patch your system. Just don't put . on your PATH. And if you do, append it - <PATH=$PATH:.>. This way, /usr/bin/foo will win against ./foo.
- dahart 7y ago
  Having a dot at the end of the path also trivially mitigates that particular scenario while still providing the convenience of being able to run an executable without having to always type dot-slash. I guess the reasons not to put the dot at the end are mainly about common typo attacks, but I’m partial to dot at the end.

nishparadox 7y ago

PROMPT_COMMAND ( is one of the powerful ones. I have modified my own to log the commands into timestamped based log files.

noisy_boy 7y ago
Setting it to show git branch information is another common use.
DonaldPShimoda 7y ago
I use it for printing the full cwd in my prompt (instead of the usual one abridged with tilde), as well as git branch info.
Pawamoy 7y ago
I use it in combination with a trap on DEBUG signal to compute information about each interactive command (start/stop timestamps, type, working dir, return code, and more) and save it in a big log file. I can then transform the data into pretty charts! It's also used by the `z` command line tool to maintain its directory "frecency" list.

mef 7y ago

@zwischenzug in case you’re not aware, visiting your site on mobile sometimes forwards on to a spammy third party ad site.

zwischenzug 7y ago
Thanks, have reported to Wordpress.

robinhouston 7y ago

If you set $CDPATH, then for goodness’ sake don’t export it. It changes the behaviour of the cd command to make it output the absolute path of the directory changed to, which breaks a common shell-scripting pattern for converting relative directory paths to absolute paths, viz: absolute=$(cd "$relative" && pwd) Conversely, if you’re writing a bash script and it needs to be robust against people who do export CDPATH, you can do it like this instead: absolute=$(CDPATH=. cd "$relative")

jsjohnst 7y ago
Or just learn the right method to do that and use `readlink` or `realpath` or similar solution.
- robinhouston 7y ago
  That’s a very Linux-centric view. Neither of these works on BSD. It’s probably fine if you’re writing scripts that only need to run on Linux, but I wouldn’t call it the “right method”.
  - jsjohnst 7y ago
    Readlink is available on FreeBSD, OpenBSD, and NetBSD. https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1 https://www.freebsd.org/cgi/man.cgi?query=readlink&sektion=1 So how is that a Linux-centric view again?
    - robinhouston 7y ago
      Sure, readlink exists. And it lets you read symlinks. But it can’t be used to convert relative to absolute directory paths, which I thought was the use you were proposing.
      - jsjohnst 7y ago
        From the man page: > if the -f option is specified, the output is canonicalized by following every symlink in every component of the given path recursively. readlink will resolve both absolute and relative paths, and return the absolute pathname corresponding to file. In this case, the argument does not need to be a symbolic link. This is the case for *BSD, GNU/Linux, and Irix. If macOS would prioritize being consistent with the other Nix OSs, it would be supported too, but alas Apple doesn’t.

maxxxxx 7y ago

My problem with bash is that things are not very discoverable. There is so much cool stuff you find out even after ten years of using it daily. I wonder if there is a way to have all its features in a less obscure language.

lucb1e 7y ago
Could that be a property of command line things in general? Is that why GUIs are so popular with non-professionals that don't want to take the time to learn to use a system properly? There is so much in terms of data manipulation that you can do with basic tools, but people will open op Excel to find a graphical button they need instead of using, I don't know, maybe `man -K` or `apt search` to find the right tool. For years I've been trying to figure out what exactly the difference is. Terminals are seen as difficult and outdated, yet the more nerdy you get, the more likely it is that you end up using (and the more time you will spend in) a terminal. There has to be a reason terminals haven't replaced GUIs altogether, and GUIs haven't replaced terminals altogether, despite the near-complete overlap in things you can do with them. The only thing that really needs a mouse/touchscreen is when you have something spatial like photo editing, but conversely, the extreme end of the power user tools almost always use a terminal and can't really work with a GUI.
- maxxxxx 7y ago
  Possible. I would say that the shell is even worse than most command line tools. I have started using PowerShell in my Mac and I find it easier to find stuff there then bash. As soon as I don’t use bash for a month I forget all the little obscure tricks.

gumby 7y ago

WARNING: this page has a dangerous typo. do not put '.' in your path. "This is similar to the confusion I felt when I realised the dot folder was not included in my more familiar PATH variable… but you should [not] do that in the PATH variable because you can get tricked into running a ‘fake’ command from some downloaded code."

fiddlerwoaroof 7y ago
Today, the threat model for a personal computer makes this a lot less important: to exploit this someone needs to be able to create a file on your computer and set its mode to +x: but if someone has that kind of access to one’s personal computer, it’s already game over.
- tsbinz 7y ago
  you mean like you checking out some repository from github or you unzipping a file that you downloaded or ...?
  - fiddlerwoaroof 7y ago
    Mount those directories noexec
    - marcosdumay 7y ago
      Most people want to execute things on their homedir.
      - fiddlerwoaroof 7y ago
        I usually mount a separate file system on ~/Downloads
        DonHopkins 7y ago
        You really are missing the point. https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf Unix-Haters Handbook Preface Things Are Going to Get Lot Worse Before Things Get Worse “I liken starting one’s computing career with Unix, say as an undergraduate, to being born in East Africa. It is intolerably hot, your body is covered with lice and flies, you are malnourished and you suffer from numerous curable diseases. But, as far as young East Africans can tell, this is simply the natural condition and they live within it. By the time they find out differently, it is too late. They already think that the writing of shell scripts is a natural act.” -Ken Pier, Xerox PARC

gavinpc 7y ago

"There are dark corners in the Bourne shell, and people use all of them." https://books.google.com/books?id=KJQRAwAAQBAJ&pg=PA36&lpg=PA36#v=onepage&q&f=false https://books.google.com/books?id=KJQRAwAAQBAJ&pg=PA36&lpg=P...

teddyh 7y ago
Why on Earth would you use a Google Books link for something in the TLDP? Here is the equivalent canonical link: http://tldp.org/LDP/abs/html/exit-status.html http://tldp.org/LDP/abs/html/exit-status.html (HTML) http://tldp.org/LDP/abs/abs-guide.pdf#page=57 http://tldp.org/LDP/abs/abs-guide.pdf#page=57 (PDF)
- gavinpc 7y ago
  Thanks, I did search a bit and at any rate couldn't find an actual source for the quote itself.
  - teddyh 7y ago
    A simple search gave me the apparent original source for the quote; an article titled Bash — The GNU Shell, first published in the Linux Journal in 1994: https://www.linuxjournal.com/article/2800 https://www.linuxjournal.com/article/2800 The latest revision of that text seems to be available here: https://tiswww.case.edu/php/chet/bash/article.pdf https://tiswww.case.edu/php/chet/bash/article.pdf

foobarian 7y ago

Surprised to see that $RANDOM was not there. Who would have known!

andrepd 7y ago

Overall, I feel like "Surprising" is the word that best describes "Bash".

ianamartin 7y ago

Bash seems like a language written by people who never intend to do the same thing twice.

tech-hacker69 7y ago

ssuhhhhhhh

tech-hacker69 7y ago

suhhhhhhh

dorfsmay 7y ago

A lot of those variables come straight from Korn shell and are well documented in the Bolsky and Korn book. One very useful one not mentioned on this page, possibly because considered well-known: $RANDOM http://tldp.org/LDP/abs/html/randomvar.html http://tldp.org/LDP/abs/html/randomvar.html

Pawamoy 7y ago

I have to say that the LINENO variable does not have the expected behavior in process substitutions, especially when using multi-line commands (with backslashes at the end, or using multi-line strings). LINENO will use the last line number of a multi-line "$(...)" command, offsetting all the commands' line numbers within the substitution. Since multi-line strings are concatenated by Bash internally, LINENO will only increase by one for those. You end up with line numbers greater than the number of lines in your file, or overlapping with empty lines or comments. In this matter, Zsh's LINENO is way better as it will have the expected value (it will always be the line where the command was started, not an offset).

deleted 7y ago

[deleted]

zouhair 7y ago

There is also BASH_REMATCH[0] used in conjunction with the regex comparison "=~" to capture subpatern Here is a function to convert HH:MM:SS to seconds: toseconds() { # https://stackoverflow.com/a/12986612/1469043 Time="$1" while :; do if [[ $Time =~ ^([0-9]{2}):([0-9]{2}):([0-9]{2})$ ]]; then if [[ ! BASH_REMATCH[3] < 60 ]]\ && [[ ! BASH_REMATCH[2] < 60 ]]\ && [[ ! BASH_REMATCH[1] < 24 ]]; then break fi fi read -p "Wrong format. Please use the HH:MM:SS format: " -e Time done echo "$Time" | awk -F: '{ print ($1 * 3600) + ($2 * 60) + $3 }' } [0]: https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html https://www.gnu.org/software/bash/manual/html_node/Bash-Vari...

nibbula 7y ago

I find it surprising that the fellow that coined the word POSIX, agreed to censoring the POSIX_ME_HARDER variable. It's also surprising that the first version of bash committed source code suicide via a bug in globbing, which makes it's name seem more poignant, and was why there was a file named "-i" in it's source tree for many years.