9 ms·

Pure Bash Bible (2018)

191 points by softwaredoug 3y ago

asicsp 3y ago
Previous discussions: https://news.ycombinator.com/item?id=30363506 https://news.ycombinator.com/item?id=30363506 (291 points | Feb 16, 2022 | 96 comments) https://news.ycombinator.com/item?id=21013150 https://news.ycombinator.com/item?id=21013150 (746 points | Sept 19, 2019 | 248 comments) See also: Pure Sh Bible https://news.ycombinator.com/item?id=35765707 https://news.ycombinator.com/item?id=35765707 (406 points | 3 months ago | 133 comments)
1vuio0pswjnm7 3y ago
dash is significantly faster than bash and lighter weight, too. I added tab completion and use dash as interactive shell as well as scripting shell.
- LucasLarson 3y ago
  The same amazing guy also published the Pure sh Bible: https://github.com/dylanaraps/pure-sh-bible https://github.com/dylanaraps/pure-sh-bible
  - 1vuio0pswjnm7 3y ago
    That's not dash, it's bash in POSIX compatibility mode. dash does not implement all the POSIX stuff in this "bible". Imagine using this "bible" to write a "pure sh" script thinking "this will be highly portable", only to have it fail because the Linux scripting shell is not bash in POSIX compatibility mode, it's dash. It would fail on NetBSD, too, whose scripting shell is the version of ash from which dash is derived.
    - prmoustache 3y ago
      Do people actually symlink /bin/dash to /bin/sh? That would be extremely stupid.
      - balou23 3y ago
        Debian and Ubuntu do it by default
      - throw0101b 3y ago
        > That would be extremely stupid. Why?
        prmoustache 3y ago
        Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell. I have no problem with people using #!/bin/dash shebang for their scripts and using dash as their default shell but it shouldn't pass itself as something else. It is fine with bash because it is bourne shell compatible.
        throw0101b 3y ago
        > Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell. The same can be said about bash. > It is fine with bash because it is bourne shell compatible. I was running an HPC cluster when we upgraded Debian during the bash->dash change, and there were all sorts of problems: IIRC most folks simply changed their scripts from /bin/sh to /bin/bash.
    - happymellon 3y ago
      > dash does not implement all the POSIX stuff Then it doesn't sound fit for scripting.
    - Crestwave 3y ago
      Do you have any specific examples to cite? The snippets are meant to be run in POSIX-compliant shells like dash/yash/ash/etc, not just Bash in compatibility mode. In fact, there used to be a section which listed workarounds[1] for bugs in dash, but they have since been fixed[2]. If you are still using an old version of dash, you may need to use them. [1] https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb2b0d54cab1b1d4db3804c654db9619a https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb... [2] https://github.com/dylanaraps/pure-sh-bible/issues/13 https://github.com/dylanaraps/pure-sh-bible/issues/13
      - 1vuio0pswjnm7 3y ago
        https://news.ycombinator.com/item?id=35768615 https://news.ycombinator.com/item?id=35768615 Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them. What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I need portability. I do not use bash for scripting. It's larger and slower. One source I consult is https://www.in-ulm.de/~mascheck/bourne/ https://www.in-ulm.de/~mascheck/bourne/ It is sometimes submitted to HN, but not IMO enough https://news.ycombinator.com/from?site=in-ulm.de https://news.ycombinator.com/from?site=in-ulm.de I'm using the latest version of dash. I just tested it and it does have these ternary operators. However, I'm not going to use them in scripts unless I know they'll work on older dash versions, OpenWRT sh, Android sh, NetBSD sh, FreeBSD sh, etc. The process of updating these shells is generally very slow. It can take years. "Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but perhaps it is something else.
        ftaghn 3y ago
        > and slower. I like to write posix sh scripts for the sake of portability and, funnily enough, future proofing as I don't like having to maintain stuff against changes that break compatibility, which is something bash does (archlinux is still on an older bash even though debian stable has the latest, because bash 5.2 broke some of archlinux's own scripts. This is why you should not write bash scripts.), but bash being slow isn't one of the good reasons. If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby. I'd suggest perl, if only because unlike the latter, perl doesn't constantly pull the rug under you and make you do busy work to make sure your scripts work on the latest runtimes.
    - throw0101b 3y ago
      > That's not dash, it's bash in POSIX compatibility mode. I was under the impression that bash's POSIX mode was 'leaky': that a lot of non-POSIX stuff/extensions are available even though it's called as /bin/sh. I know this caused a lot of our users issues when we upgraded Debian/Ubuntu during its switchover to dash. We told folks you could either rewrite those parts of the code or call /bin/bash. * https://wiki.ubuntu.com/DashAsBinSh https://wiki.ubuntu.com/DashAsBinSh * https://lwn.net/Articles/343924/ https://lwn.net/Articles/343924/
      - chasil 3y ago
        One major reason that Debian chose dash is plainly written on the bash manual page: $ man bash | sed -n '/BUGS/,/^$/p' BUGS It's too big and too slow. Another major reason is adherence to the POSIX standard; bash was written a decade prior to standardization, and has interesting issues because of this. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
        throw0101b 3y ago
        > bash was written a decade prior to standardization, and has interesting issues because of this. bash can do whatever it wants to do if it's called as "bash". But it should only do specific things when called as "sh". This is true of any shell: > When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh. > Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error. * https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-in-macos-catalina-and-beyond/ https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-... * https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compatibility https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compa... POSIX.2: Shell and Utilities (IEEE Std 1003.2-1992) is a few decades old now: regardless of when bash development started, I would think that by the 2010s it would have managed to better compartmentalize compatibility and extensions.
- deleted 3y ago
  [deleted]
- rollcat 3y ago
  I would argue reaching for the shell to do your scripting is never a good idea, unless your problem is trivial to begin with (e.g. fits in less than a hundred lines of extremely readable and straightforward code). My rule of thumb is: "can I fit it in a one-liner?" - then I refactor it until it's readable, or otherwise use something else. Once you need to do fancy array manipulation, hash tables, data structures, etc you're very likely much better off with Python, Perl, Ruby, Lua, Tcl, JS, or literally anything else. Notably, one or both of Python and Perl are available out of the box on Debian (and derivatives), macOS, most (all?) BSDs, and so on. Targeting Python 3.6+ will probably cover 99%+ of the systems you can find in the wild (e.g. Ubuntu 18.04 and newer). I have also had some minor success using Go for that purpose. The toolchain can trivially cross-compile executables for ca 40 different OS/arch combinations; you need to consider the os/arch if you're running something different than Linux/x86-64 but that will again likely put you in the 1%. I'm working on some tooling to make this approach more straightforward.
  - cutler 3y ago
    For 1-liners there are only 2 real non-shell contenders - Perl and Ruby. Yes you can do them with Python but it ain't pretty.
    - rollcat 3y ago
      One-liners are meant to be thrown away, so it does not matter if it's pretty or not. If you care enough to keep it, you should consider rewriting it for clarity. That last step does not require using any particular language; e.g. my PS1 started off as an increasingly convoluted one-liner, until I rewrote it in Go.
      - cutler 3y ago
        PS1 in Go? Overkill, doesn't come close.
        rollcat 3y ago
        Depends on what you're trying to do. If you're shelling out to git(1) or docker(1), rather than e.g. recursively checking for the presence of .git in parent directories, or inspecting ~/.docker/config.json, then the fork+exec overhead is already quite significant. Next if you're parsing ~/.docker/config.json in shell, you're most likely either asking for trouble or (again) shelling out to jq. Writing it all in an interpreted language means you're paying the cost of interpreter startup, which on underpowered systems can take hundreds of milliseconds even when idle. OTOH loading a static binary to memory happens only once, and with Go you can trivially cross-compile. I also have a fallback shell one-liner, without any of the fanciness like displaying the current git branch: https://github.com/rollcat/etc/tree/master/cmd/prompter#i-like-the-prompt-but-your-code-sucks https://github.com/rollcat/etc/tree/master/cmd/prompter#i-li...
- 1vuio0pswjnm7 3y ago
  The scripts I write are relatively short and simple. Another reason I have no interest in bash. I have seen some enormous bash scripts.
somecommit 3y ago
Less critical then yesterday because of ChatGPT, I always expected to become fluent in bash at some point, but the AI became too good at writing my scripts now... Also you can use the chat as a learning tool, it's really a game changer for whom want to really learn it.
- youngNed 3y ago
  as someone who is late to this party, can you tell me how you are using it? In an IDE? what sort of way are you using it to learn rather than just give answers? thx.
  - somecommit 3y ago
    Just use the website. You can explain (in english or even native language if relevant) your algorithm and for small functions it's giving you correct answer. You still need to concatenate them. For learning in my case I needed to learn python and I'm already expert in javascript for example. So you can ask the prompt for the equivalent in python every time you feel stuck. But using the chat is so more precise and quick, than google for this kind of questions. Also without all the noise of google. In particular without this "noise" and time and concentration to find the response, it's a game changer when you really want to just learn and memorize.
- BossingAround 3y ago
  I think this is a trap for engineers inexperienced with Bash. You can (and should of course) use ChatGPT to be more productive in writing bash scripts, but as was said in the previous discussions, bash contains so many traps which you won't be able to spot. Bash's dangers might be exacerbated by using ChatGPT, since you'll rely on the tool and you might not be able to understand what's wrong if shit hits the fan. I'd recommend not using bash for anything other than the simplest tasks, as per Google's Shell style guide [0]. [0] https://google.github.io/styleguide/shellguide.html#s1.2-when-to-use-shell https://google.github.io/styleguide/shellguide.html#s1.2-whe...
- gabrielgio 3y ago
  What could go wrong? Use a technology that is notorious for outputting wrong information to write a script that is really hard to get it right and then later reviewed by someone that can’t really understand it. At this point just use something else.
lnxg33k1 3y ago
I bought it just few days ago find it helpful but also i wish there was more context / explanation around some black magic syntax
- dredmorbius 3y ago
  Definitely this. If only to note, say, the section(s) of the Bash manpage or GNU Info Manual in which topics are discussed. Taking the first-listed example, the '%%', '##', etc., substitutions are covered in Parameter Expansion. I tend to go first to manpages, though I'm well aware that GNU deprecates these (exceedingly unwisely IMO) in favour of the Info Doc format (and its dedicated single-purpose reader). I'll note that the manpage alone renders to 120 pages (man bash | pr | less) onscreen, which is fairly formidable. The PDF-formatted GNU bash manual (a fine format for a reference of this sort) runs 196 pages: <https://www.gnu.org/software/bash/manual/bash.pdf https://www.gnu.org/software/bash/manual/bash.pdf>
  - ftaghn 3y ago
    People all too often forget that there is often a lot of extra documentation hidden in /usr/share/doc/ when you install -doc packages, at least on deb systems, and not just info and manpages. For example, the bash manual you cite is in /usr/share/doc/bash/bashref.pdf along with its html version as bashref.html. There is also an /example/ folder filled with commented programs and functions to serve as reference. And a README that suggests looking for "Advanced Bash-Scripting Guide" which is a massive online book filled with all sorts of esoteric knowledge. You can also find things like the release changelogs and one that details incompatibilities from previous versions. While there are exceptions, most packages have useful goodies in there to check first before one has to think about looking things up online.
    - dredmorbius 3y ago
      On Debian, install the dwww package (<https://packages.debian.org/bookworm/dwww https://packages.debian.org/bookworm/dwww>), and point your browser at <https://localhost/dwww/ https://localhost/dwww/>. What you'll find is both a set of pages & hierarchy of documentation as well as search (via swish++) through that space. And yes, there is an absolute trove of information available: - Manpages - Info pages - Default package information - Supplementary package documents, if installed - HOWTOs from the Linux Documentation Project. - Debian-specific documentation (installation, administration, security, policy, and other manuals) - RFCs (divided into several groups, for manageability / relevance) - Resources such as the Linux Gazette There's also a commandline utility, dwww, which will invoke your browser of choice. Point that at a terminal-based browser (w3m, lynx, links, elinks, etc.) and you can quickly scan through docs from your terminal.
bhaak 3y ago
Depending on your job description it is completely fine to not write shell scripts. For anything bigger than a screenful of lines or something that would need proper unit testing or something that other people who are likely to be less versed in shell scripting need to maintain, think twice about using shell scripts. Unixoid machines can use anything as scripts and using a language that breathes the programming language designs of the 1970s is not the best choice.
- teddyh 3y ago
  > or something that would proper unit testing <https://github.com/kward/shunit2 https://github.com/kward/shunit2>
  - nickcox 3y ago
    And several others: https://github.com/dodie/testing-in-bash https://github.com/dodie/testing-in-bash
- softwaredoug 3y ago
  I typically agree, but I don't think its as simple as a "screenful of lines" criteria. I find it useful when most of your work is wiring other command line tools together. When that's most of your work, shell scripting feels like it has the best affordances. Anything that gets too fancy beyond this, with anything that resembles an algorithm or computation, you'll feel real pain :) Bash doesn't even support floating point!
  - nextaccountic 3y ago
    > Bash doesn't even support floating point! But bc (and dc) does, and shell authors deemed this was good enough. And that's part of the problem with the "Unix philosophy"
    - softwaredoug 3y ago
      Yeah every invocation of bc is a call to a command. So if you do it in a loop, its going to be very slow. I had to do this once. It was so slow, I decided to use integers in kind of fixed-point math where I just multiply all the floats by 100 and work with ints. zsh supports float though
robbywashere_ 3y ago
I'd love to see a little language which compiles to _readable_ bash. Could it be so hard? There's a dead project that comes up now and then called bish. https://github.com/tdenniston/bish https://github.com/tdenniston/bish
- osmsucks 3y ago
  One of my current hobby projects is a scripting language which compiles to POSIX sh (for maximum portability). However, it's far from complete, the output is not exactly readable and POSIX sh is so limited that I might need to change course and target something else. EDIT: I didn't know of Bish, but from its code samples we seem to have converged on similar syntax :)
- nerdponx 3y ago
  Maybe this is a bit of a silly question, but what's the point? I do think it would be a really fun project. But is there really a practical use for it? Bash is portable but not that portable if you want to care about version compatibility. I understand that it's not always possible to install a different interpreted language like Perl or Python. Maybe the use case is for platforms where you can't install other language interpreters and can't cross-compile a compiled language? It seems like a very specific situation.
- scns 3y ago
  Not what you asked for but there is Babashka for scripting in Clojure. https://github.com/babashka/babashka https://github.com/babashka/babashka
- Shawnj2 3y ago
  Write the script in pseudo python and ask ChatGPT to convert it into bash but leave a lot of comments
deleted 3y ago
[deleted]
javier_e06 3y ago
After years writing bash scripts I stumbled with this line... set -e Which forces the script to exit on first error. This line be in the bible as Genesis 1:1
- koolba 3y ago
  Just wait till you learn about: set -euo pipefail That should be line two of every bash script.
  - worldsayshi 3y ago
    The more I use bash the less I want to use it. So much UX weirdness. I feel like making the complaints people usually have with JavaScript. I like JavaScript. Because I'm good enough at it to ignore the bad parts. If I get good enough at bash, will it get better?
    - koolba 3y ago
      > If I get good enough at bash, will it get better? Bash itself will not get better because you'll be stuck writing scripts targeting some ancient version to ensure wider compatibility. It's still a powerful skill to develop. It's a sweet spot between a one-off shell pipeline and something that involves actual logic. Bash scripts should not be used for their logic. Think of them as a way to document a series of commands that you do not want to manually type. That can be including long argument names, generating includes or classpaths, or generating log file names based on the current time. A good rule of thumb is if your script involves any if-statement other than parameter validation or basic on-success-of-x-do-y, it probably should be in a real language.
  - scns 3y ago
    Used it in every script after learning about it. Might be great if it could be set globally for all Bash scripts.
  - chasil 3y ago
    POSIX shells only support: set -eu You might want to be in the dash POSIX shell for raw speed and portability.
kjuulh 3y ago
I like bash, zsh and other shell languages as well. However, unless you're really experienced, they shouldn't be the product you ship, whether that is internally or externally to you or your company. Shell languages is the final layer of configuration and customization you apply to the setup, it should almost never be the foundation of what you build. Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok. The way we develop infrastructure automation, ci, and developer tools is binaries first and foremost, then plugins for our infrastructure (also binaries, or libraries) and then some zsh on top to provide a nice developer experience. But we've had to heavily scale back our use of zsh, it simply turned into a ball of mud, like all software that isn't maintainable, upgrade-able, or simple enough to parse. This isn't a dig on zsh, bash etc, we've just found out that we can't maintain zsh as a product to supply to our engineers, and as such have to correct.
- nerdponx 3y ago
  At least you've discovered Zsh and hopefully its significant improvements over Bash. The only Bash benefit is readarray/mapfile, Z Shell is better in just about every other way.
  - JNRowe 3y ago
    Unless I'm misreading your comment, zsh does provide similar functionality via zsh/mapfile¹. It, and a lot of other cool stuff, is documented in zshmodules(1). ¹ https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-zsh_002fmapfile-Module https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-...
    - nerdponx 3y ago
      I believe Bash readarray/mapfile is more like doing this in Zsh: lines=( "${(@f)$(my_command)}" ) or lines=( "${(@f)$(<my_file)}" ) which IMO is a little less nice than using a builtin with a name like "readarray". But this is cool too.
- 3y ago
anagri 3y ago
With LLMs/ChatGPT, i would prefer fine-tuning a model and have it auto-generate bash commands with my natural language description. I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2. Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right? 1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands. 2. They can generate data by running commands, and train themselves using this simulated data. What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?
- droopyEyelids 3y ago
  There's a middle ground alternative where the LLM can read the source, give you a high level description (aka documentation), plus cite the relevant parts of the code. Fingers crossed.
jjice 3y ago
I always hear people talk about writing portable, POSIX compliant shell scripts so they'll work in more environments (assuming those environments are needed, of course), but how often do you run into an environment where you don't have a fairly complete version of Bash available? Genuine open question, I'd be really interested to know what that situation is. Lean container images, embedded, older systems? I'm really curious.
- rascul 3y ago
  I put a recent bash everywhere I need to work. Not everyone can, though.
- pletnes 3y ago
  Containers/docker/kubernetes and embedded systems that use e.g busybox is what I’ve seen. Also macOS (used to?) ship ancient bash versions, but there’s recent zsh so that’s mostly fine.
  - js2 3y ago
    Yes, macOS is stuck with bash 3 due to the license change with bash 4 and now ships with zsh as the default shell.
- yubiox 3y ago
  AIX, Solaris, HP-UX
- nrclark 3y ago
  It's really common for embedded systems to ship without Bash. I don't include it in the production firmware on any systems that I design.
- ndsipa_pomu 3y ago
  Network equipment can often use a cut down version of Linux or flavour of BSD and they don't always include BASH
- inferiorhuman 3y ago
da39a3ee 3y ago
It's so tempting to implement in bash. I made the mistake _again_ this week! It's so seductive to be able to start subprocesses with no ceremony, use output redirection, use jq, etc. If you have even the slightest hesitation regarding whether the task in front of you would be better done in a real programming language, the answer is do it in a real programming language.
Kiala 3y ago
Apparently, this is not an interactive version of the Bible written in Bash.
E39M5S62 3y ago
A lot of what's in the Pure Bash Bible is horrifically slow. Many of those things are substantially faster, even when paying the cost of starting a new process, when you use an external and commonly available tool. I wrote a bash performance profiler that outputs data in a format that flamegraph.pl recognizes - it really helped identify where we could improve the performance of ZFSBootMenu. https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0 https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0 Don't fall in the trap of thinking things have to be written entirely in bash; it's okay to use other tools to help fill in the gaps.

191 points by softwaredoug 3y ago

Previous discussions: https://news.ycombinator.com/item?id=30363506 https://news.ycombinator.com/item?id=30363506 (291 points | Feb 16, 2022 | 96 comments) https://news.ycombinator.com/item?id=21013150 https://news.ycombinator.com/item?id=21013150 (746 points | Sept 19, 2019 | 248 comments) See also: Pure Sh Bible https://news.ycombinator.com/item?id=35765707 https://news.ycombinator.com/item?id=35765707 (406 points | 3 months ago | 133 comments)

1vuio0pswjnm7 3y ago

dash is significantly faster than bash and lighter weight, too. I added tab completion and use dash as interactive shell as well as scripting shell.

LucasLarson 3y ago
The same amazing guy also published the Pure sh Bible: https://github.com/dylanaraps/pure-sh-bible https://github.com/dylanaraps/pure-sh-bible
- 1vuio0pswjnm7 3y ago
  That's not dash, it's bash in POSIX compatibility mode. dash does not implement all the POSIX stuff in this "bible". Imagine using this "bible" to write a "pure sh" script thinking "this will be highly portable", only to have it fail because the Linux scripting shell is not bash in POSIX compatibility mode, it's dash. It would fail on NetBSD, too, whose scripting shell is the version of ash from which dash is derived.
  - prmoustache 3y ago
    Do people actually symlink /bin/dash to /bin/sh? That would be extremely stupid.
    - balou23 3y ago
      Debian and Ubuntu do it by default
    - throw0101b 3y ago
      > That would be extremely stupid. Why?
      - prmoustache 3y ago
        Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell. I have no problem with people using #!/bin/dash shebang for their scripts and using dash as their default shell but it shouldn't pass itself as something else. It is fine with bash because it is bourne shell compatible.
        throw0101b 3y ago
        > Because if you are not 100% compatible with bourne shell you shouldn't automatically execute script made for bourne shell. The same can be said about bash. > It is fine with bash because it is bourne shell compatible. I was running an HPC cluster when we upgraded Debian during the bash->dash change, and there were all sorts of problems: IIRC most folks simply changed their scripts from /bin/sh to /bin/bash.
  - happymellon 3y ago
    > dash does not implement all the POSIX stuff Then it doesn't sound fit for scripting.
  - Crestwave 3y ago
    Do you have any specific examples to cite? The snippets are meant to be run in POSIX-compliant shells like dash/yash/ash/etc, not just Bash in compatibility mode. In fact, there used to be a section which listed workarounds[1] for bugs in dash, but they have since been fixed[2]. If you are still using an old version of dash, you may need to use them. [1] https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb2b0d54cab1b1d4db3804c654db9619a https://github.com/dylanaraps/pure-sh-bible/commit/70f410ebb... [2] https://github.com/dylanaraps/pure-sh-bible/issues/13 https://github.com/dylanaraps/pure-sh-bible/issues/13
    - 1vuio0pswjnm7 3y ago
      https://news.ycombinator.com/item?id=35768615 https://news.ycombinator.com/item?id=35768615 Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them. What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I need portability. I do not use bash for scripting. It's larger and slower. One source I consult is https://www.in-ulm.de/~mascheck/bourne/ https://www.in-ulm.de/~mascheck/bourne/ It is sometimes submitted to HN, but not IMO enough https://news.ycombinator.com/from?site=in-ulm.de https://news.ycombinator.com/from?site=in-ulm.de I'm using the latest version of dash. I just tested it and it does have these ternary operators. However, I'm not going to use them in scripts unless I know they'll work on older dash versions, OpenWRT sh, Android sh, NetBSD sh, FreeBSD sh, etc. The process of updating these shells is generally very slow. It can take years. "Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but perhaps it is something else.
      - ftaghn 3y ago
        > and slower. I like to write posix sh scripts for the sake of portability and, funnily enough, future proofing as I don't like having to maintain stuff against changes that break compatibility, which is something bash does (archlinux is still on an older bash even though debian stable has the latest, because bash 5.2 broke some of archlinux's own scripts. This is why you should not write bash scripts.), but bash being slow isn't one of the good reasons. If your scripts do that much work that your shell's speed matters, you should reconsider writing shell scripts and start thinking about using something like perl, python or ruby. I'd suggest perl, if only because unlike the latter, perl doesn't constantly pull the rug under you and make you do busy work to make sure your scripts work on the latest runtimes.
  - throw0101b 3y ago
    > That's not dash, it's bash in POSIX compatibility mode. I was under the impression that bash's POSIX mode was 'leaky': that a lot of non-POSIX stuff/extensions are available even though it's called as /bin/sh. I know this caused a lot of our users issues when we upgraded Debian/Ubuntu during its switchover to dash. We told folks you could either rewrite those parts of the code or call /bin/bash. * https://wiki.ubuntu.com/DashAsBinSh https://wiki.ubuntu.com/DashAsBinSh * https://lwn.net/Articles/343924/ https://lwn.net/Articles/343924/
    - chasil 3y ago
      One major reason that Debian chose dash is plainly written on the bash manual page: $ man bash | sed -n '/BUGS/,/^$/p' BUGS It's too big and too slow. Another major reason is adherence to the POSIX standard; bash was written a decade prior to standardization, and has interesting issues because of this. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
      - throw0101b 3y ago
        > bash was written a decade prior to standardization, and has interesting issues because of this. bash can do whatever it wants to do if it's called as "bash". But it should only do specific things when called as "sh". This is true of any shell: > When interpreted with dash instead of bash, the same script will fail. This is because dash is much stricter than bash in following the sh standard. Since dash is designed as a minimal implementation of the sh standard, it has to be stricter. The double brackets [[ … ]] are a ‘bashism,’ or a feature only available in bash and other, later shells such as ksh and zsh. > Even though zsh also interprets most of these bashisms, zsh in sh compatibility mode is also stricter than bash and will error. * https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-in-macos-catalina-and-beyond/ https://scriptingosx.com/2020/06/about-bash-zsh-sh-and-dash-... * https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compatibility https://zsh.sourceforge.io/Doc/Release/Invocation.html#Compa... POSIX.2: Shell and Utilities (IEEE Std 1003.2-1992) is a few decades old now: regardless of when bash development started, I would think that by the 2010s it would have managed to better compartmentalize compatibility and extensions.
deleted 3y ago
[deleted]
rollcat 3y ago
I would argue reaching for the shell to do your scripting is never a good idea, unless your problem is trivial to begin with (e.g. fits in less than a hundred lines of extremely readable and straightforward code). My rule of thumb is: "can I fit it in a one-liner?" - then I refactor it until it's readable, or otherwise use something else. Once you need to do fancy array manipulation, hash tables, data structures, etc you're very likely much better off with Python, Perl, Ruby, Lua, Tcl, JS, or literally anything else. Notably, one or both of Python and Perl are available out of the box on Debian (and derivatives), macOS, most (all?) BSDs, and so on. Targeting Python 3.6+ will probably cover 99%+ of the systems you can find in the wild (e.g. Ubuntu 18.04 and newer). I have also had some minor success using Go for that purpose. The toolchain can trivially cross-compile executables for ca 40 different OS/arch combinations; you need to consider the os/arch if you're running something different than Linux/x86-64 but that will again likely put you in the 1%. I'm working on some tooling to make this approach more straightforward.
- cutler 3y ago
  For 1-liners there are only 2 real non-shell contenders - Perl and Ruby. Yes you can do them with Python but it ain't pretty.
  - rollcat 3y ago
    One-liners are meant to be thrown away, so it does not matter if it's pretty or not. If you care enough to keep it, you should consider rewriting it for clarity. That last step does not require using any particular language; e.g. my PS1 started off as an increasingly convoluted one-liner, until I rewrote it in Go.
    - cutler 3y ago
      PS1 in Go? Overkill, doesn't come close.
      - rollcat 3y ago
        Depends on what you're trying to do. If you're shelling out to git(1) or docker(1), rather than e.g. recursively checking for the presence of .git in parent directories, or inspecting ~/.docker/config.json, then the fork+exec overhead is already quite significant. Next if you're parsing ~/.docker/config.json in shell, you're most likely either asking for trouble or (again) shelling out to jq. Writing it all in an interpreted language means you're paying the cost of interpreter startup, which on underpowered systems can take hundreds of milliseconds even when idle. OTOH loading a static binary to memory happens only once, and with Go you can trivially cross-compile. I also have a fallback shell one-liner, without any of the fanciness like displaying the current git branch: https://github.com/rollcat/etc/tree/master/cmd/prompter#i-like-the-prompt-but-your-code-sucks https://github.com/rollcat/etc/tree/master/cmd/prompter#i-li...
1vuio0pswjnm7 3y ago
The scripts I write are relatively short and simple. Another reason I have no interest in bash. I have seen some enormous bash scripts.

somecommit 3y ago

Less critical then yesterday because of ChatGPT, I always expected to become fluent in bash at some point, but the AI became too good at writing my scripts now... Also you can use the chat as a learning tool, it's really a game changer for whom want to really learn it.

youngNed 3y ago
as someone who is late to this party, can you tell me how you are using it? In an IDE? what sort of way are you using it to learn rather than just give answers? thx.
- somecommit 3y ago
  Just use the website. You can explain (in english or even native language if relevant) your algorithm and for small functions it's giving you correct answer. You still need to concatenate them. For learning in my case I needed to learn python and I'm already expert in javascript for example. So you can ask the prompt for the equivalent in python every time you feel stuck. But using the chat is so more precise and quick, than google for this kind of questions. Also without all the noise of google. In particular without this "noise" and time and concentration to find the response, it's a game changer when you really want to just learn and memorize.
BossingAround 3y ago
I think this is a trap for engineers inexperienced with Bash. You can (and should of course) use ChatGPT to be more productive in writing bash scripts, but as was said in the previous discussions, bash contains so many traps which you won't be able to spot. Bash's dangers might be exacerbated by using ChatGPT, since you'll rely on the tool and you might not be able to understand what's wrong if shit hits the fan. I'd recommend not using bash for anything other than the simplest tasks, as per Google's Shell style guide [0]. [0] https://google.github.io/styleguide/shellguide.html#s1.2-when-to-use-shell https://google.github.io/styleguide/shellguide.html#s1.2-whe...
gabrielgio 3y ago
What could go wrong? Use a technology that is notorious for outputting wrong information to write a script that is really hard to get it right and then later reviewed by someone that can’t really understand it. At this point just use something else.

lnxg33k1 3y ago

I bought it just few days ago find it helpful but also i wish there was more context / explanation around some black magic syntax

dredmorbius 3y ago
Definitely this. If only to note, say, the section(s) of the Bash manpage or GNU Info Manual in which topics are discussed. Taking the first-listed example, the '%%', '##', etc., substitutions are covered in Parameter Expansion. I tend to go first to manpages, though I'm well aware that GNU deprecates these (exceedingly unwisely IMO) in favour of the Info Doc format (and its dedicated single-purpose reader). I'll note that the manpage alone renders to 120 pages (man bash | pr | less) onscreen, which is fairly formidable. The PDF-formatted GNU bash manual (a fine format for a reference of this sort) runs 196 pages: <https://www.gnu.org/software/bash/manual/bash.pdf https://www.gnu.org/software/bash/manual/bash.pdf>
- ftaghn 3y ago
  People all too often forget that there is often a lot of extra documentation hidden in /usr/share/doc/ when you install -doc packages, at least on deb systems, and not just info and manpages. For example, the bash manual you cite is in /usr/share/doc/bash/bashref.pdf along with its html version as bashref.html. There is also an /example/ folder filled with commented programs and functions to serve as reference. And a README that suggests looking for "Advanced Bash-Scripting Guide" which is a massive online book filled with all sorts of esoteric knowledge. You can also find things like the release changelogs and one that details incompatibilities from previous versions. While there are exceptions, most packages have useful goodies in there to check first before one has to think about looking things up online.
  - dredmorbius 3y ago
    On Debian, install the dwww package (<https://packages.debian.org/bookworm/dwww https://packages.debian.org/bookworm/dwww>), and point your browser at <https://localhost/dwww/ https://localhost/dwww/>. What you'll find is both a set of pages & hierarchy of documentation as well as search (via swish++) through that space. And yes, there is an absolute trove of information available: - Manpages - Info pages - Default package information - Supplementary package documents, if installed - HOWTOs from the Linux Documentation Project. - Debian-specific documentation (installation, administration, security, policy, and other manuals) - RFCs (divided into several groups, for manageability / relevance) - Resources such as the Linux Gazette There's also a commandline utility, dwww, which will invoke your browser of choice. Point that at a terminal-based browser (w3m, lynx, links, elinks, etc.) and you can quickly scan through docs from your terminal.

bhaak 3y ago

Depending on your job description it is completely fine to not write shell scripts. For anything bigger than a screenful of lines or something that would need proper unit testing or something that other people who are likely to be less versed in shell scripting need to maintain, think twice about using shell scripts. Unixoid machines can use anything as scripts and using a language that breathes the programming language designs of the 1970s is not the best choice.

teddyh 3y ago
> or something that would proper unit testing <https://github.com/kward/shunit2 https://github.com/kward/shunit2>
- nickcox 3y ago
  And several others: https://github.com/dodie/testing-in-bash https://github.com/dodie/testing-in-bash
softwaredoug 3y ago
I typically agree, but I don't think its as simple as a "screenful of lines" criteria. I find it useful when most of your work is wiring other command line tools together. When that's most of your work, shell scripting feels like it has the best affordances. Anything that gets too fancy beyond this, with anything that resembles an algorithm or computation, you'll feel real pain :) Bash doesn't even support floating point!
- nextaccountic 3y ago
  > Bash doesn't even support floating point! But bc (and dc) does, and shell authors deemed this was good enough. And that's part of the problem with the "Unix philosophy"
  - softwaredoug 3y ago
    Yeah every invocation of bc is a call to a command. So if you do it in a loop, its going to be very slow. I had to do this once. It was so slow, I decided to use integers in kind of fixed-point math where I just multiply all the floats by 100 and work with ints. zsh supports float though

robbywashere_ 3y ago

I'd love to see a little language which compiles to _readable_ bash. Could it be so hard? There's a dead project that comes up now and then called bish. https://github.com/tdenniston/bish https://github.com/tdenniston/bish

osmsucks 3y ago
One of my current hobby projects is a scripting language which compiles to POSIX sh (for maximum portability). However, it's far from complete, the output is not exactly readable and POSIX sh is so limited that I might need to change course and target something else. EDIT: I didn't know of Bish, but from its code samples we seem to have converged on similar syntax :)
nerdponx 3y ago
Maybe this is a bit of a silly question, but what's the point? I do think it would be a really fun project. But is there really a practical use for it? Bash is portable but not that portable if you want to care about version compatibility. I understand that it's not always possible to install a different interpreted language like Perl or Python. Maybe the use case is for platforms where you can't install other language interpreters and can't cross-compile a compiled language? It seems like a very specific situation.
scns 3y ago
Not what you asked for but there is Babashka for scripting in Clojure. https://github.com/babashka/babashka https://github.com/babashka/babashka
Shawnj2 3y ago
Write the script in pseudo python and ask ChatGPT to convert it into bash but leave a lot of comments

deleted 3y ago

[deleted]

javier_e06 3y ago

After years writing bash scripts I stumbled with this line... set -e Which forces the script to exit on first error. This line be in the bible as Genesis 1:1

koolba 3y ago
Just wait till you learn about: set -euo pipefail That should be line two of every bash script.
- worldsayshi 3y ago
  The more I use bash the less I want to use it. So much UX weirdness. I feel like making the complaints people usually have with JavaScript. I like JavaScript. Because I'm good enough at it to ignore the bad parts. If I get good enough at bash, will it get better?
  - koolba 3y ago
    > If I get good enough at bash, will it get better? Bash itself will not get better because you'll be stuck writing scripts targeting some ancient version to ensure wider compatibility. It's still a powerful skill to develop. It's a sweet spot between a one-off shell pipeline and something that involves actual logic. Bash scripts should not be used for their logic. Think of them as a way to document a series of commands that you do not want to manually type. That can be including long argument names, generating includes or classpaths, or generating log file names based on the current time. A good rule of thumb is if your script involves any if-statement other than parameter validation or basic on-success-of-x-do-y, it probably should be in a real language.
- scns 3y ago
  Used it in every script after learning about it. Might be great if it could be set globally for all Bash scripts.
- chasil 3y ago
  POSIX shells only support: set -eu You might want to be in the dash POSIX shell for raw speed and portability.

kjuulh 3y ago

I like bash, zsh and other shell languages as well. However, unless you're really experienced, they shouldn't be the product you ship, whether that is internally or externally to you or your company. Shell languages is the final layer of configuration and customization you apply to the setup, it should almost never be the foundation of what you build. Especially for a company, unless you've got really, really skilled engineers on the scripts, it becomes a complete mess. And even then it can become almost impossible to grok. The way we develop infrastructure automation, ci, and developer tools is binaries first and foremost, then plugins for our infrastructure (also binaries, or libraries) and then some zsh on top to provide a nice developer experience. But we've had to heavily scale back our use of zsh, it simply turned into a ball of mud, like all software that isn't maintainable, upgrade-able, or simple enough to parse. This isn't a dig on zsh, bash etc, we've just found out that we can't maintain zsh as a product to supply to our engineers, and as such have to correct.

nerdponx 3y ago
At least you've discovered Zsh and hopefully its significant improvements over Bash. The only Bash benefit is readarray/mapfile, Z Shell is better in just about every other way.
- JNRowe 3y ago
  Unless I'm misreading your comment, zsh does provide similar functionality via zsh/mapfile¹. It, and a lot of other cool stuff, is documented in zshmodules(1). ¹ https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-zsh_002fmapfile-Module https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-...
  - nerdponx 3y ago
    I believe Bash readarray/mapfile is more like doing this in Zsh: lines=( "${(@f)$(my_command)}" ) or lines=( "${(@f)$(<my_file)}" ) which IMO is a little less nice than using a builtin with a name like "readarray". But this is cool too.
3y ago

anagri 3y ago

With LLMs/ChatGPT, i would prefer fine-tuning a model and have it auto-generate bash commands with my natural language description. I remember, I used to keep a diary with landline phone numbers of relatives etc. I used to remember at least a dozen of those numbers for emergency. With mobile and digital contacts, now that number is down to just 2. Wonder, what the programming is going to look like in 10 yrs. If no-one dedicates so much of their time and effort to document, the programmers then would feel helpless. right? 1. Or at that point, LLMs will be intelligent enough to read the source code and figure out the commands. 2. They can generate data by running commands, and train themselves using this simulated data. What other scenarios will occur once we stop generating data using human effort because we all are so dependent on LLMs to generate those commands?

droopyEyelids 3y ago
There's a middle ground alternative where the LLM can read the source, give you a high level description (aka documentation), plus cite the relevant parts of the code. Fingers crossed.

jjice 3y ago

I always hear people talk about writing portable, POSIX compliant shell scripts so they'll work in more environments (assuming those environments are needed, of course), but how often do you run into an environment where you don't have a fairly complete version of Bash available? Genuine open question, I'd be really interested to know what that situation is. Lean container images, embedded, older systems? I'm really curious.

rascul 3y ago
I put a recent bash everywhere I need to work. Not everyone can, though.
pletnes 3y ago
Containers/docker/kubernetes and embedded systems that use e.g busybox is what I’ve seen. Also macOS (used to?) ship ancient bash versions, but there’s recent zsh so that’s mostly fine.
- js2 3y ago
  Yes, macOS is stuck with bash 3 due to the license change with bash 4 and now ships with zsh as the default shell.
yubiox 3y ago
AIX, Solaris, HP-UX
nrclark 3y ago
It's really common for embedded systems to ship without Bash. I don't include it in the production firmware on any systems that I design.
ndsipa_pomu 3y ago
Network equipment can often use a cut down version of Linux or flavour of BSD and they don't always include BASH
inferiorhuman 3y ago

da39a3ee 3y ago

It's so tempting to implement in bash. I made the mistake _again_ this week! It's so seductive to be able to start subprocesses with no ceremony, use output redirection, use jq, etc. If you have even the slightest hesitation regarding whether the task in front of you would be better done in a real programming language, the answer is do it in a real programming language.

Kiala 3y ago

Apparently, this is not an interactive version of the Bible written in Bash.

E39M5S62 3y ago

A lot of what's in the Pure Bash Bible is horrifically slow. Many of those things are substantially faster, even when paying the cost of starting a new process, when you use an external and commonly available tool. I wrote a bash performance profiler that outputs data in a format that flamegraph.pl recognizes - it really helped identify where we could improve the performance of ZFSBootMenu. https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0 https://github.com/zbm-dev/zfsbootmenu/releases/tag/v1.12.0 Don't fall in the trap of thinking things have to be written entirely in bash; it's okay to use other tools to help fill in the gaps.