11 ms·

Write Posix Shell

120 points by j3s 4y ago

jmmv 4y ago
I cringe when I hear people (often senior) saying things like: "Yikes, a 500-line shell script. We must rewrite it in Python for maintainability!!!!1!" -- without even looking at how it has been written. Yes. Shell scripts can be terrible, and most are. But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind, it is perfectly possible to write clean, maintainable, and testable shell scripts. Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. (Edited to add a couple of links to related stuff I wrote in the distant past: 1. https://jmmv.dev/series.html#Shell%20readability https://jmmv.dev/series.html#Shell%20readability and 2. https://github.com/jmmv/shtk https://github.com/jmmv/shtk)
- eviks 4y ago
  > But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind Then one would simply choose a proper language that had more of those solid principles laid at its foundation
  - enriquto 4y ago
    Unix shell is a proper language and has very solid principles beautifully laid at its foundation by Douglas McIlroy.
    - wizhi 4y ago
      Do you have some writing on this? I'd love to read it.
    - chasil 4y ago
      I understood that Stephen Borne's love of ALGOL was the chief structural influence of the POSIX shell (so much so that he used the C preprocessor to turn his source code into fake ALGOL). "Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)." https://en.m.wikipedia.org/wiki/Bourne_shell https://en.m.wikipedia.org/wiki/Bourne_shell
- als0 4y ago
  > Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. This is an important distinction, and probably why I still make shell scripts. I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.
  - sigzero 4y ago
    I love shellcheck. You can run it locally or via the web if you don't want to install it. https://www.shellcheck.net/ https://www.shellcheck.net/
- jonhohle 4y ago
  There are even great unit testing frameworks for testing across a variety of shells and versions. This can be integrated pretty easily into existing build tools and run the same way the other code it may (or may not) support is run.
- HL33tibCe7 4y ago
  > it is perfectly possible to write clean, maintainable, and testable shell scripts. Possible? Maybe. Easy? No. Especially the “testable” part. The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…) So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it? (This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).
  - jmmv 4y ago
    I see your point, but what happens in your alternate world of having to maintain the 5000-line Python version 5 years down the line? You face the same issues you describe. The people that wrote the original beautiful code aren't there anymore, and whoever comes touching the code "messes it up". Most Python "scripts" I have had to deal with that had been around for a while were pretty terrible as well.
    - HL33tibCe7 4y ago
      I don’t think you do see my point: my point is that writing clean, testable, maintainable Python is both easier and a more widespread skill than writing equivalently clean, testable and maintainable shell. Both of these factors will mean that the Python remains maintainable for longer.
    - fbdab103 4y ago
      Bad code is bad code, regardless of language. At least with Python, I more or less know all of the idioms and can at a glance understand what the flow of a program is trying to accomplish. Shell has a lot of esoteric voodoo syntax to replicate better string/path/array handling that is just a given in other languages.
    - _dain_ 4y ago
- j3s 4y ago
  shell is loose and offers a lot of power per line, so it’s pretty easy to make a spaghetti nightmare. but it’s very possible to write nice shell. i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility
  - jbverschoor 4y ago
    A program in any language can be maintainable when written properly and maintained by someone who is capable, diligent, and has a good test setup.
  - oneshtein 4y ago
    Bash is turing-complete, so it's possible to write automated test cases in bash. Example: https://github.com/vlisivka/bash-modules/blob/master/bash-modules/test/test_string.sh https://github.com/vlisivka/bash-modules/blob/master/bash-mo...
- throw0101c 4y ago
  > Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. The dehydrated ACME client is 2400 lines of bash/zsh: * https://github.com/dehydrated-io/dehydrated https://github.com/dehydrated-io/dehydrated And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell: * https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies: * https://packages.debian.org/bullseye/python3-certbot https://packages.debian.org/bullseye/python3-certbot
  - ori_b 4y ago
    I'm not sure why -- the one shipped with 9front is less than a thousand lines of C, and supports both DNS and HTTP challenges: https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd/auth/acmed.c/f.html https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm... http://man.9front.org/8/acmed http://man.9front.org/8/acmed The ACME protocol isn't that complicated.
    - throw0101c 4y ago
      Even with Python things can be written small (200 LoC) it seems: * https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.py https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p... * https://github.com/diafygi/acme-tiny https://github.com/diafygi/acme-tiny * https://packages.debian.org/stable/acme-tiny https://packages.debian.org/stable/acme-tiny
- pjmlp 4y ago
  Do you a good anecdote regarding shell scripts? Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks. With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such. I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea. Feedback from customer, slower than those scripts and not as easy to change, what a surprise.
  - kps 4y ago
    I use Korn shell (ksh93) for quick scripts. Besides arrays and such (it's approximately a superset of bash, since bash copied some but not all of ksh), the killer feature is a declarative extended `getopts` that makes it trivial to provide even the quickest and dirtiest script with short/long option processing with full help, so that the purpose and usage of the script remain comprehensible five days or five years later. https://gist.github.com/kpschoedel/91fdcfa934111f3846efb50341b7342d https://gist.github.com/kpschoedel/91fdcfa934111f3846efb5034...
- jamal-kumar 4y ago
  From the article: consider this sentence: "arrays do not exist in posix shell" ... "NOOO!" you shout, because you like data structures a LOT "now THAT'S a good reason to use bash! bash supports arrays!" absolutely not! i believe that you should use shell if your problem is: - small and scoped (~200 lines of shell or less) - unlikely to increase in size and scope as time goes on - not very complex if you find yourself desiring arrays, good error handling, static typing, structs, etc, then your problem should be solved using a different language, not shell. I tend to agree with this heuristic. It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.
  - djbusby 4y ago
    Perl and Python are part of LSB https://refspecs.linuxfoundation.org/lsb.shtml https://refspecs.linuxfoundation.org/lsb.shtml
    - hedora 4y ago
      Yeah, but perl 5 hasn’t broken compatibility (as far as I’ve noticed) since 2000 or so. Python breaks compatibility every week, in practice. Does LSB include some sort of magical frozen in time version of python 2 or something?
      - gabrielhidasy 4y ago
        In my 12 years of python I was hit with compatibility problems once, in a school project where someone started in python 2 and we changed for python 3. What libraries break that often so I can stay away?
        hedora 4y ago
        scipy / numpy / etc broke multiple times for me. I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.
- jbverschoor 4y ago
  The reason I don’t like shell scripts are spaces and the weird constructs sometimes. I never got into Perl, but I understand why it was popular. Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert. Shell is nice not because of the shell language, but the ease of executing and chaining other programs.
- pushedx 4y ago
  The shell provides an interface to the filesystem and concurrent pipelines like no other readily-available language. Many tasks are more concisely expressed and readable in shell than in any other language.
  - yjftsjthsd-h 4y ago
    Yep; shell isn't the best tool for every job, but the things it's good at, it's really good at.
  - lapinot 4y ago
    The fact that posix shell is somehow state of the art of concurrent pipeline and filesystem interface is really more a critique of state of the art than a praise of shell. I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important. Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).
    - lozenge 4y ago
      Powershell is the new shell. As it passes objects the filtering, sorting, joining is generic across files or any other data type.
      - 4y ago
- ra1231963 4y ago
  This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t. Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.
  - jmmv 4y ago
    Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
    - kaba0 4y ago
      I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
      - mananaysiempre 4y ago
        > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
- d0mine 4y ago
  > Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.
- m1keil 4y ago
  The question is if the script outgrew its original intended lifespan. In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense. So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.
- simias 4y ago
  100% disagree. Standard /bin/sh has terrible error handling facility, which on its own is enough to disqualify it as a "proper" programming language in my book. Even C is not that bad, and C is pretty terrible. I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine. The shell is not a real language and must be written as such.
  - alganet 4y ago
    There's no such thing as "standard /bin/sh". You're talking about busybox ash, or dash, or something else that is shipped as /bin/sh in your distro. Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell. In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.
    - simias 4y ago
      I meant POSIX shell or more specifically a loosely defined subset of all these shell dialects that is reasonably portable in practice. I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies. >In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6... But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking. Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.
- sureglymop 4y ago
  I cringe everytime my manager tells me not to write shell scripts but javascript instead. There is ShellCheck, Bats and there are extensions for e.g. Visual Studio Code or Jetbrains IDEA that make writing shell and bash scripts a pleasure. I find it generally much more productive to write a shell script because I already know all the gnu/unix userland tools. And i know this shell script will be portable and still running decades from now unless some version specific node script with dependencies.
throw0101c 4y ago
But also please consider using Shellcheck if you do: * https://www.shellcheck.net https://www.shellcheck.net
- jamal-kumar 4y ago
  It's such an essential tool. Always such a beast to install it though, definitely keep it off any servers. It's probably saved myself from weird hard to see mistakes so many times I wouldn't be able to count them because if I didn't have that tool I might not even see it until runtime, which is a terrible place to discover an error.
  - fbdab103 4y ago
    >Always such a beast to install it though, definitely keep it off any servers. I do not follow. Isn't it just a static executable? I have only ever apt-get installed, but I am unsure where the complication rests, unless you are compiling from source.
- blueflow 4y ago
  Shellcheck is useful if you are learning shell, but it gets really annoying over time with its amount of false positives. Its still a static checker that does not understand context. I've seen too many scripts where half of all comments lines were shellcheck disables.
  - return_to_monke 4y ago
    I'd rather have false lints than miss lints
  - Too 4y ago
    In my experience false positives are rare. You need to a take some time to actually understand the warnings to know how to avoid them, often there is some edge case lurking even the most trivial code. I see a lot of people are way too quick on dismissing the warnings, because they are overconfident that they know shell, when in fact most people don't have the slightest clue on the amounts of footguns hidden.
ape4 4y ago
I wonder of shellcheck could suggest when its time to rewrite in another language.
- kaba0 4y ago
  if (lineCount >= 3) { return true; }
_dain_ 4y ago
I'm not convinced. >when should i use shell? >in short; when the problem you're solving is small, well defined, and unlikely to change, consider shell. If you have to caveat it this much, it's a pretty weak statement. Any language works when the problem is small, well defined, and unlikely to change! Strict POSIX compatibility is something of a chimera. You don't just need to read the POSIX shell spec, because you're not just programming in POSIX shell, you have to use the coreutils as well to get anything done. Do you remember which flags on grep are POSIX-only, and which are extensions provided by the GNU version? And then the same for sed, ls, and the rest of the UNIX zoo? There's an order of magnitude more shit you have to remember and worry about. Needing strict POSIX compatibility is an increasingly niche use-case. I find it baffling that, on the one hand, we have these modern software isolation and distribution systems like containers and Nix and so on, technologies that are meant to let us run anything anywhere reliably, and then we forget we have these marvels and wear the POSIX hair-shirt anyway. Why can't you just include a better language in the container? A container is supposed to contain things! Stop hitting yourself in the face repeatedly! Finally if you somehow really do need to live in the POSIX cave, you don't need to use shell. POSIX also specifies a much saner language: awk.
- blueflow 4y ago
  > Do you remember which flags on grep are POSIX-only, and which are extensions provided by the GNU version? Use the posix man pages as reference. `man 1p grep` and you only see the portable options. Its only one step away from using the regular manpages.
  - abathur 4y ago
    til :)
  - Karellen 4y ago
    Worth noting that `man-pages-posix` is not under any kind of Free/Open Source/CC-style license, so may not be available by default for your OS. e.g. in Debian you have to enable the `non-free` section to make them apt-installable.
  - 4y ago
mastax 4y ago
For legacy reasons, my filesystem is full of files and directories with spaces. Are they impossible to deal with in shell? No. Quoting, print0, it's not even hard. But it's like taking a Sunday stroll through an unburied minefield. I'd rather walk in the park.
codeflo 4y ago
One important con that isn't mentioned: It's so unnecessarily confusing how to write a shell script that can properly deal with whitespace, dollar signs or other special characters in filenames. Of course it's possible and learnable. But even if you know the tricks, it's tedious, and human error prone. I had a team mate (many years ago) who used to test the local admins at any new workplace by putting files names named "$HOME" and similar in his home directory. He claimed it wasn't that rare for this to crash some backup script. Yet this isn't inherently hard. In fact it's trivial in literally any "proper" programming language, no matter which one you end up picking. I get why shell is "scripting of last resort", I regularly write shell scripts myself. I don't get why people would choose it over anything else where choices are available.
- fbdab103 4y ago
  >putting files names "$HOME" and similar in his home directory If you do this, the fallout is on you. Does he also make files called `~`?
  - Kwpolska 4y ago
    The backup script's job is to back up all files, not crash fatally because someone used a dollar sign, a tilde, or a space, or `rm -rf` in a file name. The sysadmin’s job is to write robust tools, not expect users to understand arcane UNIX stuff.
    - fbdab103 4y ago
      You are right of course. Just my knee jerk reaction that if someone is deliberately creating such files, they know exactly how much peril they are creating for themselves. Owing to so many years on the terminal, I still hesitate to name files with a space - a practice which the uninitiated find odd.
      - lmz 4y ago
        This would be considered silly if a web dev said this about e.g. a script breaking because someone entered a single quote into a text field and it was interpolated directly into an SQL statement. We should not tolerate this from sysadmins either.
_0w8t 4y ago
My rule of thumb is that one needs to switch to other language when one needs maps or complex array operations. Arrays for proper processing of arguments or storing a sequence of results are doable in a POSIX shell with couple of utilities.
meindnoch 4y ago
Thanks, but no thanks. I'll stick with bash. It's ubiquitious enough for my purposes, and I can't live without =~ and <().
gavinhoward 4y ago
I haven't seen anyone mention one important reason for writing POSIX shell over other shells. The article implies it, but never says in directly: portability. I wrote a 2100 line monster of a POSIX shell script. [1] The purpose was entirely for portability. That script allows me to use a straight POSIX Makefile for building my bc. The combination of the two means that my bc builds without modification on any semi-POSIX system. That said, it was hard. I basically had to constrain myself to POSIX utilities and still implement a template language. I got it done, and it was worth it, but if you don't have to, don't do it. [1]: https://git.gavinhoward.com/gavin/bc/src/branch/master/configure.sh https://git.gavinhoward.com/gavin/bc/src/branch/master/confi...
mogoh 4y ago
My reason for staying away from posix-shell as much as possible is that it has so many ways to shoot itself in the foot. This is also only indirectly touched in the article. I'd rather write 10x as long Python scripts than to shoot myself in the foot with posix-shell again.
jraph 4y ago
Many people advise against using shell scripts for non trivial stuff. Is there a language as convenient as shell for calling and piping external programs but with proper typing and error handling? People advise using Python but I don't see myself using subprocess and Popen for gluing programs together and end up writing shell scripts anyway, I'm pretty comfortable with posix shell.
- grumblehound 4y ago
  I tried https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ recently and it was quite good. You might want to write a wrapper function depending on what error or newline splitting behaviour you want.
- remram 4y ago
  What's wrong with Python and the subprocess module? It is exactly what I advise.
  - travisjungroth 4y ago
    If you have a single command it’s fine. Not great because of the list syntax, but fine. But I’ve found when I’m doing shell commands and using Python as glue, the Python takes up most of the script while doing barely any of the work. Don’t get me wrong, I like Python more and can barely write shell (ChatGPT/Copilot has made it easy). But it is not as convenient for calling and piping commands as shell.
- bqmjjx0kac 4y ago
  Once I started running Shellcheck on my scripts, I felt like they were full of footguns waiting to go off.
- nickjj 4y ago
  > I'm pretty comfortable with posix shell. It's ok to trust yourself. Write 1 file shell scripts for convenient zero dependency command line tools that will work on most systems. If things start to get rough due to shell limitations, then Python scripts without dependencies is another 1 file solution that's quite portable. You can even mix and match (mostly use shell and call Python from it for the complicated bits as a last resort). I wrote about this exact topic recently around trying to deal with string splitting in Bash: https://nickjanetakis.com/blog/splitting-strings-by-a-delimiter-with-ifs-and-bash-string-replace https://nickjanetakis.com/blog/splitting-strings-by-a-delimi... The takeaway is that for 99% of cases using IFS or Bash's string replace in a variable is great but if you have strict parsing requirements then maybe going with Python is worth it. The post links to a massive SO answer that demonstrates how string manipulation in shell scripting is hard.
joshlk 4y ago
Shell has its place as a glue language between commands. I tried writing a good CLI in bash this week and yikes it was hard. The lack of a package manager and good libraries really hurt. If you could “pip” install bash libraries like an argparse equivalent it would help a lot
- abathur 4y ago
  https://t-ravis.com/post/shell/no_look_no_leap_shell_with_nix/ https://t-ravis.com/post/shell/no_look_no_leap_shell_with_ni... (but of course, that's it's own portability/bootstrapping problem if you need genuine internet-rando-level portability)
  - abathur 4y ago
    The post I linked focuses on standalone executable dependencies, but the same concept applies to libraries. If you want a quick example, you can look at how: - shellswain depends on a little trap-namespacing library called comity: https://github.com/search?q=repo%3Aabathur%2Fshellswain+comity+path%3A**%2Fshellswain.bash+OR+language%3Anix&type=code https://github.com/search?q=repo%3Aabathur%2Fshellswain+comi... - comity in turn depends on a bash events library called bashup.events: https://github.com/search?q=repo%3Aabathur%2Fcomity+events+path%3A**%2Fcomity.*&type=code https://github.com/search?q=repo%3Aabathur%2Fcomity+events+p... Early last year I also wrote a little more generally about the idea of being able to finally build this shell code-sharing ecosystem in https://t-ravis.com/post/shell/neighborly_shell_with_bashup.events/ https://t-ravis.com/post/shell/neighborly_shell_with_bashup....
kristopolous 4y ago
I'm a big fan of not posix but instead modern bash and to all the complainers about dash and ash, I say "tough cookies". Bash has arithmetic, dictionaries, regex matching, nice string manipulation, co-processes, TCP pipes built in, it's pretty nice. However, bash has very low tolerance for incompetency. It's a brutal friend. It's cooking dinner with a pocket knife. You gotta actually read the docs and be sincere in your studies. I was around when supporting Solaris, HPUX, AIX, and Irix was a real concern. Those times sucked and those platforms are so minor I stopped caring about them except for enthusiast projects many years ago. (And I still actually have an hpux machine) Sometimes I'll even use zsh. It can even do floating point. Here's some example of a modern tool I have written for a subject I call "music discovery" https://github.com/kristopolous/music-explorer/tree/master/tools https://github.com/kristopolous/music-explorer/tree/master/t... You'll see many languages in there but you won't see posix sh. It's slow, it's clunky, it's a pain in the ass. If you don't like my practice then I guess don't use it. I've been using/developing these particular tools nearly every day for over 3 years and it works well for me. I'm not going to say bash is awesome but it's pretty great for programming. I use zsh as my interactive shell though, it's miles ahead of bash in that department. Watching people who really know what they're doing in zsh is a total mindfuck.
- User23 4y ago
  > I was around when supporting Solaris, HPUX, AIX, and Irix was a real concern. Back in the 90s it was surprising to not find gnu userspace installed on the systems I used.
  - jkmcf 4y ago
    Back in the 90s, Solaris was the big hold out -- IRIX and HPUX both came with at least a subset. First thing I had to do on Solaris was install the GNU tools, and, thankfully, eventually there were packages to manage that installation.
  - kristopolous 4y ago
    About 1/3 of the debug sessions involved looking at the screen peeved going "yes, obviously I meant gsed and ggrep you dumbass." You go through the man page of the vendor supplied version like "What do you mean you don't support feature x? This can't be real. I swear I'm being recorded right now. A bunch of people are sitting around a large conference table in mountain view right now laughing at my pain" Eventually I just read them as parody. Things like this can be really funny if you allow them to be.
RcouF1uZ4gsC 4y ago
Rust may actually be a solution in this space. It is better typed and more testable than Python. It has fewer issues with environmental dependencies due. It is easy to install and build. There is ongoing work to rewrite the coreutils in Rust, and if they were exposed with an ergonomic library interface, would provide a lot of the value of shell with much better safety.
- k3vinw 4y ago
  For sure! I used to think that python was the next logical step for a growing project that started out life as a shell script, but now I think we have even better choices like rust.
- maleldil 4y ago
  Rust is _a lot_ slower to develop than sh/bash/Python, though. If your script is in reality a semi-complex CLI tool, then Rust is great, but if it's a 1k line shell script, Python/Ruby/JS (especially with Deno) are better options.
nailer 4y ago
I used to be one of the write .sh people. Now I just ensure my shells scripts end in .bash and don’t insist people limit themselves to an effectively unmaintained standard.
ko1nksm 4y ago
Why not install bash and GNU utilities?
delta_p_delta_x 4y ago
> posix shell is portable: it'll run on debian, ubuntu, on openbsd, in an alpine container, on illumos! arch! even horrible old AIX! > lists three Linux distributions and three Unixes: calls POSIX shell 'portable'. You know where POSIX shell scripts don't run natively? Windows. If you're going to conveniently exclude about half the developer demographic and about 90% of the user demographic to push a not-very-valid point... But given the aesthetic of the website, I somehow get the feeling the author couldn't care less about Windows (or its users). Please use a proper glue language with nice types and syntax, like Python or PowerShell. IMO PowerShell >> POSIX shells and children; interop with full .NET already gives it a several point-lead over POSIX shells. P.S., I notice increasingly more bloggers abandoning sentence case, and it makes for very difficult reading.
tpoacher 4y ago
Opening "man bash" transformed my experience with bash, which has now risen to the top in the list of my favourite languages. I don't get all the hate. I used fish for a while and really liked it, but I realise now this was only because I didn't really know bash. (that and it came with some nice defaults out of the box, like prefix sensitive command history, which requires a .inputrc oneliner for bash) But now every time I try fish I'm like, eh, and immediately switch back to bash. I havent tried zsh, but it just looks gimmicky and "appleish" to me.

120 points by j3s 4y ago

jmmv 4y ago

I cringe when I hear people (often senior) saying things like: "Yikes, a 500-line shell script. We must rewrite it in Python for maintainability!!!!1!" -- without even looking at how it has been written. Yes. Shell scripts can be terrible, and most are. But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind, it is perfectly possible to write clean, maintainable, and testable shell scripts. Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. (Edited to add a couple of links to related stuff I wrote in the distant past: 1. https://jmmv.dev/series.html#Shell%20readability https://jmmv.dev/series.html#Shell%20readability and 2. https://github.com/jmmv/shtk https://github.com/jmmv/shtk)

eviks 4y ago
> But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind Then one would simply choose a proper language that had more of those solid principles laid at its foundation
- enriquto 4y ago
  Unix shell is a proper language and has very solid principles beautifully laid at its foundation by Douglas McIlroy.
  - wizhi 4y ago
    Do you have some writing on this? I'd love to read it.
  - chasil 4y ago
    I understood that Stephen Borne's love of ALGOL was the chief structural influence of the POSIX shell (so much so that he used the C preprocessor to turn his source code into fake ALGOL). "Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)." https://en.m.wikipedia.org/wiki/Bourne_shell https://en.m.wikipedia.org/wiki/Bourne_shell
als0 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. This is an important distinction, and probably why I still make shell scripts. I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.
- sigzero 4y ago
  I love shellcheck. You can run it locally or via the web if you don't want to install it. https://www.shellcheck.net/ https://www.shellcheck.net/
jonhohle 4y ago
There are even great unit testing frameworks for testing across a variety of shells and versions. This can be integrated pretty easily into existing build tools and run the same way the other code it may (or may not) support is run.
HL33tibCe7 4y ago
> it is perfectly possible to write clean, maintainable, and testable shell scripts. Possible? Maybe. Easy? No. Especially the “testable” part. The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…) So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it? (This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).
- jmmv 4y ago
  I see your point, but what happens in your alternate world of having to maintain the 5000-line Python version 5 years down the line? You face the same issues you describe. The people that wrote the original beautiful code aren't there anymore, and whoever comes touching the code "messes it up". Most Python "scripts" I have had to deal with that had been around for a while were pretty terrible as well.
  - HL33tibCe7 4y ago
    I don’t think you do see my point: my point is that writing clean, testable, maintainable Python is both easier and a more widespread skill than writing equivalently clean, testable and maintainable shell. Both of these factors will mean that the Python remains maintainable for longer.
  - fbdab103 4y ago
    Bad code is bad code, regardless of language. At least with Python, I more or less know all of the idioms and can at a glance understand what the flow of a program is trying to accomplish. Shell has a lot of esoteric voodoo syntax to replicate better string/path/array handling that is just a given in other languages.
  - _dain_ 4y ago
j3s 4y ago
shell is loose and offers a lot of power per line, so it’s pretty easy to make a spaghetti nightmare. but it’s very possible to write nice shell. i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility
- jbverschoor 4y ago
  A program in any language can be maintainable when written properly and maintained by someone who is capable, diligent, and has a good test setup.
- oneshtein 4y ago
  Bash is turing-complete, so it's possible to write automated test cases in bash. Example: https://github.com/vlisivka/bash-modules/blob/master/bash-modules/test/test_string.sh https://github.com/vlisivka/bash-modules/blob/master/bash-mo...
throw0101c 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. The dehydrated ACME client is 2400 lines of bash/zsh: * https://github.com/dehydrated-io/dehydrated https://github.com/dehydrated-io/dehydrated And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell: * https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies: * https://packages.debian.org/bullseye/python3-certbot https://packages.debian.org/bullseye/python3-certbot
- ori_b 4y ago
  I'm not sure why -- the one shipped with 9front is less than a thousand lines of C, and supports both DNS and HTTP challenges: https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd/auth/acmed.c/f.html https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm... http://man.9front.org/8/acmed http://man.9front.org/8/acmed The ACME protocol isn't that complicated.
  - throw0101c 4y ago
    Even with Python things can be written small (200 LoC) it seems: * https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.py https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p... * https://github.com/diafygi/acme-tiny https://github.com/diafygi/acme-tiny * https://packages.debian.org/stable/acme-tiny https://packages.debian.org/stable/acme-tiny
pjmlp 4y ago
Do you a good anecdote regarding shell scripts? Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks. With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such. I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea. Feedback from customer, slower than those scripts and not as easy to change, what a surprise.
- kps 4y ago
  I use Korn shell (ksh93) for quick scripts. Besides arrays and such (it's approximately a superset of bash, since bash copied some but not all of ksh), the killer feature is a declarative extended `getopts` that makes it trivial to provide even the quickest and dirtiest script with short/long option processing with full help, so that the purpose and usage of the script remain comprehensible five days or five years later. https://gist.github.com/kpschoedel/91fdcfa934111f3846efb50341b7342d https://gist.github.com/kpschoedel/91fdcfa934111f3846efb5034...
jamal-kumar 4y ago
From the article: consider this sentence: "arrays do not exist in posix shell" ... "NOOO!" you shout, because you like data structures a LOT "now THAT'S a good reason to use bash! bash supports arrays!" absolutely not! i believe that you should use shell if your problem is: - small and scoped (~200 lines of shell or less) - unlikely to increase in size and scope as time goes on - not very complex if you find yourself desiring arrays, good error handling, static typing, structs, etc, then your problem should be solved using a different language, not shell. I tend to agree with this heuristic. It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.
- djbusby 4y ago
  Perl and Python are part of LSB https://refspecs.linuxfoundation.org/lsb.shtml https://refspecs.linuxfoundation.org/lsb.shtml
  - hedora 4y ago
    Yeah, but perl 5 hasn’t broken compatibility (as far as I’ve noticed) since 2000 or so. Python breaks compatibility every week, in practice. Does LSB include some sort of magical frozen in time version of python 2 or something?
    - gabrielhidasy 4y ago
      In my 12 years of python I was hit with compatibility problems once, in a school project where someone started in python 2 and we changed for python 3. What libraries break that often so I can stay away?
      - hedora 4y ago
        scipy / numpy / etc broke multiple times for me. I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.
jbverschoor 4y ago
The reason I don’t like shell scripts are spaces and the weird constructs sometimes. I never got into Perl, but I understand why it was popular. Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert. Shell is nice not because of the shell language, but the ease of executing and chaining other programs.
pushedx 4y ago
The shell provides an interface to the filesystem and concurrent pipelines like no other readily-available language. Many tasks are more concisely expressed and readable in shell than in any other language.
- yjftsjthsd-h 4y ago
  Yep; shell isn't the best tool for every job, but the things it's good at, it's really good at.
- lapinot 4y ago
  The fact that posix shell is somehow state of the art of concurrent pipeline and filesystem interface is really more a critique of state of the art than a praise of shell. I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important. Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).
  - lozenge 4y ago
    Powershell is the new shell. As it passes objects the filtering, sorting, joining is generic across files or any other data type.
    - 4y ago
ra1231963 4y ago
This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t. Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.
- jmmv 4y ago
  Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
  - kaba0 4y ago
    I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
    - mananaysiempre 4y ago
      > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
d0mine 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.
m1keil 4y ago
The question is if the script outgrew its original intended lifespan. In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense. So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.
simias 4y ago
100% disagree. Standard /bin/sh has terrible error handling facility, which on its own is enough to disqualify it as a "proper" programming language in my book. Even C is not that bad, and C is pretty terrible. I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine. The shell is not a real language and must be written as such.
- alganet 4y ago
  There's no such thing as "standard /bin/sh". You're talking about busybox ash, or dash, or something else that is shipped as /bin/sh in your distro. Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell. In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.
  - simias 4y ago
    I meant POSIX shell or more specifically a loosely defined subset of all these shell dialects that is reasonably portable in practice. I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies. >In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6... But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking. Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.
sureglymop 4y ago
I cringe everytime my manager tells me not to write shell scripts but javascript instead. There is ShellCheck, Bats and there are extensions for e.g. Visual Studio Code or Jetbrains IDEA that make writing shell and bash scripts a pleasure. I find it generally much more productive to write a shell script because I already know all the gnu/unix userland tools. And i know this shell script will be portable and still running decades from now unless some version specific node script with dependencies.

throw0101c 4y ago

But also please consider using Shellcheck if you do: * https://www.shellcheck.net https://www.shellcheck.net

jamal-kumar 4y ago
It's such an essential tool. Always such a beast to install it though, definitely keep it off any servers. It's probably saved myself from weird hard to see mistakes so many times I wouldn't be able to count them because if I didn't have that tool I might not even see it until runtime, which is a terrible place to discover an error.
- fbdab103 4y ago
  >Always such a beast to install it though, definitely keep it off any servers. I do not follow. Isn't it just a static executable? I have only ever apt-get installed, but I am unsure where the complication rests, unless you are compiling from source.
blueflow 4y ago
Shellcheck is useful if you are learning shell, but it gets really annoying over time with its amount of false positives. Its still a static checker that does not understand context. I've seen too many scripts where half of all comments lines were shellcheck disables.
- return_to_monke 4y ago
  I'd rather have false lints than miss lints
- Too 4y ago
  In my experience false positives are rare. You need to a take some time to actually understand the warnings to know how to avoid them, often there is some edge case lurking even the most trivial code. I see a lot of people are way too quick on dismissing the warnings, because they are overconfident that they know shell, when in fact most people don't have the slightest clue on the amounts of footguns hidden.

ape4 4y ago

I wonder of shellcheck could suggest when its time to rewrite in another language.

kaba0 4y ago
if (lineCount >= 3) { return true; }

_dain_ 4y ago

I'm not convinced. >when should i use shell? >in short; when the problem you're solving is small, well defined, and unlikely to change, consider shell. If you have to caveat it this much, it's a pretty weak statement. Any language works when the problem is small, well defined, and unlikely to change! Strict POSIX compatibility is something of a chimera. You don't just need to read the POSIX shell spec, because you're not just programming in POSIX shell, you have to use the coreutils as well to get anything done. Do you remember which flags on grep are POSIX-only, and which are extensions provided by the GNU version? And then the same for sed, ls, and the rest of the UNIX zoo? There's an order of magnitude more shit you have to remember and worry about. Needing strict POSIX compatibility is an increasingly niche use-case. I find it baffling that, on the one hand, we have these modern software isolation and distribution systems like containers and Nix and so on, technologies that are meant to let us run anything anywhere reliably, and then we forget we have these marvels and wear the POSIX hair-shirt anyway. Why can't you just include a better language in the container? A container is supposed to contain things! Stop hitting yourself in the face repeatedly! Finally if you somehow really do need to live in the POSIX cave, you don't need to use shell. POSIX also specifies a much saner language: awk.

blueflow 4y ago
> Do you remember which flags on grep are POSIX-only, and which are extensions provided by the GNU version? Use the posix man pages as reference. `man 1p grep` and you only see the portable options. Its only one step away from using the regular manpages.
- abathur 4y ago
  til :)
- Karellen 4y ago
  Worth noting that `man-pages-posix` is not under any kind of Free/Open Source/CC-style license, so may not be available by default for your OS. e.g. in Debian you have to enable the `non-free` section to make them apt-installable.
- 4y ago

mastax 4y ago

For legacy reasons, my filesystem is full of files and directories with spaces. Are they impossible to deal with in shell? No. Quoting, print0, it's not even hard. But it's like taking a Sunday stroll through an unburied minefield. I'd rather walk in the park.

codeflo 4y ago

One important con that isn't mentioned: It's so unnecessarily confusing how to write a shell script that can properly deal with whitespace, dollar signs or other special characters in filenames. Of course it's possible and learnable. But even if you know the tricks, it's tedious, and human error prone. I had a team mate (many years ago) who used to test the local admins at any new workplace by putting files names named "$HOME" and similar in his home directory. He claimed it wasn't that rare for this to crash some backup script. Yet this isn't inherently hard. In fact it's trivial in literally any "proper" programming language, no matter which one you end up picking. I get why shell is "scripting of last resort", I regularly write shell scripts myself. I don't get why people would choose it over anything else where choices are available.

fbdab103 4y ago
>putting files names "$HOME" and similar in his home directory If you do this, the fallout is on you. Does he also make files called `~`?
- Kwpolska 4y ago
  The backup script's job is to back up all files, not crash fatally because someone used a dollar sign, a tilde, or a space, or `rm -rf` in a file name. The sysadmin’s job is to write robust tools, not expect users to understand arcane UNIX stuff.
  - fbdab103 4y ago
    You are right of course. Just my knee jerk reaction that if someone is deliberately creating such files, they know exactly how much peril they are creating for themselves. Owing to so many years on the terminal, I still hesitate to name files with a space - a practice which the uninitiated find odd.
    - lmz 4y ago
      This would be considered silly if a web dev said this about e.g. a script breaking because someone entered a single quote into a text field and it was interpolated directly into an SQL statement. We should not tolerate this from sysadmins either.

_0w8t 4y ago

My rule of thumb is that one needs to switch to other language when one needs maps or complex array operations. Arrays for proper processing of arguments or storing a sequence of results are doable in a POSIX shell with couple of utilities.

meindnoch 4y ago

Thanks, but no thanks. I'll stick with bash. It's ubiquitious enough for my purposes, and I can't live without =~ and <().

gavinhoward 4y ago

I haven't seen anyone mention one important reason for writing POSIX shell over other shells. The article implies it, but never says in directly: portability. I wrote a 2100 line monster of a POSIX shell script. [1] The purpose was entirely for portability. That script allows me to use a straight POSIX Makefile for building my bc. The combination of the two means that my bc builds without modification on any semi-POSIX system. That said, it was hard. I basically had to constrain myself to POSIX utilities and still implement a template language. I got it done, and it was worth it, but if you don't have to, don't do it. [1]: https://git.gavinhoward.com/gavin/bc/src/branch/master/configure.sh https://git.gavinhoward.com/gavin/bc/src/branch/master/confi...

mogoh 4y ago

My reason for staying away from posix-shell as much as possible is that it has so many ways to shoot itself in the foot. This is also only indirectly touched in the article. I'd rather write 10x as long Python scripts than to shoot myself in the foot with posix-shell again.

jraph 4y ago

Many people advise against using shell scripts for non trivial stuff. Is there a language as convenient as shell for calling and piping external programs but with proper typing and error handling? People advise using Python but I don't see myself using subprocess and Popen for gluing programs together and end up writing shell scripts anyway, I'm pretty comfortable with posix shell.

grumblehound 4y ago
I tried https://plumbum.readthedocs.io/en/latest/ https://plumbum.readthedocs.io/en/latest/ recently and it was quite good. You might want to write a wrapper function depending on what error or newline splitting behaviour you want.
remram 4y ago
What's wrong with Python and the subprocess module? It is exactly what I advise.
- travisjungroth 4y ago
  If you have a single command it’s fine. Not great because of the list syntax, but fine. But I’ve found when I’m doing shell commands and using Python as glue, the Python takes up most of the script while doing barely any of the work. Don’t get me wrong, I like Python more and can barely write shell (ChatGPT/Copilot has made it easy). But it is not as convenient for calling and piping commands as shell.
bqmjjx0kac 4y ago
Once I started running Shellcheck on my scripts, I felt like they were full of footguns waiting to go off.
nickjj 4y ago
> I'm pretty comfortable with posix shell. It's ok to trust yourself. Write 1 file shell scripts for convenient zero dependency command line tools that will work on most systems. If things start to get rough due to shell limitations, then Python scripts without dependencies is another 1 file solution that's quite portable. You can even mix and match (mostly use shell and call Python from it for the complicated bits as a last resort). I wrote about this exact topic recently around trying to deal with string splitting in Bash: https://nickjanetakis.com/blog/splitting-strings-by-a-delimiter-with-ifs-and-bash-string-replace https://nickjanetakis.com/blog/splitting-strings-by-a-delimi... The takeaway is that for 99% of cases using IFS or Bash's string replace in a variable is great but if you have strict parsing requirements then maybe going with Python is worth it. The post links to a massive SO answer that demonstrates how string manipulation in shell scripting is hard.

joshlk 4y ago

Shell has its place as a glue language between commands. I tried writing a good CLI in bash this week and yikes it was hard. The lack of a package manager and good libraries really hurt. If you could “pip” install bash libraries like an argparse equivalent it would help a lot

abathur 4y ago
https://t-ravis.com/post/shell/no_look_no_leap_shell_with_nix/ https://t-ravis.com/post/shell/no_look_no_leap_shell_with_ni... (but of course, that's it's own portability/bootstrapping problem if you need genuine internet-rando-level portability)
- abathur 4y ago
  The post I linked focuses on standalone executable dependencies, but the same concept applies to libraries. If you want a quick example, you can look at how: - shellswain depends on a little trap-namespacing library called comity: https://github.com/search?q=repo%3Aabathur%2Fshellswain+comity+path%3A**%2Fshellswain.bash+OR+language%3Anix&type=code https://github.com/search?q=repo%3Aabathur%2Fshellswain+comi... - comity in turn depends on a bash events library called bashup.events: https://github.com/search?q=repo%3Aabathur%2Fcomity+events+path%3A**%2Fcomity.*&type=code https://github.com/search?q=repo%3Aabathur%2Fcomity+events+p... Early last year I also wrote a little more generally about the idea of being able to finally build this shell code-sharing ecosystem in https://t-ravis.com/post/shell/neighborly_shell_with_bashup.events/ https://t-ravis.com/post/shell/neighborly_shell_with_bashup....

kristopolous 4y ago

I'm a big fan of not posix but instead modern bash and to all the complainers about dash and ash, I say "tough cookies". Bash has arithmetic, dictionaries, regex matching, nice string manipulation, co-processes, TCP pipes built in, it's pretty nice. However, bash has very low tolerance for incompetency. It's a brutal friend. It's cooking dinner with a pocket knife. You gotta actually read the docs and be sincere in your studies. I was around when supporting Solaris, HPUX, AIX, and Irix was a real concern. Those times sucked and those platforms are so minor I stopped caring about them except for enthusiast projects many years ago. (And I still actually have an hpux machine) Sometimes I'll even use zsh. It can even do floating point. Here's some example of a modern tool I have written for a subject I call "music discovery" https://github.com/kristopolous/music-explorer/tree/master/tools https://github.com/kristopolous/music-explorer/tree/master/t... You'll see many languages in there but you won't see posix sh. It's slow, it's clunky, it's a pain in the ass. If you don't like my practice then I guess don't use it. I've been using/developing these particular tools nearly every day for over 3 years and it works well for me. I'm not going to say bash is awesome but it's pretty great for programming. I use zsh as my interactive shell though, it's miles ahead of bash in that department. Watching people who really know what they're doing in zsh is a total mindfuck.

User23 4y ago
> I was around when supporting Solaris, HPUX, AIX, and Irix was a real concern. Back in the 90s it was surprising to not find gnu userspace installed on the systems I used.
- jkmcf 4y ago
  Back in the 90s, Solaris was the big hold out -- IRIX and HPUX both came with at least a subset. First thing I had to do on Solaris was install the GNU tools, and, thankfully, eventually there were packages to manage that installation.
- kristopolous 4y ago
  About 1/3 of the debug sessions involved looking at the screen peeved going "yes, obviously I meant gsed and ggrep you dumbass." You go through the man page of the vendor supplied version like "What do you mean you don't support feature x? This can't be real. I swear I'm being recorded right now. A bunch of people are sitting around a large conference table in mountain view right now laughing at my pain" Eventually I just read them as parody. Things like this can be really funny if you allow them to be.

RcouF1uZ4gsC 4y ago

Rust may actually be a solution in this space. It is better typed and more testable than Python. It has fewer issues with environmental dependencies due. It is easy to install and build. There is ongoing work to rewrite the coreutils in Rust, and if they were exposed with an ergonomic library interface, would provide a lot of the value of shell with much better safety.

k3vinw 4y ago
For sure! I used to think that python was the next logical step for a growing project that started out life as a shell script, but now I think we have even better choices like rust.
maleldil 4y ago
Rust is _a lot_ slower to develop than sh/bash/Python, though. If your script is in reality a semi-complex CLI tool, then Rust is great, but if it's a 1k line shell script, Python/Ruby/JS (especially with Deno) are better options.

nailer 4y ago

I used to be one of the write .sh people. Now I just ensure my shells scripts end in .bash and don’t insist people limit themselves to an effectively unmaintained standard.

ko1nksm 4y ago

Why not install bash and GNU utilities?

delta_p_delta_x 4y ago

> posix shell is portable: it'll run on debian, ubuntu, on openbsd, in an alpine container, on illumos! arch! even horrible old AIX! > lists three Linux distributions and three Unixes: calls POSIX shell 'portable'. You know where POSIX shell scripts don't run natively? Windows. If you're going to conveniently exclude about half the developer demographic and about 90% of the user demographic to push a not-very-valid point... But given the aesthetic of the website, I somehow get the feeling the author couldn't care less about Windows (or its users). Please use a proper glue language with nice types and syntax, like Python or PowerShell. IMO PowerShell >> POSIX shells and children; interop with full .NET already gives it a several point-lead over POSIX shells. P.S., I notice increasingly more bloggers abandoning sentence case, and it makes for very difficult reading.

tpoacher 4y ago

Opening "man bash" transformed my experience with bash, which has now risen to the top in the list of my favourite languages. I don't get all the hate. I used fish for a while and really liked it, but I realise now this was only because I didn't really know bash. (that and it came with some nice defaults out of the box, like prefix sensitive command history, which requires a .inputrc oneliner for bash) But now every time I try fish I'm like, eh, and immediately switch back to bash. I havent tried zsh, but it just looks gimmicky and "appleish" to me.