5 ms·

I've always been confused that in ${current_year} people are still directly writing huge + complex shell scripts. Compare/contrast Javascript, another language

by derefr 5y ago

I've always been confused that in ${current_year} people are still directly writing huge + complex shell scripts. Compare/contrast Javascript, another language we're "stuck with": while you can still directly write it, many people don't, instead using transpilers to target Javascript as an "object code" from some other, stricter language (e.g. TypeScript, ClojureScript, arbitrary native languages via Emscripten / WASM compiler targets, etc.) Via these languages/compilers, everybody who wants compile-time strictness can have it. And you can even get runtime strictness that Javascript itself doesn't have, as the checked semantics of these languages will often be "lowered" into the resulting Javascript by generating explicit assertions (usually this is on by default, unless you ask for an optimized build.) But the output is still just "Javascript", that any browser can run. Why hasn't a similar thing happened for POSIX-standard Bourne shell? Why are we seemingly "satisfied" with writing + maintaining 5000-line install.sh scripts in our codebases, with a bunch of extra mental overhead / tooling required to keep the code sane and clean; when we could just be writing in a sane and clean language to begin with? Is it just that nobody's bothered to create such a language/compiler? Do I need to be the one to step up, here?

airza 5y ago
I don't think I understand. Javascript is required in the browser as the end target, but the operating system exposes the APIs which are consumed by bash to do useful work. The normal solution to not wanting to use bash is to... use another language (including, ironically, javascript).
- derefr 5y ago
  Much of the time that people are writing shell scripts, they're writing them not because they prefer shell syntax to that of some other language, but rather because they're creating a script that needs to be widely disseminated/deployed to all sorts of machines with unpredictable toolchain availability. This is why a large fraction of the shell scripts that exist in the world still hold to Bourne shell syntax, rather than using any of the syntax extensions from its descendant shells: Bourne shell (or at least, something at /bin/sh that interprets Bourne-shell syntax) is part of the POSIX standard. So you can expect any POSIX system — no matter how weird — to be able to run (Bourne) shell scripts. You can run them on Alpine. You can run them on Busybox. You can run them on your NAS. You can run them on your router. You can run them in your initramfs, on your Kubernetes nodes, on your Mosix cluster, on your smart TV, on your watch, whatever. They run on Windows; they run on macOS; they (obviously) run on Linux and BSD. New Bourne shell scripts written today, run on 40-year-old operating systems nobody even remembers. (If you think about it, the whole way that GNU autotools does its job, is by relying on the portability of POSIX-compliant Bourne shell syntax — i.e. the ./configure script — to bootstrap out enough of an understanding of whatever system it lands on, to find the tools required to then probe the system for the availability + capabilities of its compilation toolchain.) For a small example of the type of script I'm talking about — see e.g. the script you download+run when you run the command-line on https://rustup.rs https://rustup.rs: https://github.com/hsivonen/rustup.rs/blob/master/rustup-init.sh https://github.com/hsivonen/rustup.rs/blob/master/rustup-ini.... For a much better example (that I sadly can't link to), see the install.sh for VMWare Workstation for Linux. There's absolutely no benefit that these scripts get from being developed directly in POSIX-compiliant Bourne shell syntax, rather than being developed in something that compiles to said syntax; any more than programs for your PC would benefit from being developed directly in ASM, rather than in something that compiles to it.
wging 5y ago
The niche for such a language would be extremely narrow compared to existing better-than-shell-script alternatives. Wasn't that perl's selling point? Compiling to bash doesn't seem like that much of a better option than shipping (or just requiring implicitly) a $BETTER_LANGUAGE runtime, and writing in that language. (I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare.) My go-to is either Python or Ruby when shell scripts start to get unwieldy - lately it's been Ruby but I'm mentioning Python because I have experience with the type-checking options for Python, and you mentioned that as a benefit. I'm sure someone will point out that type-checking exists for Ruby too (Sorbet, others?); I've just never tried it.
- j1elo 5y ago
  > I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare. Not to comment on how frequent this is for other people, but my rare, obscure, and carefully constructed scenario is this one: docker run --rm -it ubuntu
noahtallen 5y ago
I don’t think many are happy writing that much bash. I certainly wouldn’t be! I think this problem is completely solved by other languages though. Bash is popular because it’s available (nearly) everywhere, is very portable, and has a massive “standard library.” (Technically, not a standard library, but you can basically guarantee that things like sed, curl, rsync etc. will be available or easy to get anywhere.) And its constructs are very well fitted to smaller scripts and automations. You loose those benefits with a compilation step. Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo. These scripts are used for the build steps of other programs. So what would you write for automations related to building a compiled bash project? Probably bash lol If you’re writing thousands of lines, you can just use any other language, right? Pretty much any other scripting language can execute commands in bash anyways. To my knowledge, stepping into a different language once you need more refined flow control and data structures is pretty normal. Afaik many languages can either produce output which can be run directly on the CLI (rust, C++, C), and others can be executed if you have the language runtime installed (JS via node, python, etc.)
- derefr 5y ago
  > Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo. What you're talking about are scripts you (or developers you work closely with) write to then use yourselves, in development workflows. These tend to be small and simple, and also places where the pain-points of shell-scripting (e.g. spaces screwing up path parsing) don't tend to come up, because developers have enough sense to not make their development environment actively hostile in these ways. But this is not the only, or even the main, use-case of writing Bourne shell scripts. (If you have a development toolchain installed, you probably already have Perl and/or Python and/or Ruby — so why not script your development workflow in one of those? Or, if everyone is at least running a modern Linux or macOS+Homebrew, why not assume the existence of Bash or Zsh, and write for one of those, instead of Bourne shell?) Instead, the main places where large, complex shell scripts are necessary, are in: 1. multi-platform software distribution; 2. systems-level glue code (think: initrd scripts; sysvinit back when that was a thing; etc) that must be runnable "at early boot" or in a "rescue environment" — i.e. one that can't guarantee that /usr is mounted, or that /lib is set up correctly to make non-static binaries work, etc. (Or in an "embedded environment" — in Busybox, /usr doesn't even exist!) In the first case, the person writing the shell script is not the person running the shell script. Tons of different people run the shell script, on (sometimes drastically) different environments, and these are not developers, and their machines mostly aren't guaranteed to have anything like a development toolchain installed. In the second case, despite targeting a single environment, said environment just happens to have a Bourne-shell compatible interpreter as its only scripting environment. (And probably even that was only included because the environment wanted to claim POSIX compatibility, never expecting people to want to script it rather than compiling native tools statically into the environment. Even though most sysadmins who might need to modify these environments, have no idea how to do that.) Mind you, some of these systems — e.g. UEFI — aren't POSIX-compatible at all, but rather require you to "script" them in something totally different, e.g. Forth. > If you’re writing thousands of lines, you can just use any other language, right? No, not if you can't make any assumptions about the target environment beyond "it's POSIX"; or if the target environment really doesn't offer anything not strictly required by POSIX. There is no POSIX-standard scripting environment other than Bourne shell. (And you can't just ship a binary, because you don't know the target architecture. Detecting the architecture you've landed on is probably part of what you're trying to do!)
naniwaduni 5y ago
This has been tried before. One of the results is autoconf. At the end of the day, it's probably less painful to just bootstrap some saner interfaces if you really need them. The POSIX C API is just as standardized and exposes more than the shell interfaces.
- derefr 5y ago
  That assumes you're targeting systems with a local compiler toolchain. A valid assumption for autotools itself, but "having a C compiler available" isn't a requirement of POSIX. (POSIX merely standardizes the interface that such a toolchain should present, if one is available.) I reiterate my example: the install.sh for the Linux edition of VMWare Workstation. Its job is to take an arbitrary machine running a desktop Linux OS (i.e. it does assume an X installation), and turn it into a hypervisor dom0. To do this, it needs to compile its proprietary Linux kernel modules using DKMS. But 1. it can't assume that you have a compiler toolchain installed; and 2. it can't assume that you have the kernel header package installed for your kernel. So this install script needs to detect your OS distro-family, and use that to determine how to forcibly install some packages, in order to then have the tools (and headers) available to compile things. How would you suggest that this installation logic be implemented? Because right now it's "thousands of lines of extremely grody Bourne shell script" — and AFAICT, that's because that was the optimal choice given the problem domain. I suppose you could build such an installer as a collection of static binaries, one for each architecture. (Static, because different distros name their libs differently, so dynamic linkage isn't portable.) You could then create a little install.sh that runs them. And, if you wanted all this to be tucked into a single file so you could run it with the unsafe curl-to-shell hack, you could even base64-armor those binaries and drop them into some heredocs inside a bootstrap shell-script, such that you do get to have a single "install.sh", which just pops out copy of the 'correct' binary into existence and runs it. But IMHO, by doing something like this, you're just Greenspunning a Bourne-shell object-code format; the same one a compiler could emit. But without the benefit of getting to directly use a linker to generate the result.
  - deleted 5y ago
    [deleted]
machiaweliczny 5y ago
Sane people don't write shell scripts. I script everything little more complex with Ruby/JS.
CannisterFlux 5y ago
Oh people have tried - here are a few https://stackoverflow.com/questions/10239235/are-there-any-languages-that-compile-to-bash https://stackoverflow.com/questions/10239235/are-there-any-l... I vaguely remember quite liking bish when I saw it years ago https://github.com/tdenniston/bish https://github.com/tdenniston/bish but it looks like no commits in 6 years. This shelljs thing looks more promising, but really tedious to use https://github.com/shelljs/shelljs https://github.com/shelljs/shelljs - shell.rm('-rf', 'out/Release'); I'd rather suffer proper bash than have to do that sort of thing. Nothing seems to have really caught on so far. Bash is easy to learn and hack on, and before you know it, that simple install.sh that started out moving a few files around is 5000 lines, unmaintainable, and critical to bootstrapping your software :)

by derefr 5y ago

airza 5y ago

I don't think I understand. Javascript is required in the browser as the end target, but the operating system exposes the APIs which are consumed by bash to do useful work. The normal solution to not wanting to use bash is to... use another language (including, ironically, javascript).

derefr 5y ago
Much of the time that people are writing shell scripts, they're writing them not because they prefer shell syntax to that of some other language, but rather because they're creating a script that needs to be widely disseminated/deployed to all sorts of machines with unpredictable toolchain availability. This is why a large fraction of the shell scripts that exist in the world still hold to Bourne shell syntax, rather than using any of the syntax extensions from its descendant shells: Bourne shell (or at least, something at /bin/sh that interprets Bourne-shell syntax) is part of the POSIX standard. So you can expect any POSIX system — no matter how weird — to be able to run (Bourne) shell scripts. You can run them on Alpine. You can run them on Busybox. You can run them on your NAS. You can run them on your router. You can run them in your initramfs, on your Kubernetes nodes, on your Mosix cluster, on your smart TV, on your watch, whatever. They run on Windows; they run on macOS; they (obviously) run on Linux and BSD. New Bourne shell scripts written today, run on 40-year-old operating systems nobody even remembers. (If you think about it, the whole way that GNU autotools does its job, is by relying on the portability of POSIX-compliant Bourne shell syntax — i.e. the ./configure script — to bootstrap out enough of an understanding of whatever system it lands on, to find the tools required to then probe the system for the availability + capabilities of its compilation toolchain.) For a small example of the type of script I'm talking about — see e.g. the script you download+run when you run the command-line on https://rustup.rs https://rustup.rs: https://github.com/hsivonen/rustup.rs/blob/master/rustup-init.sh https://github.com/hsivonen/rustup.rs/blob/master/rustup-ini.... For a much better example (that I sadly can't link to), see the install.sh for VMWare Workstation for Linux. There's absolutely no benefit that these scripts get from being developed directly in POSIX-compiliant Bourne shell syntax, rather than being developed in something that compiles to said syntax; any more than programs for your PC would benefit from being developed directly in ASM, rather than in something that compiles to it.

wging 5y ago

The niche for such a language would be extremely narrow compared to existing better-than-shell-script alternatives. Wasn't that perl's selling point? Compiling to bash doesn't seem like that much of a better option than shipping (or just requiring implicitly) a $BETTER_LANGUAGE runtime, and writing in that language. (I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare.) My go-to is either Python or Ruby when shell scripts start to get unwieldy - lately it's been Ruby but I'm mentioning Python because I have experience with the type-checking options for Python, and you mentioned that as a benefit. I'm sure someone will point out that type-checking exists for Ruby too (Sorbet, others?); I've just never tried it.

j1elo 5y ago
> I imagine someone might respond with some carefully constructed scenario where you can only use a shell script, but I'd expect that sort of scenario to be rare. Not to comment on how frequent this is for other people, but my rare, obscure, and carefully constructed scenario is this one: docker run --rm -it ubuntu

noahtallen 5y ago

I don’t think many are happy writing that much bash. I certainly wouldn’t be! I think this problem is completely solved by other languages though. Bash is popular because it’s available (nearly) everywhere, is very portable, and has a massive “standard library.” (Technically, not a standard library, but you can basically guarantee that things like sed, curl, rsync etc. will be available or easy to get anywhere.) And its constructs are very well fitted to smaller scripts and automations. You loose those benefits with a compilation step. Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo. These scripts are used for the build steps of other programs. So what would you write for automations related to building a compiled bash project? Probably bash lol If you’re writing thousands of lines, you can just use any other language, right? Pretty much any other scripting language can execute commands in bash anyways. To my knowledge, stepping into a different language once you need more refined flow control and data structures is pretty normal. Afaik many languages can either produce output which can be run directly on the CLI (rust, C++, C), and others can be executed if you have the language runtime installed (JS via node, python, etc.)

derefr 5y ago
> Nearly all the scripts I interact with are just smaller automations put into a “bin” folder in the repo. What you're talking about are scripts you (or developers you work closely with) write to then use yourselves, in development workflows. These tend to be small and simple, and also places where the pain-points of shell-scripting (e.g. spaces screwing up path parsing) don't tend to come up, because developers have enough sense to not make their development environment actively hostile in these ways. But this is not the only, or even the main, use-case of writing Bourne shell scripts. (If you have a development toolchain installed, you probably already have Perl and/or Python and/or Ruby — so why not script your development workflow in one of those? Or, if everyone is at least running a modern Linux or macOS+Homebrew, why not assume the existence of Bash or Zsh, and write for one of those, instead of Bourne shell?) Instead, the main places where large, complex shell scripts are necessary, are in: 1. multi-platform software distribution; 2. systems-level glue code (think: initrd scripts; sysvinit back when that was a thing; etc) that must be runnable "at early boot" or in a "rescue environment" — i.e. one that can't guarantee that /usr is mounted, or that /lib is set up correctly to make non-static binaries work, etc. (Or in an "embedded environment" — in Busybox, /usr doesn't even exist!) In the first case, the person writing the shell script is not the person running the shell script. Tons of different people run the shell script, on (sometimes drastically) different environments, and these are not developers, and their machines mostly aren't guaranteed to have anything like a development toolchain installed. In the second case, despite targeting a single environment, said environment just happens to have a Bourne-shell compatible interpreter as its only scripting environment. (And probably even that was only included because the environment wanted to claim POSIX compatibility, never expecting people to want to script it rather than compiling native tools statically into the environment. Even though most sysadmins who might need to modify these environments, have no idea how to do that.) Mind you, some of these systems — e.g. UEFI — aren't POSIX-compatible at all, but rather require you to "script" them in something totally different, e.g. Forth. > If you’re writing thousands of lines, you can just use any other language, right? No, not if you can't make any assumptions about the target environment beyond "it's POSIX"; or if the target environment really doesn't offer anything not strictly required by POSIX. There is no POSIX-standard scripting environment other than Bourne shell. (And you can't just ship a binary, because you don't know the target architecture. Detecting the architecture you've landed on is probably part of what you're trying to do!)

naniwaduni 5y ago

This has been tried before. One of the results is autoconf. At the end of the day, it's probably less painful to just bootstrap some saner interfaces if you really need them. The POSIX C API is just as standardized and exposes more than the shell interfaces.

derefr 5y ago
That assumes you're targeting systems with a local compiler toolchain. A valid assumption for autotools itself, but "having a C compiler available" isn't a requirement of POSIX. (POSIX merely standardizes the interface that such a toolchain should present, if one is available.) I reiterate my example: the install.sh for the Linux edition of VMWare Workstation. Its job is to take an arbitrary machine running a desktop Linux OS (i.e. it does assume an X installation), and turn it into a hypervisor dom0. To do this, it needs to compile its proprietary Linux kernel modules using DKMS. But 1. it can't assume that you have a compiler toolchain installed; and 2. it can't assume that you have the kernel header package installed for your kernel. So this install script needs to detect your OS distro-family, and use that to determine how to forcibly install some packages, in order to then have the tools (and headers) available to compile things. How would you suggest that this installation logic be implemented? Because right now it's "thousands of lines of extremely grody Bourne shell script" — and AFAICT, that's because that was the optimal choice given the problem domain. I suppose you could build such an installer as a collection of static binaries, one for each architecture. (Static, because different distros name their libs differently, so dynamic linkage isn't portable.) You could then create a little install.sh that runs them. And, if you wanted all this to be tucked into a single file so you could run it with the unsafe curl-to-shell hack, you could even base64-armor those binaries and drop them into some heredocs inside a bootstrap shell-script, such that you do get to have a single "install.sh", which just pops out copy of the 'correct' binary into existence and runs it. But IMHO, by doing something like this, you're just Greenspunning a Bourne-shell object-code format; the same one a compiler could emit. But without the benefit of getting to directly use a linker to generate the result.
- deleted 5y ago
  [deleted]

machiaweliczny 5y ago

Sane people don't write shell scripts. I script everything little more complex with Ruby/JS.

CannisterFlux 5y ago

Oh people have tried - here are a few https://stackoverflow.com/questions/10239235/are-there-any-languages-that-compile-to-bash https://stackoverflow.com/questions/10239235/are-there-any-l... I vaguely remember quite liking bish when I saw it years ago https://github.com/tdenniston/bish https://github.com/tdenniston/bish but it looks like no commits in 6 years. This shelljs thing looks more promising, but really tedious to use https://github.com/shelljs/shelljs https://github.com/shelljs/shelljs - shell.rm('-rf', 'out/Release'); I'd rather suffer proper bash than have to do that sort of thing. Nothing seems to have really caught on so far. Bash is easy to learn and hack on, and before you know it, that simple install.sh that started out moving a few files around is 5000 lines, unmaintainable, and critical to bootstrapping your software :)