9 ms·

I cringe when I hear people (often senior) saying things like: "Yikes, a 500-line shell script. We must rewrite it in Python for maintainability!!!!1!" -- witho

by jmmv 4y ago

I cringe when I hear people (often senior) saying things like: "Yikes, a 500-line shell script. We must rewrite it in Python for maintainability!!!!1!" -- without even looking at how it has been written. Yes. Shell scripts can be terrible, and most are. But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind, it is perfectly possible to write clean, maintainable, and testable shell scripts. Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. (Edited to add a couple of links to related stuff I wrote in the distant past: 1. https://jmmv.dev/series.html#Shell%20readability https://jmmv.dev/series.html#Shell%20readability and 2. https://github.com/jmmv/shtk https://github.com/jmmv/shtk)

eviks 4y ago
> But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind Then one would simply choose a proper language that had more of those solid principles laid at its foundation
- enriquto 4y ago
  Unix shell is a proper language and has very solid principles beautifully laid at its foundation by Douglas McIlroy.
  - wizhi 4y ago
    Do you have some writing on this? I'd love to read it.
  - chasil 4y ago
    I understood that Stephen Borne's love of ALGOL was the chief structural influence of the POSIX shell (so much so that he used the C preprocessor to turn his source code into fake ALGOL). "Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)." https://en.m.wikipedia.org/wiki/Bourne_shell https://en.m.wikipedia.org/wiki/Bourne_shell
als0 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. This is an important distinction, and probably why I still make shell scripts. I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.
- sigzero 4y ago
  I love shellcheck. You can run it locally or via the web if you don't want to install it. https://www.shellcheck.net/ https://www.shellcheck.net/
jonhohle 4y ago
There are even great unit testing frameworks for testing across a variety of shells and versions. This can be integrated pretty easily into existing build tools and run the same way the other code it may (or may not) support is run.
HL33tibCe7 4y ago
> it is perfectly possible to write clean, maintainable, and testable shell scripts. Possible? Maybe. Easy? No. Especially the “testable” part. The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…) So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it? (This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).
- jmmv 4y ago
  I see your point, but what happens in your alternate world of having to maintain the 5000-line Python version 5 years down the line? You face the same issues you describe. The people that wrote the original beautiful code aren't there anymore, and whoever comes touching the code "messes it up". Most Python "scripts" I have had to deal with that had been around for a while were pretty terrible as well.
  - HL33tibCe7 4y ago
    I don’t think you do see my point: my point is that writing clean, testable, maintainable Python is both easier and a more widespread skill than writing equivalently clean, testable and maintainable shell. Both of these factors will mean that the Python remains maintainable for longer.
  - fbdab103 4y ago
    Bad code is bad code, regardless of language. At least with Python, I more or less know all of the idioms and can at a glance understand what the flow of a program is trying to accomplish. Shell has a lot of esoteric voodoo syntax to replicate better string/path/array handling that is just a given in other languages.
  - _dain_ 4y ago
j3s 4y ago
shell is loose and offers a lot of power per line, so it’s pretty easy to make a spaghetti nightmare. but it’s very possible to write nice shell. i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility
- jbverschoor 4y ago
  A program in any language can be maintainable when written properly and maintained by someone who is capable, diligent, and has a good test setup.
- oneshtein 4y ago
  Bash is turing-complete, so it's possible to write automated test cases in bash. Example: https://github.com/vlisivka/bash-modules/blob/master/bash-modules/test/test_string.sh https://github.com/vlisivka/bash-modules/blob/master/bash-mo...
throw0101c 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. The dehydrated ACME client is 2400 lines of bash/zsh: * https://github.com/dehydrated-io/dehydrated https://github.com/dehydrated-io/dehydrated And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell: * https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies: * https://packages.debian.org/bullseye/python3-certbot https://packages.debian.org/bullseye/python3-certbot
- ori_b 4y ago
  I'm not sure why -- the one shipped with 9front is less than a thousand lines of C, and supports both DNS and HTTP challenges: https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd/auth/acmed.c/f.html https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm... http://man.9front.org/8/acmed http://man.9front.org/8/acmed The ACME protocol isn't that complicated.
  - throw0101c 4y ago
    Even with Python things can be written small (200 LoC) it seems: * https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.py https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p... * https://github.com/diafygi/acme-tiny https://github.com/diafygi/acme-tiny * https://packages.debian.org/stable/acme-tiny https://packages.debian.org/stable/acme-tiny
pjmlp 4y ago
Do you a good anecdote regarding shell scripts? Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks. With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such. I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea. Feedback from customer, slower than those scripts and not as easy to change, what a surprise.
- kps 4y ago
  I use Korn shell (ksh93) for quick scripts. Besides arrays and such (it's approximately a superset of bash, since bash copied some but not all of ksh), the killer feature is a declarative extended `getopts` that makes it trivial to provide even the quickest and dirtiest script with short/long option processing with full help, so that the purpose and usage of the script remain comprehensible five days or five years later. https://gist.github.com/kpschoedel/91fdcfa934111f3846efb50341b7342d https://gist.github.com/kpschoedel/91fdcfa934111f3846efb5034...
jamal-kumar 4y ago
From the article: consider this sentence: "arrays do not exist in posix shell" ... "NOOO!" you shout, because you like data structures a LOT "now THAT'S a good reason to use bash! bash supports arrays!" absolutely not! i believe that you should use shell if your problem is: - small and scoped (~200 lines of shell or less) - unlikely to increase in size and scope as time goes on - not very complex if you find yourself desiring arrays, good error handling, static typing, structs, etc, then your problem should be solved using a different language, not shell. I tend to agree with this heuristic. It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.
- djbusby 4y ago
  Perl and Python are part of LSB https://refspecs.linuxfoundation.org/lsb.shtml https://refspecs.linuxfoundation.org/lsb.shtml
  - hedora 4y ago
    Yeah, but perl 5 hasn’t broken compatibility (as far as I’ve noticed) since 2000 or so. Python breaks compatibility every week, in practice. Does LSB include some sort of magical frozen in time version of python 2 or something?
    - gabrielhidasy 4y ago
      In my 12 years of python I was hit with compatibility problems once, in a school project where someone started in python 2 and we changed for python 3. What libraries break that often so I can stay away?
      - hedora 4y ago
        scipy / numpy / etc broke multiple times for me. I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.
jbverschoor 4y ago
The reason I don’t like shell scripts are spaces and the weird constructs sometimes. I never got into Perl, but I understand why it was popular. Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert. Shell is nice not because of the shell language, but the ease of executing and chaining other programs.
pushedx 4y ago
The shell provides an interface to the filesystem and concurrent pipelines like no other readily-available language. Many tasks are more concisely expressed and readable in shell than in any other language.
- yjftsjthsd-h 4y ago
  Yep; shell isn't the best tool for every job, but the things it's good at, it's really good at.
- lapinot 4y ago
  The fact that posix shell is somehow state of the art of concurrent pipeline and filesystem interface is really more a critique of state of the art than a praise of shell. I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important. Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).
  - lozenge 4y ago
    Powershell is the new shell. As it passes objects the filtering, sorting, joining is generic across files or any other data type.
    - 4y ago
ra1231963 4y ago
This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t. Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.
- jmmv 4y ago
  Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
  - kaba0 4y ago
    I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
    - mananaysiempre 4y ago
      > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)
d0mine 4y ago
> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.
m1keil 4y ago
The question is if the script outgrew its original intended lifespan. In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense. So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.
simias 4y ago
100% disagree. Standard /bin/sh has terrible error handling facility, which on its own is enough to disqualify it as a "proper" programming language in my book. Even C is not that bad, and C is pretty terrible. I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine. The shell is not a real language and must be written as such.
- alganet 4y ago
  There's no such thing as "standard /bin/sh". You're talking about busybox ash, or dash, or something else that is shipped as /bin/sh in your distro. Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell. In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.
  - simias 4y ago
    I meant POSIX shell or more specifically a loosely defined subset of all these shell dialects that is reasonably portable in practice. I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies. >In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6... But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking. Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.
sureglymop 4y ago
I cringe everytime my manager tells me not to write shell scripts but javascript instead. There is ShellCheck, Bats and there are extensions for e.g. Visual Studio Code or Jetbrains IDEA that make writing shell and bash scripts a pleasure. I find it generally much more productive to write a shell script because I already know all the gnu/unix userland tools. And i know this shell script will be portable and still running decades from now unless some version specific node script with dependencies.

by jmmv 4y ago

eviks 4y ago

> But if one approaches them thinking that “the shell is a real language and must be written as such”, with solid principles in mind Then one would simply choose a proper language that had more of those solid principles laid at its foundation

enriquto 4y ago
Unix shell is a proper language and has very solid principles beautifully laid at its foundation by Douglas McIlroy.
- wizhi 4y ago
  Do you have some writing on this? I'd love to read it.
- chasil 4y ago
  I understood that Stephen Borne's love of ALGOL was the chief structural influence of the POSIX shell (so much so that he used the C preprocessor to turn his source code into fake ALGOL). "Stephen Bourne's coding style was influenced by his experience with the ALGOL 68C compiler that he had been working on at Cambridge University... Moreover, – although the v7 shell is written in C – Bourne took advantage of some macros to give the C source code an ALGOL 68 flavor. These macros (along with the finger command distributed in Unix version 4.2BSD) inspired the International Obfuscated C Code Contest (IOCCC)." https://en.m.wikipedia.org/wiki/Bourne_shell https://en.m.wikipedia.org/wiki/Bourne_shell

als0 4y ago

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. This is an important distinction, and probably why I still make shell scripts. I've found utilities like `shellcheck` invaluable for making robust scripts and maintaining them.

sigzero 4y ago
I love shellcheck. You can run it locally or via the web if you don't want to install it. https://www.shellcheck.net/ https://www.shellcheck.net/

jonhohle 4y ago

There are even great unit testing frameworks for testing across a variety of shells and versions. This can be integrated pretty easily into existing build tools and run the same way the other code it may (or may not) support is run.

HL33tibCe7 4y ago

> it is perfectly possible to write clean, maintainable, and testable shell scripts. Possible? Maybe. Easy? No. Especially the “testable” part. The vast majority of engineers do not know how to write clean shell, let alone testable shell. There’s a reason for this: it’s extremely difficult and non-obvious to do, and generally isn’t worth investing the time in learning (you can just use a different language instead: perhaps one with support for advanced features like… arrays…) So sure, you can write your 500 line shell script and it may well end up being beautiful. But what happens in 5 years when someone who isn’t as enlightened as you has to maintain it? (This all without mentioning the worst aspect of shell scripting: security. Good luck handling secrets without leaking them in one way or another).

jmmv 4y ago
I see your point, but what happens in your alternate world of having to maintain the 5000-line Python version 5 years down the line? You face the same issues you describe. The people that wrote the original beautiful code aren't there anymore, and whoever comes touching the code "messes it up". Most Python "scripts" I have had to deal with that had been around for a while were pretty terrible as well.
- HL33tibCe7 4y ago
  I don’t think you do see my point: my point is that writing clean, testable, maintainable Python is both easier and a more widespread skill than writing equivalently clean, testable and maintainable shell. Both of these factors will mean that the Python remains maintainable for longer.
- fbdab103 4y ago
  Bad code is bad code, regardless of language. At least with Python, I more or less know all of the idioms and can at a glance understand what the flow of a program is trying to accomplish. Shell has a lot of esoteric voodoo syntax to replicate better string/path/array handling that is just a given in other languages.
- _dain_ 4y ago

j3s 4y ago

shell is loose and offers a lot of power per line, so it’s pretty easy to make a spaghetti nightmare. but it’s very possible to write nice shell. i dunno about testing it, i feel like that’s getting pretty close to the barrier of shell’s utility

jbverschoor 4y ago
A program in any language can be maintainable when written properly and maintained by someone who is capable, diligent, and has a good test setup.
oneshtein 4y ago
Bash is turing-complete, so it's possible to write automated test cases in bash. Example: https://github.com/vlisivka/bash-modules/blob/master/bash-modules/test/test_string.sh https://github.com/vlisivka/bash-modules/blob/master/bash-mo...

throw0101c 4y ago

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. The dehydrated ACME client is 2400 lines of bash/zsh: * https://github.com/dehydrated-io/dehydrated https://github.com/dehydrated-io/dehydrated And its external dependencies are OpenSSL and cURL. The acme.sh shell ACME client is 8000 lines of shell: * https://github.com/acmesh-official/acme.sh https://github.com/acmesh-official/acme.sh The official Let's Encrypt client is written in Python, and the core 'executable' is much longer, and in addition it pulls in a boatload of dependencies: * https://packages.debian.org/bullseye/python3-certbot https://packages.debian.org/bullseye/python3-certbot

ori_b 4y ago
I'm not sure why -- the one shipped with 9front is less than a thousand lines of C, and supports both DNS and HTTP challenges: https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd/auth/acmed.c/f.html https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm... http://man.9front.org/8/acmed http://man.9front.org/8/acmed The ACME protocol isn't that complicated.
- throw0101c 4y ago
  Even with Python things can be written small (200 LoC) it seems: * https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.py https://github.com/diafygi/acme-tiny/blob/master/acme_tiny.p... * https://github.com/diafygi/acme-tiny https://github.com/diafygi/acme-tiny * https://packages.debian.org/stable/acme-tiny https://packages.debian.org/stable/acme-tiny

pjmlp 4y ago

Do you a good anecdote regarding shell scripts? Long time ago some dude created a couple of korn shell scripts on Aix to automate file uploads and ETL related tasks. With time that work moved from Aix into Red-Hat Linux, still korn shell, with the adaptations from Aix into GNU/Linux command line parameters and such. I got tasked to port them into Java, because the team that got the job to maintain the servers lacked UNIX experience, and they thought selling the porting project was a good idea. Feedback from customer, slower than those scripts and not as easy to change, what a surprise.

kps 4y ago
I use Korn shell (ksh93) for quick scripts. Besides arrays and such (it's approximately a superset of bash, since bash copied some but not all of ksh), the killer feature is a declarative extended `getopts` that makes it trivial to provide even the quickest and dirtiest script with short/long option processing with full help, so that the purpose and usage of the script remain comprehensible five days or five years later. https://gist.github.com/kpschoedel/91fdcfa934111f3846efb50341b7342d https://gist.github.com/kpschoedel/91fdcfa934111f3846efb5034...

jamal-kumar 4y ago

From the article: consider this sentence: "arrays do not exist in posix shell" ... "NOOO!" you shout, because you like data structures a LOT "now THAT'S a good reason to use bash! bash supports arrays!" absolutely not! i believe that you should use shell if your problem is: - small and scoped (~200 lines of shell or less) - unlikely to increase in size and scope as time goes on - not very complex if you find yourself desiring arrays, good error handling, static typing, structs, etc, then your problem should be solved using a different language, not shell. I tend to agree with this heuristic. It's also why you see people bothering to use perl 5 at all in this day and age. It just happens to be preinstalled on a decent majority of *nix systems.

djbusby 4y ago
Perl and Python are part of LSB https://refspecs.linuxfoundation.org/lsb.shtml https://refspecs.linuxfoundation.org/lsb.shtml
- hedora 4y ago
  Yeah, but perl 5 hasn’t broken compatibility (as far as I’ve noticed) since 2000 or so. Python breaks compatibility every week, in practice. Does LSB include some sort of magical frozen in time version of python 2 or something?
  - gabrielhidasy 4y ago
    In my 12 years of python I was hit with compatibility problems once, in a school project where someone started in python 2 and we changed for python 3. What libraries break that often so I can stay away?
    - hedora 4y ago
      scipy / numpy / etc broke multiple times for me. I’m usually dealing with other people’s python, so I don’t remember which ones broke. Usually stuff around 3rd party services, async I/O or parsing.

jbverschoor 4y ago

The reason I don’t like shell scripts are spaces and the weird constructs sometimes. I never got into Perl, but I understand why it was popular. Sure it’s possible, but it’s therefore also brittle, when someone touches it and is not a complete expert. Shell is nice not because of the shell language, but the ease of executing and chaining other programs.

pushedx 4y ago

The shell provides an interface to the filesystem and concurrent pipelines like no other readily-available language. Many tasks are more concisely expressed and readable in shell than in any other language.

yjftsjthsd-h 4y ago
Yep; shell isn't the best tool for every job, but the things it's good at, it's really good at.
lapinot 4y ago
The fact that posix shell is somehow state of the art of concurrent pipeline and filesystem interface is really more a critique of state of the art than a praise of shell. I'm doing programming language research and it's just infuriating how retrograde shell is. Its design is a fossil of history, and boy did our understanding of what makes a good language evolve. As a proof of this look at the jq tool. It is so powerful and clean. The implementation is simple and reasonably fast. It has been designed by a researcher from functional languages (an ocaml guy). This is a way to build a language around pipelining. At a lesser level, look at how moving from vimscript to lua in neovim has brought so many high-quality new plugins, look at how many people now tinker with low-level stuff and write high-perf libraries now that C/C++ is not the only choice. Surface language and having very little very orthogonal concepts is important. Some core implementation patterns for PL are very old, but there have been lots of simplifications and rationalizations in surface language. Error handling, values hygiene, scoping, simple datastructures. Posix shell might have been a prowess in the 70s, now it is mostly a hack and this is a fact. Let's be a bit less superficial when honoring the unix mantras and dream about something better (making a better shell for its core ideas(!), that is, handling processes and fds).
- lozenge 4y ago
  Powershell is the new shell. As it passes objects the filtering, sorting, joining is generic across files or any other data type.
  - 4y ago

ra1231963 4y ago

This is terrible advice. Senior people say don’t do this because they’ve seen it done and it’s an absolute nightmare (I’ve seen it happen multiple times). It’s untyped, untestable, arcane, and unmaintabale. Even the link says don’t do this. Just don’t. Congrats on having the chops (and neckbeard) to pull this off. If you want to do it in a personal project, more power to you. But it doesn’t scale at organizations.

jmmv 4y ago
Anyone writing code in a language they don't know will write nightmarish code. No matter if it is shell, Python, Go, or what have you. I have seen it happen multiple times as well, with all of these languages. And /that/'s the problem. So, it depends on the context. "Organization" is a very broad term. If the team you are in is familiar with the shell, and the shell is the right tool for the job, there isn't that much of a problem. I'm not arguing for writing large shell scripts when there are better alternatives. But, sometimes, it's the right choice and it can be written properly. And even if it isn't... 500 lines of poorly written shell aren't that many in any case, and they aren't that bad compared to 5000 lines of poorly written Python for example.
- kaba0 4y ago
  I fail to see why you think it would take more lines to do in Python. And the problem with bash is similar to C: many people think they can write correct C code, but as it has been shown plenty of times, that doesn’t really work out in the end. Languages have different tradeoffs and bash really has no redeeming qualities, imo.
  - mananaysiempre 4y ago
    > I fail to see why you think it would take more lines to do in Python. Because Bourne shell is not a Python-class language like Ruby or perhaps even Scheme or AWK are. Things that are easy in shell can be hard in Python and vice versa: awkward wrangling of subprocess.Popen in Python can be a single line of shell; simple tree munging with lxml in Python can easily be a hundred or more lines of wasteful xmlstarlet invocations in shell. Parallel processing, even without the heavy artillery of make or GNU parallel, is easy in shell but hard in Python. Structured data is hard in shell but easy in Python. (Similarly, I wouldn’t use ML or Haskell to write a web scraper, and I wouldn’t use Python to write a compiler.)

d0mine 4y ago

> Oh, and that 500-line shell script probably ends up being a 5000-line Python monster anyway. Agreed on avoiding rewrites if possible but 1/10 ratio is laughable.

m1keil 4y ago

The question is if the script outgrew its original intended lifespan. In my opinion, shell scripts should be used either as glue or due to portability reasons. Rewriting a perfectly good and working script just because makes no sense. So if the script started as some quick and easy way to deploy to prod but slowly becomes a massive cli "one tool to rule them all" kind of thing.. maybe it's time to change course.

simias 4y ago

100% disagree. Standard /bin/sh has terrible error handling facility, which on its own is enough to disqualify it as a "proper" programming language in my book. Even C is not that bad, and C is pretty terrible. I'm proficient at POSIX shell scripting and use it all the time, but every time I discover that a tool I need to maintain or, god forbid, improve upon is written in pure shell I immediately get a migraine. The shell is not a real language and must be written as such.

alganet 4y ago
There's no such thing as "standard /bin/sh". You're talking about busybox ash, or dash, or something else that is shipped as /bin/sh in your distro. Take a look at winetricks, or clitest, or crossroot-ng, or shellspec. They are modular, easy to read, well written pieces of software that are written in shell. In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were wrong, because they only saw browser differences and the issues with it, not the great things about it. The bourne shell is in a similar state.
- simias 4y ago
  I meant POSIX shell or more specifically a loosely defined subset of all these shell dialects that is reasonably portable in practice. I'm not saying that it's impossible to do, I'm saying that it's very difficult, requires every contributor to be extremely well versed into shell coding and the particular convention of the codebase, and small issues can easily snowball out of control. It's doable, but it's almost never worth it IMO. One notable exception is when you need max portability with effectively zero dependencies. >In the 2000s, people used to say that it was impossible to write large stuff in JavaScript, and it was not a real language. They were partly right, which is why modern web development bears very little resemblance with early 2000's JS hacking. Remember that even JQuery only appeared in 2006. And those were the dark days of IE6... But IMO that's not exactly the same thing: the issue with Javascript was less with its syntax and more with the very poor Javascript API and standard lib (or lack thereof), shell scripting is the other way around in a sense: you have access to all the power of the OS and all the programs in it, but it's the actual programming language that's lacking. Actually a decent language that would transpile to POSIX sh would be interesting, now that I think about it.

sureglymop 4y ago

I cringe everytime my manager tells me not to write shell scripts but javascript instead. There is ShellCheck, Bats and there are extensions for e.g. Visual Studio Code or Jetbrains IDEA that make writing shell and bash scripts a pleasure. I find it generally much more productive to write a shell script because I already know all the gnu/unix userland tools. And i know this shell script will be portable and still running decades from now unless some version specific node script with dependencies.