6 ms·

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" a

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been. I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?

blabitty 6y ago
I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.
- kubanczyk 6y ago
  Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.
  - fillest 6y ago
    It is OK to call whatever one needs actually - just minimize the usage of Bash language features (Python provides most of them). It is not complicated and requires no dependencies, here's a self-contained helper: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39d453 https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39...
    - imiric 6y ago
      Great, would that work with pipes and redirections too? At that point you might as well use sh[1] which is a more advanced abstraction. My issue with these approaches is that you'll inevitably hit a bug or limitation where just using a shell language would've been easier to maintain and more portable. Python has its uses, but replacing shell scripts is not one of them IMO. It it was then the problem didn't need to be a shell script to begin with (doesn't shell out often, needs complex data structures, etc.). [1]: https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
  - stormbrew 6y ago
    It's worth noting that there are still a shocking number of production machines in the world that don't have stable installations of python3.x yet.
    - pmiller2 6y ago
      So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.
      - stormbrew 6y ago
        So, a bash script written ten years ago probably still works on any version of bash since then. A python script written ten years ago probably doesn't even parse on python3.0, let alone python3.6 (because there have been backwards incompatible changes since 3.0 even!). And that's also true vice versa. So like, this is to the point that there's some "reasonable subset" of python you can use to make it portable (ie. no external packages so you don't need to worry about pip vs. setuptools or venvs or whatever, let alone whether they'll build). I'm asserting that there is not.
- ptx 6y ago
  > file and directory manipulations are way more complicated in native Python Have you looked at the pathlib[1] module that was added in Python 3.4? It's fantastic. Makes things both more convenient and more correct, in my experience, and it lets you manipulate Windows paths on Unix and vice versa. [1] https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/pathlib.html
marcosdumay 6y ago
Hum, no those people are probably writing the same kind of script you do. Their Python versions are indeed longer and more verbose. I always recommend Python anyway if you care about edge cases (what is not always). That longer script handles them on the obvious way, while the short and more readable shell script does something absolutely crazy every time a detail is different from planed. And if you try to correctly handle the edge cases in Bash (you'll fail), you'll get something much more complicated than the Python version anyway.
- rectang 6y ago
  Which of these is simpler? Rust: fn add(a: i32, b: i32) -> i32 { a + b } Python: def add(a, b): return a + b Arguably the Python function is "simpler" because it occupies fewer characters. And yet, the Rust function is more tightly specified; it does less. It can't throw exceptions, unlike the Python function. Invalid programs where you pass the wrong type to the Rust function won't even compile, whereas the Python function will blow up at runtime. I figure that these supposedly "simpler" Bash programs are not simple at all.
  - marcosdumay 6y ago
    Oh, I'm really not comparing it with Rust. The comparison is between a general language and a shell. If it's Python, Rust, PHP, Java, or whatever is of secondary relevance. I've done some ops work in C, Python, and C#. I'm also waiting for a good problem to try some Haskell shell monad. All the same old language pros and cons apply.
    - rectang 6y ago
      I'm just making a general argument about what "simple" means. A well-specified program may or may not be superficially simpler, but it will exhibit simpler behavior.
      - marcosdumay 6y ago
        Oh, ok. The Rust version is at least as simple as the Python one, depending on what kinds of detail you care about. The thing about shell code is that there are plenty of cases where you don't care about things like error management, bad inputs, and etc. So it's not immediately obvious what language is simpler. When you care about those, the shell becomes the most complex choice by far, and that's the duality problem that generates all those different opinions you see around.
dragonwriter 6y ago
> I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"? A replacement for a shell script may not itself be a shell script. A fairly typical python-as-shell-replacement I have that (among other things) creates/manipulates test resources in AWS corresponding to git feature branches, shells out less than it would if it was a typical shell language (using boto3 rather than shelling to the AWS CLI, etc.) but still shells out to git. But it would be replacing a shell script even if I had a git-repository-interaction library that eliminated the need to shell out.
- stormbrew 6y ago
  I mean I'd agree that's a good candidate for replacement; once you start doing anything more complicated than maybe idempotent curls to an external service I think it ceases to be a 'shell script'. This may sound a little like "no true scotsman" I guess, but "what is the scotsman" is kind of the key issue that I'm saying gets glossed over when people say blanket things like "never write shell, always use python."
  - dragonwriter 6y ago
    I’m not sure I’d agree with defining the scope of things for which an actual shell language is ideal as inherently “shell scripts” (for a few reasons, including the evolving variety of shell languages, the fuzzy boundaries of that set, etc.), I think that it is probably the case that the choice between bash (or powershell or any other shell language) and Python isn’s as simple as “always use Python” and that there is plenty of space for discussion around which tasks each is appropriate for, with the answer differing based on which shell languages you are looking at.
    - stormbrew 6y ago
      This (and a loooong history of similar posts) is the post that I was originally responding to: > Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead. This may be slightly weaker than "never use shell" or "always use python" but it's pretty close to the former (you can almost always "avoid" using shell somehow), and people seem to believe that python is somehow particularly uniquely suited to use for shell tasks that I think there's some degree of the latter implied. So... I agree with you? I am saying that there is room for discussion around what constitutes a good use of shell script, and that there is a space in which that is true. I'm suggesting that a lot of people who believe otherwise may just mostly be writing things that aren't in that space, and I think this is actually a pretty charitable argument tbh.
lukeschlather 6y ago
I agree, but I use Powershell rather than Python. (I've actually written Powershell scripts that implement pieces in Python.) Because gluing together bits of Powershell and Python is still better than trying to write it in bash.
svnpenn 6y ago
As someone who has gold in Bash on Stack Overflow, I can say this with confidence: Bash is just a shitty programming language. If you are writing anything larger than about 60 lines, don't use Bash. Almost anything else will be better. Python, PHP, Ruby, JavaScript, even Go.
- snidane 6y ago
  True. However 60 lines of bash would amount to 300 lines of Python.
  - diarrhea 6y ago
    Possibly, not necessarily, and I'd argue it's for the better. If a bash one-liner requires a wizard to decipher (especially because people way overuse short options over long ones in scripts), and 10 lines of equivalent Python can be understood by someone who has never actually written any Python, I know what I'd pick. This also helps in eradicating bugs. Bash is very hard to get right (hence this thread in the first place), even for the simplest stuff. This simply won't happen in Python (or any other proper language).

blabitty 6y ago

I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.

kubanczyk 6y ago
Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.
- fillest 6y ago
  It is OK to call whatever one needs actually - just minimize the usage of Bash language features (Python provides most of them). It is not complicated and requires no dependencies, here's a self-contained helper: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39d453 https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39...
  - imiric 6y ago
    Great, would that work with pipes and redirections too? At that point you might as well use sh[1] which is a more advanced abstraction. My issue with these approaches is that you'll inevitably hit a bug or limitation where just using a shell language would've been easier to maintain and more portable. Python has its uses, but replacing shell scripts is not one of them IMO. It it was then the problem didn't need to be a shell script to begin with (doesn't shell out often, needs complex data structures, etc.). [1]: https://amoffat.github.io/sh/ https://amoffat.github.io/sh/
- stormbrew 6y ago
  It's worth noting that there are still a shocking number of production machines in the world that don't have stable installations of python3.x yet.
  - pmiller2 6y ago
    So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.
    - stormbrew 6y ago
      So, a bash script written ten years ago probably still works on any version of bash since then. A python script written ten years ago probably doesn't even parse on python3.0, let alone python3.6 (because there have been backwards incompatible changes since 3.0 even!). And that's also true vice versa. So like, this is to the point that there's some "reasonable subset" of python you can use to make it portable (ie. no external packages so you don't need to worry about pip vs. setuptools or venvs or whatever, let alone whether they'll build). I'm asserting that there is not.
ptx 6y ago
> file and directory manipulations are way more complicated in native Python Have you looked at the pathlib[1] module that was added in Python 3.4? It's fantastic. Makes things both more convenient and more correct, in my experience, and it lets you manipulate Windows paths on Unix and vice versa. [1] https://docs.python.org/3/library/pathlib.html https://docs.python.org/3/library/pathlib.html

marcosdumay 6y ago

Hum, no those people are probably writing the same kind of script you do. Their Python versions are indeed longer and more verbose. I always recommend Python anyway if you care about edge cases (what is not always). That longer script handles them on the obvious way, while the short and more readable shell script does something absolutely crazy every time a detail is different from planed. And if you try to correctly handle the edge cases in Bash (you'll fail), you'll get something much more complicated than the Python version anyway.

rectang 6y ago
Which of these is simpler? Rust: fn add(a: i32, b: i32) -> i32 { a + b } Python: def add(a, b): return a + b Arguably the Python function is "simpler" because it occupies fewer characters. And yet, the Rust function is more tightly specified; it does less. It can't throw exceptions, unlike the Python function. Invalid programs where you pass the wrong type to the Rust function won't even compile, whereas the Python function will blow up at runtime. I figure that these supposedly "simpler" Bash programs are not simple at all.
- marcosdumay 6y ago
  Oh, I'm really not comparing it with Rust. The comparison is between a general language and a shell. If it's Python, Rust, PHP, Java, or whatever is of secondary relevance. I've done some ops work in C, Python, and C#. I'm also waiting for a good problem to try some Haskell shell monad. All the same old language pros and cons apply.
  - rectang 6y ago
    I'm just making a general argument about what "simple" means. A well-specified program may or may not be superficially simpler, but it will exhibit simpler behavior.
    - marcosdumay 6y ago
      Oh, ok. The Rust version is at least as simple as the Python one, depending on what kinds of detail you care about. The thing about shell code is that there are plenty of cases where you don't care about things like error management, bad inputs, and etc. So it's not immediately obvious what language is simpler. When you care about those, the shell becomes the most complex choice by far, and that's the duality problem that generates all those different opinions you see around.

dragonwriter 6y ago

> I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"? A replacement for a shell script may not itself be a shell script. A fairly typical python-as-shell-replacement I have that (among other things) creates/manipulates test resources in AWS corresponding to git feature branches, shells out less than it would if it was a typical shell language (using boto3 rather than shelling to the AWS CLI, etc.) but still shells out to git. But it would be replacing a shell script even if I had a git-repository-interaction library that eliminated the need to shell out.

stormbrew 6y ago
I mean I'd agree that's a good candidate for replacement; once you start doing anything more complicated than maybe idempotent curls to an external service I think it ceases to be a 'shell script'. This may sound a little like "no true scotsman" I guess, but "what is the scotsman" is kind of the key issue that I'm saying gets glossed over when people say blanket things like "never write shell, always use python."
- dragonwriter 6y ago
  I’m not sure I’d agree with defining the scope of things for which an actual shell language is ideal as inherently “shell scripts” (for a few reasons, including the evolving variety of shell languages, the fuzzy boundaries of that set, etc.), I think that it is probably the case that the choice between bash (or powershell or any other shell language) and Python isn’s as simple as “always use Python” and that there is plenty of space for discussion around which tasks each is appropriate for, with the answer differing based on which shell languages you are looking at.
  - stormbrew 6y ago
    This (and a loooong history of similar posts) is the post that I was originally responding to: > Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead. This may be slightly weaker than "never use shell" or "always use python" but it's pretty close to the former (you can almost always "avoid" using shell somehow), and people seem to believe that python is somehow particularly uniquely suited to use for shell tasks that I think there's some degree of the latter implied. So... I agree with you? I am saying that there is room for discussion around what constitutes a good use of shell script, and that there is a space in which that is true. I'm suggesting that a lot of people who believe otherwise may just mostly be writing things that aren't in that space, and I think this is actually a pretty charitable argument tbh.

lukeschlather 6y ago

I agree, but I use Powershell rather than Python. (I've actually written Powershell scripts that implement pieces in Python.) Because gluing together bits of Powershell and Python is still better than trying to write it in bash.

svnpenn 6y ago

As someone who has gold in Bash on Stack Overflow, I can say this with confidence: Bash is just a shitty programming language. If you are writing anything larger than about 60 lines, don't use Bash. Almost anything else will be better. Python, PHP, Ruby, JavaScript, even Go.

snidane 6y ago
True. However 60 lines of bash would amount to 300 lines of Python.
- diarrhea 6y ago
  Possibly, not necessarily, and I'd argue it's for the better. If a bash one-liner requires a wizard to decipher (especially because people way overuse short options over long ones in scripts), and 10 lines of equivalent Python can be understood by someone who has never actually written any Python, I know what I'd pick. This also helps in eradicating bugs. Bash is very hard to get right (hence this thread in the first place), even for the simplest stuff. This simply won't happen in Python (or any other proper language).