6 ms·

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still mi

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for each little string manipulation task. And if I ever need to ‘shell out’ to an external process, there's always the subprocess module that is much more robust than ‘set -e’ ever was. Seriously, if you can avoid it, just don't write shell scripts at all. Use a real programming language instead.

lanstin 6y ago
All the deployment things are just wrappers to shell commands so if you want to help people debug user-data.sh, cron, ansible, docker files, or puppet, you need to understand shell. Also, it is better to have something in bash as code than not have it be fully automated. I draw the line at arrays tho. If you want arrays don’t do bash. But if you want to just run a bunch of commands on your compute, bash is there. One thing I have started doing for stuff that need to go via ssh is I just do a here doc for an embedded shell script that I scp to the remote and then just use ssh to invoke it. The quoting and pseudotty stuff just got to be too dumb to understand. Also, and maybe this is just for non-interactive scripts, I find everything from writing them to using them is a lot easier if you take pains to make them idempotent, so you can run them one or 100 times with the same effect.
yeowMeng 6y ago
> The [python] scripts are much faster... Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell. Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? I like python, but if I am in a rush it’s shell.
- neolog 6y ago
  > Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? Are you asking why people make recommendations in general?
- marcosdumay 6y ago
  Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.
- kaliszad 6y ago
  Actually, the one thing that I as a systems engineer really like about Clojure is the threading macro (-> some-data some-function-applied-on-some-data some-otherfunction) or (->> ...) for inserting not as a second, but last argument. That is like a pipeline but usually more readable to me even though I am proficient with the shell (and I must admit even PowerShell, which really is more comparable to Perl with a bit of SQL and other flavours here and there than bash). Sysadmins/ system engineers might love Babashka: https://github.com/babashka/babashka https://github.com/babashka/babashka which is a large subset of Clojure + some frequently used libraries as a native GraalVM image. It is portable, has very fast startup and if the script becomes a larger program, you can easily switch to ClojureScript + Node.js (e.g. for still very fast startup) or Clojure (on the normal JVM) or perhaps build your own GraalVM image. You might also just open a REPL and run it as a single session but that is rather unique in the sysadmin/ systems engineer space, where most things are launched on schedule e.g. each 5 minutes by a script and in case the startup time is somewhat long, it might dominate the execution time. Btw. babashka seems to be about twice as fast to start on my Debian: time bb -e '(+ 1 1)' executes on average in about 11 ms vs time python3 -c 'print(1 + 1)' executes on average in about 23 ms
IshKebab 6y ago
100% agree. This article should just be "rewrite your script in Python". It's not the best scripting language ever, but it works fairly well and it's about 100 times better than Bash.
stormbrew 6y ago
This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been. I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?
- blabitty 6y ago
  I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.
  - kubanczyk 6y ago
    Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.
jrockway 6y ago
Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". The shell is a pretty good REPL, and glue like seq, find, xargs, grep, etc. are very good. But honestly, I find myself reaching for purpose-built tools rather than ad-hoc pipelines more and more. fdfind does more of what I want on average than find; rg does more of what I want than 'find | xargs grep'. I also find myself using more and more structured data, and write more jq pipelines than I do bash pipelines. Finally, I am getting more and more frustrated at basic shell mechanics like history handling. I have 5 terminals open on average, and I don't understand why C-r in one won't find history in another (I know why, of course, but I don't like it). I'm getting pretty close to just writing my own shell and toolchain. I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't. They just hacked Unicode and remote RPCs into shell prompts so that pressing "enter" on an empty shell prompt takes 25 seconds to run. I'm not sure why people are so focused on the polish and not the core inadequacies, but they are, and I feel like I'm going to have to fix that myself in the next couple years...
- chubot 6y ago
  Have you tried any of the shells here? https://github.com/oilshell/oil/wiki/Alternative-Shells https://github.com/oilshell/oil/wiki/Alternative-Shells Discussed recently: https://news.ycombinator.com/item?id=26121592 https://news.ycombinator.com/item?id=26121592 A lot of them are "fundamentally reimagining" shell -- actually I'd say MOST of them are; whether that's good or bad depends on the user's POV. Oil is reimagining shell, but also providing a graceful upgrade path. Out of all the shells I'd say it's most focused on the fundamental language and runtime, and less on the interactive issues (right now).
- deleted 6y ago
  [deleted]
- imiric 6y ago
  I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks. > I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls. > I find myself reaching for purpose-built tools rather than ad-hoc pipelines It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell. > I am getting more and more frustrated at basic shell mechanics like history handling That's mostly a solved problem in many shells. Zsh supports it natively[2]. I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost. [1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ [2]: https://nuclearsquid.com/writings/shared-history-in-zsh/ https://nuclearsquid.com/writings/shared-history-in-zsh/
chubot 6y ago
Shouldn't scripts over 100 lines be rewritten in Python or Ruby? http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh... tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write) Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Professional-Computng/dp/0131429019 https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe... Online for free: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/
Supersaiyan_IV 6y ago
At least shell follows POSIX standard which is more than one can say about Python. Have witnessed self-proclaimed "shell replacing" python scripts invoking commands, and never checking the exit code. Debugging hell.
macromagnon 6y ago
I like bash because it's easy to connect different data together. I can output a column from a sql query to a file and easily write a for loop that goes over the data and use it. I work mostly in payment related code, so most often I use bash to hot fix in production. Like if we need to refund certain people, I'll convert the refund api call to a curl call, get the data from sql and then just run a simple loop.
gorgoiler 6y ago
Bash is fine as long as everything works. Handling errors in bash is a fool’s errand and you’re quite right to use Python for resilience in the face of errors or malformed input. Sometimes you just gotta eat. Python is a sit-down meal. Bash is a can of beans. I’m happy eating both for dinner but it depends who (if anyone) I’m sharing my meal with.
lenkite 6y ago
Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!

lanstin 6y ago

All the deployment things are just wrappers to shell commands so if you want to help people debug user-data.sh, cron, ansible, docker files, or puppet, you need to understand shell. Also, it is better to have something in bash as code than not have it be fully automated. I draw the line at arrays tho. If you want arrays don’t do bash. But if you want to just run a bunch of commands on your compute, bash is there. One thing I have started doing for stuff that need to go via ssh is I just do a here doc for an embedded shell script that I scp to the remote and then just use ssh to invoke it. The quoting and pseudotty stuff just got to be too dumb to understand. Also, and maybe this is just for non-interactive scripts, I find everything from writing them to using them is a lot easier if you take pains to make them idempotent, so you can run them one or 100 times with the same effect.

yeowMeng 6y ago

> The [python] scripts are much faster... Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell. Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? I like python, but if I am in a rush it’s shell.

neolog 6y ago
> Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? Are you asking why people make recommendations in general?
marcosdumay 6y ago
Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.
kaliszad 6y ago
Actually, the one thing that I as a systems engineer really like about Clojure is the threading macro (-> some-data some-function-applied-on-some-data some-otherfunction) or (->> ...) for inserting not as a second, but last argument. That is like a pipeline but usually more readable to me even though I am proficient with the shell (and I must admit even PowerShell, which really is more comparable to Perl with a bit of SQL and other flavours here and there than bash). Sysadmins/ system engineers might love Babashka: https://github.com/babashka/babashka https://github.com/babashka/babashka which is a large subset of Clojure + some frequently used libraries as a native GraalVM image. It is portable, has very fast startup and if the script becomes a larger program, you can easily switch to ClojureScript + Node.js (e.g. for still very fast startup) or Clojure (on the normal JVM) or perhaps build your own GraalVM image. You might also just open a REPL and run it as a single session but that is rather unique in the sysadmin/ systems engineer space, where most things are launched on schedule e.g. each 5 minutes by a script and in case the startup time is somewhat long, it might dominate the execution time. Btw. babashka seems to be about twice as fast to start on my Debian: time bb -e '(+ 1 1)' executes on average in about 11 ms vs time python3 -c 'print(1 + 1)' executes on average in about 23 ms

IshKebab 6y ago

100% agree. This article should just be "rewrite your script in Python". It's not the best scripting language ever, but it works fairly well and it's about 100 times better than Bash.

stormbrew 6y ago

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more complicated than an equivalent shell script would have been. I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?

blabitty 6y ago
I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as reasoning for this replacememt love is usually one or a combination of a few things - a claimed write once run anywhere ability that I think is generally fantasy, a belief that Python is a "real" language and shell is not which usually just means "I don't like shell or think it is unfashionable", or a belief that Python code can be made more reusable. The person in question almost always has a background of feature developer that is now doing devops type tasks. All that said I like python and there are tasks I absolutely prefer it for, usually anything to do with scraping some web API that returns a complex piece of JSON or xml.
- kubanczyk 6y ago
  Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.

jrockway 6y ago

Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". The shell is a pretty good REPL, and glue like seq, find, xargs, grep, etc. are very good. But honestly, I find myself reaching for purpose-built tools rather than ad-hoc pipelines more and more. fdfind does more of what I want on average than find; rg does more of what I want than 'find | xargs grep'. I also find myself using more and more structured data, and write more jq pipelines than I do bash pipelines. Finally, I am getting more and more frustrated at basic shell mechanics like history handling. I have 5 terminals open on average, and I don't understand why C-r in one won't find history in another (I know why, of course, but I don't like it). I'm getting pretty close to just writing my own shell and toolchain. I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't. They just hacked Unicode and remote RPCs into shell prompts so that pressing "enter" on an empty shell prompt takes 25 seconds to run. I'm not sure why people are so focused on the polish and not the core inadequacies, but they are, and I feel like I'm going to have to fix that myself in the next couple years...

chubot 6y ago
Have you tried any of the shells here? https://github.com/oilshell/oil/wiki/Alternative-Shells https://github.com/oilshell/oil/wiki/Alternative-Shells Discussed recently: https://news.ycombinator.com/item?id=26121592 https://news.ycombinator.com/item?id=26121592 A lot of them are "fundamentally reimagining" shell -- actually I'd say MOST of them are; whether that's good or bad depends on the user's POV. Oil is reimagining shell, but also providing a graceful upgrade path. Out of all the shells I'd say it's most focused on the fundamental language and runtime, and less on the interactive issues (right now).
deleted 6y ago
[deleted]
imiric 6y ago
I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks. > I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls. > I find myself reaching for purpose-built tools rather than ad-hoc pipelines It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell. > I am getting more and more frustrated at basic shell mechanics like history handling That's mostly a solved problem in many shells. Zsh supports it natively[2]. I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost. [1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/ http://redsymbol.net/articles/unofficial-bash-strict-mode/ [2]: https://nuclearsquid.com/writings/shared-history-in-zsh/ https://nuclearsquid.com/writings/shared-history-in-zsh/

chubot 6y ago

Shouldn't scripts over 100 lines be rewritten in Python or Ruby? http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby http://www.oilshell.org/blog/2021/01/why-a-new-shell.html#sh... tl;dr I think the main skill that is missing is being able to write a Python script that fits well within a shell script. The shell script often invokes tools written in other languages like C, Rust, JavaScript, etc. (most of which you didn't write) Good book on this: https://www.amazon.com/UNIX-Programming-Addison-Wesley-Professional-Computng/dp/0131429019 https://www.amazon.com/UNIX-Programming-Addison-Wesley-Profe... Online for free: http://www.catb.org/esr/writings/taoup/html/ http://www.catb.org/esr/writings/taoup/html/

Supersaiyan_IV 6y ago

At least shell follows POSIX standard which is more than one can say about Python. Have witnessed self-proclaimed "shell replacing" python scripts invoking commands, and never checking the exit code. Debugging hell.

macromagnon 6y ago

I like bash because it's easy to connect different data together. I can output a column from a sql query to a file and easily write a for loop that goes over the data and use it. I work mostly in payment related code, so most often I use bash to hot fix in production. Like if we need to refund certain people, I'll convert the refund api call to a curl call, get the data from sql and then just run a simple loop.

gorgoiler 6y ago

Bash is fine as long as everything works. Handling errors in bash is a fool’s errand and you’re quite right to use Python for resilience in the face of errors or malformed input. Sometimes you just gotta eat. Python is a sit-down meal. Bash is a can of beans. I’m happy eating both for dinner but it depends who (if anyone) I’m sharing my meal with.

lenkite 6y ago

Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!