5 ms·

The superpower of shell scripting languages is not just that it's easy to run external processes, it's also that you can run them extremely flexibly: chain them

by OskarS 1y ago

The superpower of shell scripting languages is not just that it's easy to run external processes, it's also that you can run them extremely flexibly: chain them into pipelines, run them in background, redirecting streams, process substitution, etc. These kinds of things is why "just use Python or whatever!" is not always a great answer to "shell scripting sucks" (which it does, I agree). I mean, you CAN do all those things in Python, but it's a huge pain in the ass to write compared to the dedicated syntax of shell scripting. If the developers of ABS are reading this, it would be great some examples of that kind of thing to the homepage. Like, how would you do `curl <url> | grep something | sort | uniq | tee output.txt`? What about process substitution? What about redirecting stderr to stdout? And, the biggest problem with shell scripting is escaping problems, how does ABS handle that? Can you run a command with an array of strings as argument instead of having the shell split on (unescaped and unquoted) spaces?

frou_dh 1y ago
I remember someone made an interesting point that the notable thing about traditional shell scripting languages (as opposed to conventional scripting languages) is that the filesystem is effectively the "top level scope" of the language. i.e. you can refer to files (paths) much like they are variables that are ambiently defined.
- codethief 1y ago
  Agreed. Then again nothing stops a general-purpose language from providing a built-in syntax & data type for paths, see e.g. Nix.
- OskarS 1y ago
  Yeah, and environment variables as well. It's notable that the syntax for reading an environment variable in bash is identical to the syntax for reading a "normal", process local variable (though setting is slightly different, you have to use "export"). It's like the ground scope is "the computing environment" (files, processes, env vars, ...), not the "program sandbox".
7thaccount 1y ago
I think a common strategy (at least for me) is to make good use of Linux commands and things like Awk and Sed. If it grows more than a few lines, it's often best to switch to something like Python, Perl, Tcl...etc.
- kjs3 1y ago
  That's a (self-)discipline thing. One has to do something about "this script has become a big conky" other than default to "oh, what's another 10 lines". Which we all do.
  - 7thaccount 1y ago
    At some point, the clear features of modern languages far outweigh what I get out of the command line tools and Awk/Bash. For me, it doesn't take long, so it's less discipline and more not wasting my time. Your mileage may vary though.
    - kjs3 1y ago
      Indeed. I cannot tell you how many time I've 'wasted time' on a one-line sed script or 10 lines of awk that obviously would have been better spent on several times that number of lines of a 'modern' language, that I would have to install before I could use. Terrible...something only a boomer would do, amiright. YMMV, though.
      - 7thaccount 1y ago
        I might do the same if there isn't anything already pre installed. On most of the RHEL machines I've used there has at least been some Python or Perl interpreter installed. Maybe that was just something the IT folks did. I'm not really talking about making something longer just to be more clear either. I just meant all the nice libraries and Python data structures seem to allow me to do complex things in a short amount of code. Bash and Awk and all the Linux commands are great for simple things, but I haven't figured out how to handle more complex data outputs.
zokier 1y ago
> it's also that you can run them extremely flexibly: chain them into pipelines, run them in background, redirecting streams Imho traditional shells are actually pretty limited in this regard; you generally can only compose programs in simple linear pipelines with one input one output. Technically bash does allow doing some stuff with additional fds, but it is not very convenient nor flexible in that regard. There have been some attempts to make more flexible shells, like for example dgsh: https://www2.dmst.aueb.gr/dds/sw/dgsh/ https://www2.dmst.aueb.gr/dds/sw/dgsh/
- kjs3 1y ago
  simple linear pipelines Which is the common case, for which it does a reasonable job. I think if I was looking at most of the directed graph examples, I'd look to another tool than a shell to orchestrate that, rather than say "you know what my shell lacks? More special case syntax.".
  - zokier 1y ago
    Sure, sh might be fit for it's purpose, but I wouldn't call it "extremely flexible"
- OskarS 1y ago
  You can do that by using mkfifo relatively easy, but I take your point. In that sense, the most "flexible" language is pure C, because you can do anything there. I guess a better word is "ergonomic": if you want to do `curl something | grep whatever > output.txt` in C or Python or whatever other general purpose language, it's MUCH more difficult. Writing lines like that is trivial for me in shell scripting, I just did it in this comment. If I had to do it in Python, I would have to spend 20 minutes studying the precise flags to pass to subprocess.Popen and think really hard about which streams to call .close() on and when. God help me if I was doing it in C and using fork()/exec(). This is the shell scripting super-power, all this stuff is just trivial.
hacb 1y ago
It seems that some kind of piping is possible: https://github.com/abs-lang/abs/blob/master/examples/pipe.abs https://github.com/abs-lang/abs/blob/master/examples/pipe.ab...
- chubot 1y ago
  I'm pretty sure that's "interior" piping to ABS functions like split() and map() As opposed to "exterior" piping to executables like grep and xargs, which may be written in any language I believe the comment you're replying to is saying that's a design MISTAKE And I agree -- in trying to make shell more consistent, you've actually REDUCED its power I wrote a blog post about this issue: Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html (which was aimed at people working on Oils, not general users. Hopefully a more concise/illustrative post will appear, though I also welcome anyone else like the OP who wants to argue this ! )
taeric 1y ago
They are also much more interaction based than people typically ack. Ctrl-z sending something to the background and being able to list the currently active jobs for the current shell are specifically affordances for someone sitting at the keyboard. Even the pushd/popd workflow is again for the operator. Python and a few others such as LISP, smalltalk, etc., have some affordances made for a developer at the keyboard. Oddly, most of them are not taught to developers, it seems. The number of people I have worked with that didn't know of python's `help` is stark.
chubot 1y ago
That's a major reason that https://oils.pub/ https://oils.pub/ started out with the POSIX- and bash-compatible OSH, and then added YSH on top of it (while removing legacy) In YSH, you can do everything that bash does with processes, because OSH is the most bash-compatible shell in the world: https://pages.oils.pub/spec-compat/2025-09-14/renamed-tmp/spec/compat/TOP.html https://pages.oils.pub/spec-compat/2025-09-14/renamed-tmp/sp... For example, I think many of the newer shells don't implement process subs, a ksh/bash thing: ysh-0.35$ diff -u <(seq 2 3) <(seq 1 4) ... +1 2 3 +4 YSH also improves on bash by checking the error code from process subs, which bash simply ignores: cat <(seq ) <(seq 1 4); echo hi ^~ [ interactive ]:39: errexit PID 693013: Command failed with status 1 BUT you can also do Python like stuff, pretty much exactly like ABS's example on the home page var obj = {} for n in (1 ..< 10) { if (n % 2 === 0) { setvar obj[str(n)] = 6 ** 2 # no rand function now } } = obj # pretty print with = # => (Dict) {'2': 36, '4': 36, '6': 36, '8': 36} (hm I wonder if str(n) should be relaxed ...) ABS is listed on this wiki page, along with many Python- and JavaScript- influenced shells: https://github.com/oils-for-unix/oils/wiki/Alternative-Shells https://github.com/oils-for-unix/oils/wiki/Alternative-Shell... https://koi-lang.dev/ https://koi-lang.dev/ https://github.com/alexst07/shell-plus-plus https://github.com/alexst07/shell-plus-plus https://hush-shell.github.io/ https://hush-shell.github.io/ But YSH is the only one that has a process runtime that's a SUPERSET of bash --- I also want to add to add xargs-like functionality to the runtime. I mentioned in the latest release that bash/xargs actually has a problem with waiting for processes in parallel, while ALSO checking for errors: Eight Releases of Oils in 6 Months: https://oils.pub/blog/2025/09/releases.html https://oils.pub/blog/2025/09/releases.html

by OskarS 1y ago

frou_dh 1y ago

I remember someone made an interesting point that the notable thing about traditional shell scripting languages (as opposed to conventional scripting languages) is that the filesystem is effectively the "top level scope" of the language. i.e. you can refer to files (paths) much like they are variables that are ambiently defined.

codethief 1y ago
Agreed. Then again nothing stops a general-purpose language from providing a built-in syntax & data type for paths, see e.g. Nix.
OskarS 1y ago
Yeah, and environment variables as well. It's notable that the syntax for reading an environment variable in bash is identical to the syntax for reading a "normal", process local variable (though setting is slightly different, you have to use "export"). It's like the ground scope is "the computing environment" (files, processes, env vars, ...), not the "program sandbox".

7thaccount 1y ago

I think a common strategy (at least for me) is to make good use of Linux commands and things like Awk and Sed. If it grows more than a few lines, it's often best to switch to something like Python, Perl, Tcl...etc.

kjs3 1y ago
That's a (self-)discipline thing. One has to do something about "this script has become a big conky" other than default to "oh, what's another 10 lines". Which we all do.
- 7thaccount 1y ago
  At some point, the clear features of modern languages far outweigh what I get out of the command line tools and Awk/Bash. For me, it doesn't take long, so it's less discipline and more not wasting my time. Your mileage may vary though.
  - kjs3 1y ago
    Indeed. I cannot tell you how many time I've 'wasted time' on a one-line sed script or 10 lines of awk that obviously would have been better spent on several times that number of lines of a 'modern' language, that I would have to install before I could use. Terrible...something only a boomer would do, amiright. YMMV, though.
    - 7thaccount 1y ago
      I might do the same if there isn't anything already pre installed. On most of the RHEL machines I've used there has at least been some Python or Perl interpreter installed. Maybe that was just something the IT folks did. I'm not really talking about making something longer just to be more clear either. I just meant all the nice libraries and Python data structures seem to allow me to do complex things in a short amount of code. Bash and Awk and all the Linux commands are great for simple things, but I haven't figured out how to handle more complex data outputs.

zokier 1y ago

> it's also that you can run them extremely flexibly: chain them into pipelines, run them in background, redirecting streams Imho traditional shells are actually pretty limited in this regard; you generally can only compose programs in simple linear pipelines with one input one output. Technically bash does allow doing some stuff with additional fds, but it is not very convenient nor flexible in that regard. There have been some attempts to make more flexible shells, like for example dgsh: https://www2.dmst.aueb.gr/dds/sw/dgsh/ https://www2.dmst.aueb.gr/dds/sw/dgsh/

kjs3 1y ago
simple linear pipelines Which is the common case, for which it does a reasonable job. I think if I was looking at most of the directed graph examples, I'd look to another tool than a shell to orchestrate that, rather than say "you know what my shell lacks? More special case syntax.".
- zokier 1y ago
  Sure, sh might be fit for it's purpose, but I wouldn't call it "extremely flexible"
OskarS 1y ago
You can do that by using mkfifo relatively easy, but I take your point. In that sense, the most "flexible" language is pure C, because you can do anything there. I guess a better word is "ergonomic": if you want to do `curl something | grep whatever > output.txt` in C or Python or whatever other general purpose language, it's MUCH more difficult. Writing lines like that is trivial for me in shell scripting, I just did it in this comment. If I had to do it in Python, I would have to spend 20 minutes studying the precise flags to pass to subprocess.Popen and think really hard about which streams to call .close() on and when. God help me if I was doing it in C and using fork()/exec(). This is the shell scripting super-power, all this stuff is just trivial.

hacb 1y ago

It seems that some kind of piping is possible: https://github.com/abs-lang/abs/blob/master/examples/pipe.abs https://github.com/abs-lang/abs/blob/master/examples/pipe.ab...

chubot 1y ago
I'm pretty sure that's "interior" piping to ABS functions like split() and map() As opposed to "exterior" piping to executables like grep and xargs, which may be written in any language I believe the comment you're replying to is saying that's a design MISTAKE And I agree -- in trying to make shell more consistent, you've actually REDUCED its power I wrote a blog post about this issue: Oils Is Exterior-First (Code, Text, and Structured Data) - https://www.oilshell.org/blog/2023/06/ysh-design.html https://www.oilshell.org/blog/2023/06/ysh-design.html (which was aimed at people working on Oils, not general users. Hopefully a more concise/illustrative post will appear, though I also welcome anyone else like the OP who wants to argue this ! )

taeric 1y ago

They are also much more interaction based than people typically ack. Ctrl-z sending something to the background and being able to list the currently active jobs for the current shell are specifically affordances for someone sitting at the keyboard. Even the pushd/popd workflow is again for the operator. Python and a few others such as LISP, smalltalk, etc., have some affordances made for a developer at the keyboard. Oddly, most of them are not taught to developers, it seems. The number of people I have worked with that didn't know of python's `help` is stark.

chubot 1y ago

That's a major reason that https://oils.pub/ https://oils.pub/ started out with the POSIX- and bash-compatible OSH, and then added YSH on top of it (while removing legacy) In YSH, you can do everything that bash does with processes, because OSH is the most bash-compatible shell in the world: https://pages.oils.pub/spec-compat/2025-09-14/renamed-tmp/spec/compat/TOP.html https://pages.oils.pub/spec-compat/2025-09-14/renamed-tmp/sp... For example, I think many of the newer shells don't implement process subs, a ksh/bash thing: ysh-0.35$ diff -u <(seq 2 3) <(seq 1 4) ... +1 2 3 +4 YSH also improves on bash by checking the error code from process subs, which bash simply ignores: cat <(seq ) <(seq 1 4); echo hi ^~ [ interactive ]:39: errexit PID 693013: Command failed with status 1 BUT you can also do Python like stuff, pretty much exactly like ABS's example on the home page var obj = {} for n in (1 ..< 10) { if (n % 2 === 0) { setvar obj[str(n)] = 6 ** 2 # no rand function now } } = obj # pretty print with = # => (Dict) {'2': 36, '4': 36, '6': 36, '8': 36} (hm I wonder if str(n) should be relaxed ...) ABS is listed on this wiki page, along with many Python- and JavaScript- influenced shells: https://github.com/oils-for-unix/oils/wiki/Alternative-Shells https://github.com/oils-for-unix/oils/wiki/Alternative-Shell... https://koi-lang.dev/ https://koi-lang.dev/ https://github.com/alexst07/shell-plus-plus https://github.com/alexst07/shell-plus-plus https://hush-shell.github.io/ https://hush-shell.github.io/ But YSH is the only one that has a process runtime that's a SUPERSET of bash --- I also want to add to add xargs-like functionality to the runtime. I mentioned in the latest release that bash/xargs actually has a problem with waiting for processes in parallel, while ALSO checking for errors: Eight Releases of Oils in 6 Months: https://oils.pub/blog/2025/09/releases.html https://oils.pub/blog/2025/09/releases.html