7 ms·

I wish that shell would be more sane. I don't think that a shell should be a complete programming language. If you need a programming language, then better use

I wish that shell would be more sane. I don't think that a shell should be a complete programming language. If you need a programming language, then better use one. There is xonsh if you are looking for something like this. I think there should be a better bash with an very clean and consistent interface. Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions.

thyrsus 6y ago
If the data is small, then expressivness is more important. If the data is large, then multiple processes connected by pipes lets you employ more CPUs.
- mooreds 6y ago
  Have to pull this old chestnut out: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-... I am not the author of that piece, but the analysis is wonderful.
enriquto 6y ago
> I don't think that a shell should be a complete programming language. What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?
- tutfbhuf 6y ago
  I mean shell should not be python, like xonsh. In my opinion it should not add programming language features (like types, futures, ADTs,...), but instead should clean up the existing built-ins + maybe little bit coreutils as built-ins.
  - enriquto 6y ago
    Never heard about xonsh, it looks really cool. Seems like a step in the good direction (from python to shell). It needs a few steps more to attain the elegant perfection of the shell language, for example, removing this "type" stuff from the variables ;)
Aloha 6y ago
It wouldn't be coherently unixy if it was. Small tools that do a limited subset of things, and reliably take/output data from stdin/stdout is the way unix is done. sh is just the glue we use to tack it all together. Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.
- CJefferson 6y ago
  One of the best things (in my opinion) shells could have done is do newline separated filenames instead of spaces, which would make filenames with spaces much easier to handle (you can have newlines in filenames, but in this magic world let's ban those).
  - enriquto 6y ago
    That would pose problems also. The real solution, with no need to magic as you say, is to disallow spaces at the filesystem level (just like slashes and the null character are forbidden). For users typing filenames, this shouldn't be a problem, as those can be encoded e.g., as unicode's non-breaking space. Using space as a separator is a very important power, that other programming languages share. In what other programming language can you put spaces in variable names? In the shell, filenames are variable names! It is only natural that a space cannot appear inside a filename (but of course, most other unicode characters can).
    - CJefferson 6y ago
      This would require filenames became valid utf-8. This would be a good thing to do nowadays, irritatingly (having tried handling Unix filenames as utf 8 in the past), surprisingly many users have non-utf8 filenames.
      - a1369209993 6y ago
        > This would require filenames became valid utf-8. No, it wouldn't; you'd just get EINVAL if you pass a filename containing "\x20" to open or other syscalls. If a application wants to use "\xC2\xA0" in a filename, it can do that, or it can not do that, same as currently. Same applies if it wants to use, say, "\xAA\xFF".
todd8 6y ago
Yeah, Perl started out a bit like this. I never adopted it for myself or my company, but I did get a chance to have lunch with Larry Wall and talked to him about Perl. This was back when Perl was quite a young and promising language for a certain class of problems. Perl had very good performance, despite being a "scripting" language, because so many of the tasks that a typical shell script might need to do were supported by built in features in the language (for example handling regular expressions, interacting with command line arguments, exec'ing other programs). Today, once a shell script requires very much logic, I'm inclined to use Python. I know Python and find programming in it easier than Perl and bash and other shells. Furthermore, other programmers understand Python better than bash.
akkartik 6y ago
I've been trying to build a computer with a single principled HLL included that can also double as a shell. There's also http://www.oilshell.org http://www.oilshell.org which _also_ tries to be compatible with existing shell. It also tries to include features from awk and sed: https://www.oilshell.org/blog/2016/11/13.html https://www.oilshell.org/blog/2016/11/13.html If these design points turn out to be over-constrained and have to make major compromises, yes the next thing I would try would be a small separate language for shell.
- chubot 6y ago
  BTW I think the general idea of Oil's syntax is working out: start in "command mode", and then when you see the RHS of =, changing modes to Python-/JS- like expressions. There a few other cases where you switch to expression mode, like proc p(a, b) [1] So far nobody has complained about any of this, I'm guessing because it looks very familiar, and that was intentional. Oil takes some pains to literally look like shell + Python syntactically, with better semantics. I noticed that a few other shells are having problems with this command/expression distinction, and I discussed it like 3-4 years ago with Ilya Sher (of NGS) and a few other people. They were also having the same problem. For example, does / mean a path separator or the division operator? Does * mean a glob or a multiplication? In Oil, this is no problem. It's obvious depending on the context. Though I'm interested in more feedback on this, and the latest release is available to try as always :) https://www.oilshell.org/release/latest/ https://www.oilshell.org/release/latest/ --- The awk and make integration is still doable but not done. Awk might require a notion of "lazy expressions" or "lazy arg lists", which would be shared with dplyr-like functionality. Oil's Zulip is open for discussion on these ideas :) [1] https://www.oilshell.org/release/0.8.5/doc/command-vs-expression-mode.html https://www.oilshell.org/release/0.8.5/doc/command-vs-expres...
theon144 6y ago
> I wish that shell would be more sane. Absolutely agreed, shell is in many respects terrible. >I don't think that a shell should be a complete programming language. On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling... I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it. >Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions. UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.
- alpaca128 6y ago
  Keeping those small utilities separate is definitely the right choice as it keeps everything very flexible. I could just replace grep with ripgrep in my workflow and suddenly get a drastically quicker search without having to change anything else. If all those features were integrated in the shell the user is more or less stuck with the whole package. Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.
- geophile 6y ago
  Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly typed. Instead of bailing out and redoing everything in Python (when you reach the end of your pipeline), you can add a bit of Python code in your command. For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes: ls -fr \ | select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \ | read -l \ | select (path, line: line.startswith('class ')) read -l reads and labels a file, i.e., filename -> stream of (filename, line in file) (Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)
thrower123 6y ago
If you think the unix shell sucks, try fucking around with the Windows shell - either of them. You're either in a world of legacy and inconsistency dating to DOS and the days of 8.3 filenames, or you're in the also wildly inconsistent, but very verbose, world of PowersHell.
- geophile 6y ago
  About powershell's verbosity: My pandemic project is yet another pipe-objects-instead-of-strings shell, marcel: https://marceltheshell.org https://marceltheshell.org. It's "home" is Linux, not Windows. Marcel is also somewhat verbose. It exposes Python functions on the command line, so when you need to write such a function, it starts getting long. E.g. if you want to sort files by last time modified: ls | sort (f: f.mtime) There are abstraction mechanisms to help deal with this. E.g. this defines a command to sort piped-in files by time. bytime = [sort (f: f.mtime)] So you can then do this: ls | bytime Still kind of wordy. I'm thinking about a way of having flags expand to filters. That would get things as compact as bash.
Closi 6y ago
> I think there should be a better bash with an very clean and consistent interface. Absolutely! Some work should be put into making it way more intuitive. Having to remember what every flag means (which is different in every app!) is not intuitive at all. By default there should be some sort of intelisense auto-complete which can also provide guidance on what on earth all the flags mean, and maybe eventually it could tell you what it expects to do before you run the command. For most users otherwise it ends up being "paste in this command you googled with a load of random flags you don't understand and won't remember"
- dafoex 6y ago
  Just suggested it in another comment, but Fish might be similar to what you describe. It has tab-completable flags that it gets from the manpages, and remembers and suggests previously run commands which might be close to the auto-complete you're after.
  - Closi 6y ago
    Fish is better, but it still only gets 1/10th of the way. Why does terminal have to look like terminal? Like why is the 1980's ASCII telnet style the only way to do this? The ASCII fish image on boot and fully ASCII menus only reinforces again that this isn't a modern interface, its an improved 1980's interface. Why, for example, can autocomplete not look like this? https://code.visualstudio.com/docs/editor/intellisense https://code.visualstudio.com/docs/editor/intellisense Why do progress bars when you are downloading from pip not look something like this? (still in-line with the terminal like a sparkline) https://docs.microsoft.com/en-us/windows/win32/uxguide/progress-bars https://docs.microsoft.com/en-us/windows/win32/uxguide/progr... Like surely everything doesn't have to be SO 1980's if we want shell to stay relevant.
    - southpawflo 6y ago
      because then your shell will be slower than a snail. for real though, after looking at your examples I feel that features like those would fall on your terminal emulator to bring to the table, not the shell itself. specifically, I don't want to imagine trying to connect to a headless server during some crisis over bad wifi and have the shell think it needs to send a bunch of graphics to me.
chubot 6y ago
Oil is making shell more sane, in a way that you can realistically adopt, since it runs existing shell scripts. See: Four Features That Justify a New Unix Shell http://www.oilshell.org/blog/2020/10/osh-features.html http://www.oilshell.org/blog/2020/10/osh-features.html
tannhaeuser 6y ago
Your wish has been granted: awk '{ print $2 }' can be replaced by 'cut -f2'. And all of the commands are part of the set of POSIX command-line utilities having been included on Unix systems since basically forever, with awk (the one true awk) having been introduced only in 1983 or so.
- tedunangst 6y ago
  cut doesn't use the same field delimiters as awk.
  - Scarbutt 6y ago
    And awk doesn't let you easily select a range of fields like cut
IshKebab 6y ago
I agree. There's nushell which looks a lot more sane, so I have hope that we won't be stuck with insane tools from the 80s for the rest of time! Maybe people will stop deifying those old flawed tools too.

thyrsus 6y ago

If the data is small, then expressivness is more important. If the data is large, then multiple processes connected by pipes lets you employ more CPUs.

mooreds 6y ago
Have to pull this old chestnut out: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html https://adamdrake.com/command-line-tools-can-be-235x-faster-... I am not the author of that piece, but the analysis is wonderful.

enriquto 6y ago

> I don't think that a shell should be a complete programming language. What do you mean exactly? If you have pipes and lists (e.g., the lines on a file) you are Turing-complete and people can and will make arbitrary programs. How would you like, precisely, to restrict the shell language so that it is not Turing-complete?

tutfbhuf 6y ago
I mean shell should not be python, like xonsh. In my opinion it should not add programming language features (like types, futures, ADTs,...), but instead should clean up the existing built-ins + maybe little bit coreutils as built-ins.
- enriquto 6y ago
  Never heard about xonsh, it looks really cool. Seems like a step in the good direction (from python to shell). It needs a few steps more to attain the elegant perfection of the shell language, for example, removing this "type" stuff from the variables ;)

Aloha 6y ago

It wouldn't be coherently unixy if it was. Small tools that do a limited subset of things, and reliably take/output data from stdin/stdout is the way unix is done. sh is just the glue we use to tack it all together. Bash (and other shells too) is like it is, because it's an accretion of 50 years of history rather than a singular top down design.

CJefferson 6y ago
One of the best things (in my opinion) shells could have done is do newline separated filenames instead of spaces, which would make filenames with spaces much easier to handle (you can have newlines in filenames, but in this magic world let's ban those).
- enriquto 6y ago
  That would pose problems also. The real solution, with no need to magic as you say, is to disallow spaces at the filesystem level (just like slashes and the null character are forbidden). For users typing filenames, this shouldn't be a problem, as those can be encoded e.g., as unicode's non-breaking space. Using space as a separator is a very important power, that other programming languages share. In what other programming language can you put spaces in variable names? In the shell, filenames are variable names! It is only natural that a space cannot appear inside a filename (but of course, most other unicode characters can).
  - CJefferson 6y ago
    This would require filenames became valid utf-8. This would be a good thing to do nowadays, irritatingly (having tried handling Unix filenames as utf 8 in the past), surprisingly many users have non-utf8 filenames.
    - a1369209993 6y ago
      > This would require filenames became valid utf-8. No, it wouldn't; you'd just get EINVAL if you pass a filename containing "\x20" to open or other syscalls. If a application wants to use "\xC2\xA0" in a filename, it can do that, or it can not do that, same as currently. Same applies if it wants to use, say, "\xAA\xFF".

todd8 6y ago

Yeah, Perl started out a bit like this. I never adopted it for myself or my company, but I did get a chance to have lunch with Larry Wall and talked to him about Perl. This was back when Perl was quite a young and promising language for a certain class of problems. Perl had very good performance, despite being a "scripting" language, because so many of the tasks that a typical shell script might need to do were supported by built in features in the language (for example handling regular expressions, interacting with command line arguments, exec'ing other programs). Today, once a shell script requires very much logic, I'm inclined to use Python. I know Python and find programming in it easier than Perl and bash and other shells. Furthermore, other programmers understand Python better than bash.

akkartik 6y ago

I've been trying to build a computer with a single principled HLL included that can also double as a shell. There's also http://www.oilshell.org http://www.oilshell.org which _also_ tries to be compatible with existing shell. It also tries to include features from awk and sed: https://www.oilshell.org/blog/2016/11/13.html https://www.oilshell.org/blog/2016/11/13.html If these design points turn out to be over-constrained and have to make major compromises, yes the next thing I would try would be a small separate language for shell.

chubot 6y ago
BTW I think the general idea of Oil's syntax is working out: start in "command mode", and then when you see the RHS of =, changing modes to Python-/JS- like expressions. There a few other cases where you switch to expression mode, like proc p(a, b) [1] So far nobody has complained about any of this, I'm guessing because it looks very familiar, and that was intentional. Oil takes some pains to literally look like shell + Python syntactically, with better semantics. I noticed that a few other shells are having problems with this command/expression distinction, and I discussed it like 3-4 years ago with Ilya Sher (of NGS) and a few other people. They were also having the same problem. For example, does / mean a path separator or the division operator? Does * mean a glob or a multiplication? In Oil, this is no problem. It's obvious depending on the context. Though I'm interested in more feedback on this, and the latest release is available to try as always :) https://www.oilshell.org/release/latest/ https://www.oilshell.org/release/latest/ --- The awk and make integration is still doable but not done. Awk might require a notion of "lazy expressions" or "lazy arg lists", which would be shared with dplyr-like functionality. Oil's Zulip is open for discussion on these ideas :) [1] https://www.oilshell.org/release/0.8.5/doc/command-vs-expression-mode.html https://www.oilshell.org/release/0.8.5/doc/command-vs-expres...

theon144 6y ago

> I wish that shell would be more sane. Absolutely agreed, shell is in many respects terrible. >I don't think that a shell should be a complete programming language. On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling... I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it. >Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions. UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.

alpaca128 6y ago
Keeping those small utilities separate is definitely the right choice as it keeps everything very flexible. I could just replace grep with ripgrep in my workflow and suddenly get a drastically quicker search without having to change anything else. If all those features were integrated in the shell the user is more or less stuck with the whole package. Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.
geophile 6y ago
Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly typed. Instead of bailing out and redoing everything in Python (when you reach the end of your pipeline), you can add a bit of Python code in your command. For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes: ls -fr \ | select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \ | read -l \ | select (path, line: line.startswith('class ')) read -l reads and labels a file, i.e., filename -> stream of (filename, line in file) (Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)

thrower123 6y ago

If you think the unix shell sucks, try fucking around with the Windows shell - either of them. You're either in a world of legacy and inconsistency dating to DOS and the days of 8.3 filenames, or you're in the also wildly inconsistent, but very verbose, world of PowersHell.

geophile 6y ago
About powershell's verbosity: My pandemic project is yet another pipe-objects-instead-of-strings shell, marcel: https://marceltheshell.org https://marceltheshell.org. It's "home" is Linux, not Windows. Marcel is also somewhat verbose. It exposes Python functions on the command line, so when you need to write such a function, it starts getting long. E.g. if you want to sort files by last time modified: ls | sort (f: f.mtime) There are abstraction mechanisms to help deal with this. E.g. this defines a command to sort piped-in files by time. bytime = [sort (f: f.mtime)] So you can then do this: ls | bytime Still kind of wordy. I'm thinking about a way of having flags expand to filters. That would get things as compact as bash.

Closi 6y ago

> I think there should be a better bash with an very clean and consistent interface. Absolutely! Some work should be put into making it way more intuitive. Having to remember what every flag means (which is different in every app!) is not intuitive at all. By default there should be some sort of intelisense auto-complete which can also provide guidance on what on earth all the flags mean, and maybe eventually it could tell you what it expects to do before you run the command. For most users otherwise it ends up being "paste in this command you googled with a load of random flags you don't understand and won't remember"

dafoex 6y ago
Just suggested it in another comment, but Fish might be similar to what you describe. It has tab-completable flags that it gets from the manpages, and remembers and suggests previously run commands which might be close to the auto-complete you're after.
- Closi 6y ago
  Fish is better, but it still only gets 1/10th of the way. Why does terminal have to look like terminal? Like why is the 1980's ASCII telnet style the only way to do this? The ASCII fish image on boot and fully ASCII menus only reinforces again that this isn't a modern interface, its an improved 1980's interface. Why, for example, can autocomplete not look like this? https://code.visualstudio.com/docs/editor/intellisense https://code.visualstudio.com/docs/editor/intellisense Why do progress bars when you are downloading from pip not look something like this? (still in-line with the terminal like a sparkline) https://docs.microsoft.com/en-us/windows/win32/uxguide/progress-bars https://docs.microsoft.com/en-us/windows/win32/uxguide/progr... Like surely everything doesn't have to be SO 1980's if we want shell to stay relevant.
  - southpawflo 6y ago
    because then your shell will be slower than a snail. for real though, after looking at your examples I feel that features like those would fall on your terminal emulator to bring to the table, not the shell itself. specifically, I don't want to imagine trying to connect to a headless server during some crisis over bad wifi and have the shell think it needs to send a bunch of graphics to me.

chubot 6y ago

Oil is making shell more sane, in a way that you can realistically adopt, since it runs existing shell scripts. See: Four Features That Justify a New Unix Shell http://www.oilshell.org/blog/2020/10/osh-features.html http://www.oilshell.org/blog/2020/10/osh-features.html

tannhaeuser 6y ago

Your wish has been granted: awk '{ print $2 }' can be replaced by 'cut -f2'. And all of the commands are part of the set of POSIX command-line utilities having been included on Unix systems since basically forever, with awk (the one true awk) having been introduced only in 1983 or so.

tedunangst 6y ago
cut doesn't use the same field delimiters as awk.
- Scarbutt 6y ago
  And awk doesn't let you easily select a range of fields like cut

IshKebab 6y ago

I agree. There's nushell which looks a lot more sane, so I have hope that we won't be stuck with insane tools from the 80s for the rest of time! Maybe people will stop deifying those old flawed tools too.