4 ms·

Mostly off topic, but I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being

Mostly off topic, but I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being easy to write 40+ line scripts in. Why does that need to have the exact same syntax? This is the root cause of why bash (and batch and etc etc etc) are so terrible to script in (and why powershell is so terrible to type into a terminal). This weird shit like [ being a command, and having to think long and hard about exactly when variables are replaced and when globs are executed and how errors happen.. It's just pretty pointless. Like, if you write a script using Python or Ruby or Perl or whatever, there's easy ways to call into the shell. Often it's even built-in, like using backticks `echo like this`. These languages aren't designed for shell scripting but even they are often easier and cleaner than a bash script. It could be much better thoughk I bet. I'd love to have a language that is easy and familiar to type into a terminal (and thus, sufficiently bash-like probably), while being significantly easier to write scripts in. You could require the script to have a bit more syntax, a bit more structure, but essentially have the same semantics as what you'd type on the command line, and you'd have the best of both worlds. If well designed, it'd still "feel" like the same thing (much like how all JS programmers immediately grokked Coffeescript when it came out, because it really was just JS); everything you'd learn on the command line, you'd be able to apply in a script and vice versa. The script would just require more structure and thus provide more comfort (good error messages, editor support, etc). For example, globbing. The idea that the shell expands ".foo" into a list of files, before evaluating the command, is handy, but it's also an enormous footgun. What if on the terminal it would do that, but when scripting, you'd be required to write `glob(".foo")` or else it'd error out? THat would force you to clearly control when the expansion happens, it would let you assign it to a variable in a readable way etc. Imagine similar stuff for argument lists, escaping, variable substitution, etc - all the magic available on terminal, but contained in the script. Does this exist? I'd love to have it.

chubot 7y ago
(author here) Your first and third paragraph seem to be in conflict? I have heard the feedback that an interactive shell and a programming language should be different, which I disagree with. (For one, shell is already used as a programming language, so the ship has sailed. Also I use shell in the Lisp tradition: build your program up with a REPL.) But yes Oil is designed to "give the best of both worlds" as you say in the third paragraph. This recent post will probably give you the best idea of it: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... Rather than echo $mystr # silently splits echo "mystr" # no split In Oil, it's: echo $mystr # no split by default echo @split(mystr) # opt in explicitly This is enabled with options so existing shell scripts work. Also @split isn't really idiomatic since you would likely use arrays, but it could be useful for porting. I've also thought of adding a more explicit alternative for glob, but I actually like the shorter syntax here: echo *.py echo @glob('*.py') # not implemented but is consistent But I'm probably going to implement this safety feature https://github.com/oilshell/oil/issues/552 https://github.com/oilshell/oil/issues/552 to solve the well known issue where files that look like flags can confuse code and data. e.g. 'rm *' with a file named '-rf'.
- acqq 7y ago
  In "rather than" part > echo "mystr" # no split You probably meant to include $ echo "$mystr" # no split Still, echo is not good for these examples as for it the split doesn't matter and can't be detected, whereas for other commands (or even calls of functions) it indeed can matter. This example, I think, illustrates the differences between quoted and unquoted use of $name (as present in the current shells) better (to those who aren't familiar with the nuances of shells): f() { echo "param 1: |$1|" echo "param 2: |$2|" echo "param 3: |$3|" } x= y="55 66" f "$x" "$y" echo f $x $y #Output is: #param 1: || #param 2: |55 66| #param 3: || # #param 1: |55| #param 2: |66| #param 3: ||
  - chubot 7y ago
    Yes that was a typo (too late to fix it, oops) BTW, the fact that shell splits and echo joins is VERY confusing to a beginner. For example: mystr='a b c' # two spaces between each $ echo $mystr a b c # why is there only one space? I think this was "feature" during the times when it was OK to write C code like: char line[100]; // truncates lines longer than 100 chars But it's no longer OK! Filenames contain spaces, etc.
frou_dh 7y ago
> I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being easy to write 40+ line scripts in. It was probably less about that explicitly being considered the optimum design, and more about it being irresistible to save a lot of labor by getting 2 for the price of 1.
- analognoise 7y ago
  Spot on. You end up building a lot of things with hammers, saws and nails when all you have are hammers, saws and nails.
GSGBen 7y ago
(Replied to op instead of comment): Is your complaint about PowerShell the verbosity? Because I love its approach to this: full verbosity is the default (great for scripts), then terminal usage is improved by aliases (ForEach-Object is just %), great autocomplete via psreadline, and automatic parameter shortening: -re and -fo are automatically understood as -Recurse and -Force. Honestly makes every other shell a pain to use now for me. Also having everything be object based is just...I have no idea how I lived before.
- chubot 7y ago
  Here are my comments about PowerShell: https://news.ycombinator.com/item?id=22155943 https://news.ycombinator.com/item?id=22155943 Oil is really aimed at bash users and not PowerShell users. There is some overlap between the set, but only a bit.
- desc 7y ago
  Indeed, I do like scripting languages which are specific. Whatever the language, I do insist that people spell stuff out in a properly readable and standard fashion if writing a script. Aliases and shorthand are fine at a prompt but by damn if I ever see them in a Git commit... For the record, most other shells seem to understand autocompletion of command parameters these days, so hitting <tab> after --re should complete --recurse. Tangentially related and not aimed at parent post: <rant> My main complaint with Powershell is that it's a mess. I've had to write scripts in Powershell since version 1. We have to work with servers which are still running Windows 2008, and trying to get a Powershell script to run in the same way across 2008 - 2019 is next to impossible. People use Powershell because it's there and it's 'object-oriented' and the resulting maintenance is a terror because it started as a hack and actually fixing that will only make the compatibility story worse (some say this has already happened). (That option to run it in a versioned compatibility mode? Doesn't seem to work properly in practice. I have a suspicion that the differences in this case were down to some quirk of the console/terminal behaviour on varying Windows versions rather than Powershell itself, but that doesn't cut a lot of ice when there is literally no way to fix such-and-such-a-system in a way which works across all relevant machines without, you know, replacing 10 megabytes of PS with something else entirely. Unix shells tend not to have this problem because the integration points tend to just be STDIN, STDOUT and STDERR, and the shell does very little to mangle them.) If you write something for POSIX shell (no, not Bash) then it'll basically work on pretty much any Unix from the last few decades. No guarantees regarding the other things you invoke from it, but the shell itself is fairly minimal and consistent. If it's confining, use a more powerful language; you're probably outside its use case. My other complaint is that I find the object-oriented nature of Powershell to be nothing but a bloody nuisance. I can see how it might be useful sometimes (when wandering around at the prompt it's OK for discoverability) but trying to debug any nontrivial script is a fucking nightmare. 80% of any script ends up just being sanity checks producing useful error messages, because the errors provided by PS itself are rarely useful and usually misleading. (Actually that's inaccurate. It's more common that there are no sanity checks whatsoever, and the output of the script has no correlation with its purpose whatsoever because one function called early on yielded two values instead of one, so a variable was an array when the rest of the script assumed it was a scalar. The rest of the script then blithely continued on as if nothing was wrong, because some combination of properties on the result was missing/null (because it's an array, not a single object). The POSIX shell behaviour in such cases is usually to pass two paths instead of one to a command at some stage, which might cause an interesting error message, kill the script (`set -e` is your friend), or erase your hard disk (any of which may raise a ticket and maybe even get someone to fix the script).) Object types are great with statically-typed compiled languages (because your compiler/IDE will usually yell at your mistakes) and they're fine with any project which has a proper build process (because your test suite should yell at your mistakes) but for standalone scripts they're a spectacular example of a foot-seeking tactical nuke launcher. Net debugging time is less with a text-oriented shell because, when necessary, I can pipe STDIN or a text file to an individual command to see what it does, and what it prints to the terminal is exactly what it gave to my script. Some or all of the above might be fixed by having someone's personal definition of 'competent programmers', but fewer ways for things to go wrong before the code is actually run would be a good start. And no, 'Kubernetes' and 'Cloud' are not general solutions. Some of us still work with individual, separate machines in many legally distinct environments which do not permit the 'herd of cattle' and 'staged rollout' approach to server management. </rant>
kragen 7y ago
I agree that there's some tension between the design of a user interface and the design of a scripting language. But there are also some synergies. Two have been mentioned in the comments: you only have to implement one language interpreter instead of two, and you can build up a program interactively in the REPL before committing it to a file. But a third is that if a program can be invoked from a shell script, it can also be invoked interactively; you don't need to write a separate user interface for every program. And this, in many cases, results in programs being automatable with no extra effort. I agree that the Bourne Shell in particular is a pretty suboptimal language, and I think we can probably do a lot better. Tcl, es, rc, and PowerShell are four attempts. But it's easy to do worse. One way to reduce the tension is through interactions that go beyond just typing sequences of characters as if we were using an ASR-33. Filename tab-completion, for example, eliminates most of the uses of `ls` and `cd`, and current programmable completions for bash and zsh are pretty good at listing available completions for things that aren't even filenames. ^X * is a very handy way to do globbing, but you could usefully do it automatically, the way Slack's hated new WYSIWYG UI automatically applies pseudo-Markdown formatting to your messages as you type. John Cowan suggests that tab-completion should invoke the command with -h or --help to offer candidate completions, and Javier Candeira pointed out that this could be made safe by doing the invocation inside a new container to prevent things like du -h from taking undesired actions.
- frou_dh 7y ago
  > I agree that the Bourne Shell in particular is a pretty suboptimal language, and I think we can probably do a lot better. [...] But it's easy to do worse. I found this startlingly true when having to write nontrivial scripts using the classic Windows cmd.exe language (what's its actual name? Batch?). Bourne Shell's feature set, archaic as it is, is so much more conceptually unified. Exactly as you say, it's sobering to keep in mind that things can be a LOT worse.

chubot 7y ago

(author here) Your first and third paragraph seem to be in conflict? I have heard the feedback that an interactive shell and a programming language should be different, which I disagree with. (For one, shell is already used as a programming language, so the ship has sailed. Also I use shell in the Lisp tradition: build your program up with a REPL.) But yes Oil is designed to "give the best of both worlds" as you say in the third paragraph. This recent post will probably give you the best idea of it: http://www.oilshell.org/blog/2020/01/simplest-explanation.html http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... Rather than echo $mystr # silently splits echo "mystr" # no split In Oil, it's: echo $mystr # no split by default echo @split(mystr) # opt in explicitly This is enabled with options so existing shell scripts work. Also @split isn't really idiomatic since you would likely use arrays, but it could be useful for porting. I've also thought of adding a more explicit alternative for glob, but I actually like the shorter syntax here: echo *.py echo @glob('*.py') # not implemented but is consistent But I'm probably going to implement this safety feature https://github.com/oilshell/oil/issues/552 https://github.com/oilshell/oil/issues/552 to solve the well known issue where files that look like flags can confuse code and data. e.g. 'rm *' with a file named '-rf'.

acqq 7y ago
In "rather than" part > echo "mystr" # no split You probably meant to include $ echo "$mystr" # no split Still, echo is not good for these examples as for it the split doesn't matter and can't be detected, whereas for other commands (or even calls of functions) it indeed can matter. This example, I think, illustrates the differences between quoted and unquoted use of $name (as present in the current shells) better (to those who aren't familiar with the nuances of shells): f() { echo "param 1: |$1|" echo "param 2: |$2|" echo "param 3: |$3|" } x= y="55 66" f "$x" "$y" echo f $x $y #Output is: #param 1: || #param 2: |55 66| #param 3: || # #param 1: |55| #param 2: |66| #param 3: ||
- chubot 7y ago
  Yes that was a typo (too late to fix it, oops) BTW, the fact that shell splits and echo joins is VERY confusing to a beginner. For example: mystr='a b c' # two spaces between each $ echo $mystr a b c # why is there only one space? I think this was "feature" during the times when it was OK to write C code like: char line[100]; // truncates lines longer than 100 chars But it's no longer OK! Filenames contain spaces, etc.

frou_dh 7y ago

> I've never understood why shells are designed to have this weird dual use case of both being easy to type into a command prompt, and being easy to write 40+ line scripts in. It was probably less about that explicitly being considered the optimum design, and more about it being irresistible to save a lot of labor by getting 2 for the price of 1.

analognoise 7y ago
Spot on. You end up building a lot of things with hammers, saws and nails when all you have are hammers, saws and nails.

GSGBen 7y ago

(Replied to op instead of comment): Is your complaint about PowerShell the verbosity? Because I love its approach to this: full verbosity is the default (great for scripts), then terminal usage is improved by aliases (ForEach-Object is just %), great autocomplete via psreadline, and automatic parameter shortening: -re and -fo are automatically understood as -Recurse and -Force. Honestly makes every other shell a pain to use now for me. Also having everything be object based is just...I have no idea how I lived before.

chubot 7y ago
Here are my comments about PowerShell: https://news.ycombinator.com/item?id=22155943 https://news.ycombinator.com/item?id=22155943 Oil is really aimed at bash users and not PowerShell users. There is some overlap between the set, but only a bit.
desc 7y ago
Indeed, I do like scripting languages which are specific. Whatever the language, I do insist that people spell stuff out in a properly readable and standard fashion if writing a script. Aliases and shorthand are fine at a prompt but by damn if I ever see them in a Git commit... For the record, most other shells seem to understand autocompletion of command parameters these days, so hitting <tab> after --re should complete --recurse. Tangentially related and not aimed at parent post: <rant> My main complaint with Powershell is that it's a mess. I've had to write scripts in Powershell since version 1. We have to work with servers which are still running Windows 2008, and trying to get a Powershell script to run in the same way across 2008 - 2019 is next to impossible. People use Powershell because it's there and it's 'object-oriented' and the resulting maintenance is a terror because it started as a hack and actually fixing that will only make the compatibility story worse (some say this has already happened). (That option to run it in a versioned compatibility mode? Doesn't seem to work properly in practice. I have a suspicion that the differences in this case were down to some quirk of the console/terminal behaviour on varying Windows versions rather than Powershell itself, but that doesn't cut a lot of ice when there is literally no way to fix such-and-such-a-system in a way which works across all relevant machines without, you know, replacing 10 megabytes of PS with something else entirely. Unix shells tend not to have this problem because the integration points tend to just be STDIN, STDOUT and STDERR, and the shell does very little to mangle them.) If you write something for POSIX shell (no, not Bash) then it'll basically work on pretty much any Unix from the last few decades. No guarantees regarding the other things you invoke from it, but the shell itself is fairly minimal and consistent. If it's confining, use a more powerful language; you're probably outside its use case. My other complaint is that I find the object-oriented nature of Powershell to be nothing but a bloody nuisance. I can see how it might be useful sometimes (when wandering around at the prompt it's OK for discoverability) but trying to debug any nontrivial script is a fucking nightmare. 80% of any script ends up just being sanity checks producing useful error messages, because the errors provided by PS itself are rarely useful and usually misleading. (Actually that's inaccurate. It's more common that there are no sanity checks whatsoever, and the output of the script has no correlation with its purpose whatsoever because one function called early on yielded two values instead of one, so a variable was an array when the rest of the script assumed it was a scalar. The rest of the script then blithely continued on as if nothing was wrong, because some combination of properties on the result was missing/null (because it's an array, not a single object). The POSIX shell behaviour in such cases is usually to pass two paths instead of one to a command at some stage, which might cause an interesting error message, kill the script (`set -e` is your friend), or erase your hard disk (any of which may raise a ticket and maybe even get someone to fix the script).) Object types are great with statically-typed compiled languages (because your compiler/IDE will usually yell at your mistakes) and they're fine with any project which has a proper build process (because your test suite should yell at your mistakes) but for standalone scripts they're a spectacular example of a foot-seeking tactical nuke launcher. Net debugging time is less with a text-oriented shell because, when necessary, I can pipe STDIN or a text file to an individual command to see what it does, and what it prints to the terminal is exactly what it gave to my script. Some or all of the above might be fixed by having someone's personal definition of 'competent programmers', but fewer ways for things to go wrong before the code is actually run would be a good start. And no, 'Kubernetes' and 'Cloud' are not general solutions. Some of us still work with individual, separate machines in many legally distinct environments which do not permit the 'herd of cattle' and 'staged rollout' approach to server management. </rant>

kragen 7y ago

I agree that there's some tension between the design of a user interface and the design of a scripting language. But there are also some synergies. Two have been mentioned in the comments: you only have to implement one language interpreter instead of two, and you can build up a program interactively in the REPL before committing it to a file. But a third is that if a program can be invoked from a shell script, it can also be invoked interactively; you don't need to write a separate user interface for every program. And this, in many cases, results in programs being automatable with no extra effort. I agree that the Bourne Shell in particular is a pretty suboptimal language, and I think we can probably do a lot better. Tcl, es, rc, and PowerShell are four attempts. But it's easy to do worse. One way to reduce the tension is through interactions that go beyond just typing sequences of characters as if we were using an ASR-33. Filename tab-completion, for example, eliminates most of the uses of `ls` and `cd`, and current programmable completions for bash and zsh are pretty good at listing available completions for things that aren't even filenames. ^X * is a very handy way to do globbing, but you could usefully do it automatically, the way Slack's hated new WYSIWYG UI automatically applies pseudo-Markdown formatting to your messages as you type. John Cowan suggests that tab-completion should invoke the command with -h or --help to offer candidate completions, and Javier Candeira pointed out that this could be made safe by doing the invocation inside a new container to prevent things like du -h from taking undesired actions.

frou_dh 7y ago
> I agree that the Bourne Shell in particular is a pretty suboptimal language, and I think we can probably do a lot better. [...] But it's easy to do worse. I found this startlingly true when having to write nontrivial scripts using the classic Windows cmd.exe language (what's its actual name? Batch?). Bourne Shell's feature set, archaic as it is, is so much more conceptually unified. Exactly as you say, it's sobering to keep in mind that things can be a LOT worse.