4 ms·

I've thought about this a lot, but assuming you want to be pleasant for interactive use, you're subject to a tremendous number of constraints. Just to start wit

I've thought about this a lot, but assuming you want to be pleasant for interactive use, you're subject to a tremendous number of constraints. Just to start with: rm foo bar rm -rf * must be invocations of the rm command with the appropriate arguments. You need bare strings, you need variable interpolation, you need redirects. I think there's some room to break a few idioms and make something nicer, but it really is a hard problem. The choice to break things has to be based on how common an idiom is, how painful it is to not use it, and how much benefit you can get.

laumars 8y ago
I think you've hit the nail on the head there. I've been experimenting with writing my own shell and the original syntax was a lot more like Javascript but having to encapsulate parameters in parentheses, comma delimited and quotation was a massive pain in the arse. So I dropped the requirement for quotation marks, then made commas optional, then I dropped using periods to pipe (ala methods in OOP) and then eventually ended up rewriting the entire parser to follow a more traditional POSIX syntax. As it turns out, as ugly as Bash et al is, it actually makes a lot of sense for writing one liners in an interactive shell (as you stated yourself) So where I decided to deviate was areas I felt could be enhanced without breaking the POSIX syntax too significantly: * shell configuration, * error handling (something traditional shells are appallingly bad at) * and support for complex data structures (eg so you can grep through items in a minified JSON array as smartly as you can with a traditional stream of lines. I always respected the power of POSIX shells even before embarking on my pet project. However I never quite appreciated just how sane it's ugly syntax was until I attempted to replace it with something more readable. I mean sure there are still some specific areas I really don't agree with but that can always be argued as personal preference. As an aside note, one thing I didn't quite appreciate until writing my shell is just how much shells have in common with functional programming. Yes I know it's a far cry from LISP machines; but if you take away the variable interpolation and environmental variables then you're left with a functional pipeline that take something from STDIN and write it to STDOUT and STDERR and does so in a multi-threaded, concurrent, workflow by default. Weirdly this still makes POSIX shells more efficient for some types of data processing than writing a monolithic routine in a more powerful (or should that be "more descriptive"?) programming language. So there is some surprising elegance amongst all that ugliness.
- greenyouse 8y ago
  On the data processing with bash part, Joyent Manta always looked interesting[0]. Reading about data processing via bash makes it sound pretty fun. [0]https://github.com/joyent/manta/blob/master/README.md https://github.com/joyent/manta/blob/master/README.md
- adrianN 8y ago
  jq is pretty neat for dealing with JSON in a shell script.
  - laumars 8y ago
    jq is a great tool but I wanted something that was more seamless than piping data to jq with a jq script quoted as a parameter. I wanted something where the shell itself could fragment and understand structured data (and not just JSON but YAML, TOML, HCL, CSV, XML, etc - though XML is proving more a great deal more troublesome than I'd hoped). I'm not suggesting the work I'm doing is any better that jq though - there's a lot of areas where jq will run circles around my shell. I see it more as different design goals but with a fairly large area of overlap.
    - vram22 8y ago
      What is HCL? This? https://github.com/hashicorp/hcl https://github.com/hashicorp/hcl
      - laumars 8y ago
        Yeah. Personally I hate it. But it's what Terraform uses so sometimes it's handy having another tool to pull bits from Terraform scripts.
- madmulita 8y ago
  Why don't we have something similar to vi modes to jump between different 'interpretations'? I believe scsh had something to that effect using chars to switch modes ala quote/unquote. Maybe something based on Color Forth's ideas?
chubot 8y ago
In OSH I use the technique of "lexer modes" to handle all the cases you mention: http://www.oilshell.org/blog/2017/12/17.html http://www.oilshell.org/blog/2017/12/17.html I have a few more blog posts on that topic I haven't published, but hopefully that gives the idea. I probably used to agree with you (if I'm understanding correctly; it's not entirely clear what the conflict you see is). But after writing the parser in this style, I don't think it's too big a deal. I plan to use the same technique to parse the Oil language, which is a new language without the legacy. The command syntax is largely the same, but compound constructs like function, if, while/for, etc. are different. The parser is regularly tested on over a million lines of shell: http://www.oilshell.org/blog/2017/11/10.html http://www.oilshell.org/blog/2017/11/10.html Note that Python, JavaScript, and Swift all need something like this too, now that they have arbitrary code inside string literals, just like shell: print(f'three = {1 + 2}') # Python f-strings console.log(`three = ${1 + 2}`) # JavaScript println("three = \(1 + 2)") # Swift

laumars 8y ago

I think you've hit the nail on the head there. I've been experimenting with writing my own shell and the original syntax was a lot more like Javascript but having to encapsulate parameters in parentheses, comma delimited and quotation was a massive pain in the arse. So I dropped the requirement for quotation marks, then made commas optional, then I dropped using periods to pipe (ala methods in OOP) and then eventually ended up rewriting the entire parser to follow a more traditional POSIX syntax. As it turns out, as ugly as Bash et al is, it actually makes a lot of sense for writing one liners in an interactive shell (as you stated yourself) So where I decided to deviate was areas I felt could be enhanced without breaking the POSIX syntax too significantly: * shell configuration, * error handling (something traditional shells are appallingly bad at) * and support for complex data structures (eg so you can grep through items in a minified JSON array as smartly as you can with a traditional stream of lines. I always respected the power of POSIX shells even before embarking on my pet project. However I never quite appreciated just how sane it's ugly syntax was until I attempted to replace it with something more readable. I mean sure there are still some specific areas I really don't agree with but that can always be argued as personal preference. As an aside note, one thing I didn't quite appreciate until writing my shell is just how much shells have in common with functional programming. Yes I know it's a far cry from LISP machines; but if you take away the variable interpolation and environmental variables then you're left with a functional pipeline that take something from STDIN and write it to STDOUT and STDERR and does so in a multi-threaded, concurrent, workflow by default. Weirdly this still makes POSIX shells more efficient for some types of data processing than writing a monolithic routine in a more powerful (or should that be "more descriptive"?) programming language. So there is some surprising elegance amongst all that ugliness.

greenyouse 8y ago
On the data processing with bash part, Joyent Manta always looked interesting[0]. Reading about data processing via bash makes it sound pretty fun. [0]https://github.com/joyent/manta/blob/master/README.md https://github.com/joyent/manta/blob/master/README.md
adrianN 8y ago
jq is pretty neat for dealing with JSON in a shell script.
- laumars 8y ago
  jq is a great tool but I wanted something that was more seamless than piping data to jq with a jq script quoted as a parameter. I wanted something where the shell itself could fragment and understand structured data (and not just JSON but YAML, TOML, HCL, CSV, XML, etc - though XML is proving more a great deal more troublesome than I'd hoped). I'm not suggesting the work I'm doing is any better that jq though - there's a lot of areas where jq will run circles around my shell. I see it more as different design goals but with a fairly large area of overlap.
  - vram22 8y ago
    What is HCL? This? https://github.com/hashicorp/hcl https://github.com/hashicorp/hcl
    - laumars 8y ago
      Yeah. Personally I hate it. But it's what Terraform uses so sometimes it's handy having another tool to pull bits from Terraform scripts.
madmulita 8y ago
Why don't we have something similar to vi modes to jump between different 'interpretations'? I believe scsh had something to that effect using chars to switch modes ala quote/unquote. Maybe something based on Color Forth's ideas?

chubot 8y ago

In OSH I use the technique of "lexer modes" to handle all the cases you mention: http://www.oilshell.org/blog/2017/12/17.html http://www.oilshell.org/blog/2017/12/17.html I have a few more blog posts on that topic I haven't published, but hopefully that gives the idea. I probably used to agree with you (if I'm understanding correctly; it's not entirely clear what the conflict you see is). But after writing the parser in this style, I don't think it's too big a deal. I plan to use the same technique to parse the Oil language, which is a new language without the legacy. The command syntax is largely the same, but compound constructs like function, if, while/for, etc. are different. The parser is regularly tested on over a million lines of shell: http://www.oilshell.org/blog/2017/11/10.html http://www.oilshell.org/blog/2017/11/10.html Note that Python, JavaScript, and Swift all need something like this too, now that they have arbitrary code inside string literals, just like shell: print(f'three = {1 + 2}') # Python f-strings console.log(`three = ${1 + 2}`) # JavaScript println("three = \(1 + 2)") # Swift