4 ms·

Es: A shell with higher-order functions (1993)

66 points by catern 7y ago

lolc 7y ago
Interesting! I keep getting frustrated with the arcane Bourne syntax and its limitations. I've toyed with a lot of shells. But when it comes down to it, I don't want to switch because the alternative is not ubiquitous.
- ra88it 7y ago
  I feel the same way. Would love to get behind a new shell that has a chance at ubiquity.
  - chubot 7y ago
    The basic thesis behind my Oil project is that a new shell won't become ubiquitous unless it's compatible with bash. In other words, the goal of the project is to replace bash. Just like any new compiled language needs the ability to reuse C code, I think any new shell has to somehow "reuse" existing shell code, rather than starting from scratch. I'm even more convinced of that now than I was 3 years ago when I started the project. Even I didn't realize how deep the "shell ecosystem" is, i.e. how much inertia there is to overcome. Why Create a New Unix Shell? http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html I'm looking for people to test it now! I made a release a couple days ago and have gotten several good bug reports, but I need more. It runs many real bash scripts: (1) Thousands of lines of distro build scripts (unmodified) (2) Thousands of lines of bash completion scripts (with slight patches) (3) scripts generated by virtualenv, git-prompt.sh, etc. https://github.com/oilshell/oil/wiki/How-To-Test-OSH https://github.com/oilshell/oil/wiki/How-To-Test-OSH Testing it out is straightforward but maybe not super friendly. That's what I would like feedback on and help with :) Feel free to join https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ (log in with Github) and give feedback. ---- EDIT: An important goal is to fill out this page of shell programs that Oil can run: https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run-Under-OSH https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run... Basically anything you use "in real life" is important!
    - ufo 7y ago
      Am I right to say that OSH is a subset of bash? If so, when does it differ from bash? I couldn't find this info on the blog. There are a lot of links :)
      - chubot 7y ago
        Roughly speaking, it's a "sane" subset. But it's also a large subset -- it aims to run all bash programs either unchanged or with trivial patches. So far that's been working out better than I expected. Occasionally a shell script will rely on something really odd, but that hasn't happened too often. For example, arithmetic in ksh/bash has some really horrible corner cases, but nobody relies on those. They mostly just increment counters and so forth. OSH also has some features that bash doesn't have, so it's not really a subset. Eventually there is supposed to be a legacy-free shell language called "Oil", but I've had my hands full with OSH, so I don't have much to say about that. But I think replacing bash is more important than creating an incompatible shell language that probably won't get used!
        ufo 7y ago
        Do you know if there is a comprdhensive list of known differences somewhere?
        ufo 7y ago
        Do you know if there is a comprehensive list of known differences somewhere?
        chubot 7y ago
        There are some difference listed here: https://github.com/oilshell/oil/blob/master/doc/osh-manual.md https://github.com/oilshell/oil/blob/master/doc/osh-manual.m... Those are the ones that I thought people might hit. There are surely some other differences, which may or may not matter in practice. This is why I'm looking for people to test OSH now. It's easier to change it sooner than later. If you want to use your favorite shell script unmodified with OSH, please test it :) Or let me know if you have difficulties testing it.
- bryan2 7y ago
  You could try zsh which is close enough to bash, but has a fair amount of modern features.
- mamcx 7y ago
  > I don't want to switch because the alternative is not ubiquitous. This is the sad truth about us, developers. "Disruptive" "Progress" "Tech advance" except, still suffering using C, C++, Shell, etc. Developers at large are so change averse that is not matched by any other group I think.
- nielsbot 7y ago
  I've switched to using python where possible.. but I guess there are reasons not to do that?
  - vthriller 7y ago
    Python feels way too cumbersome for day to day use, unless you do a lot of data crunching. There are bash/python hybrids like [0], but good luck getting them on some legacy production server that still runs distro from 8 years ago, or more modern but isolated server, or... There are (were?) also some language quirks that I can't remember because it's been a while since I last tried xonsh. [0] https://xon.sh https://xon.sh (EDIT: changed URL from an outdated one.)
  - claudiawerner 7y ago
    I've switched to Perl (and Perl 6) which seem to be better suited.
  - lolc 7y ago
    When it comes to scripting, I do prefer Python over Bash.
- codemac 7y ago
  I use Byron's rc every day, and it's installable on all machines I use. Though I also use dvorak and other things where I control the computer I input on 99%+ of the time. Do you not have access to a compiler or package manager on machines you use?
  - saagarjha 7y ago
    I may have access to them but lack the permissions or resources to install things.
  - lolc 7y ago
    Thanks, I'll check it out. But the thing is, if only a fraction of the time I spend in shells is a different shell, I can't build productive habits.
TimTheTinker 7y ago
> We applied concepts from modern functional programming languages, such as Scheme and ML, to shells Note that ML here refers to a functional programming language[0], not machine learning. [0] https://en.wikipedia.org/wiki/ML_(programming_language) https://en.wikipedia.org/wiki/ML_(programming_language)
xelxebar 7y ago
First order functions and the ability to override builtins. Definitely giving this a try. I've tried out several shells as primary drivers---bash, mksh, zsh, fish. These got me thinking more generally in terms of "userspace tools" instead of just interfaces with coreutils etc, which made me wonder, can we use our favorite REPLs in lieu of a standard shell? I've tried using Haskell's ghci as a "shell" and, fixable ergonomics aside, it seems doable. We already have interfaces to kernel syscalls, so in principle it's "just" a libraries problem. File I/O is probably the easiest, and with /proc and /sys, that's enough to replace several info-gathering system tools, e.g. df, lscpu, mount (just listing). Really wish I could immerse myself in Haskell for a few years and see if I could learn enough to take this REPLshell idea somewhere.
injinj 7y ago
The problem with first class functions in a shell is that the main duty of a shell is to fork processes. Once the shell is forked, the only communication available are inter process. All of the environment is no longer shared. For example, in es: ; let (i = 0) { fn hello {echo $i; i = <={%sum $i 1}}} ; hello 0 ; hello 1 The closure variable i is incremented after every invocation of hello, but if hello in a subprocess, then i doesn't change: ; hello | wc -c 2 ; hello 2 This is because hello is a subprocess of the main shell when it is in a pipeline. When it is used without the pipe, then it is executed within the shell environment, not a child. I have been using a fork of es since Dec, mine has the %sum builtin above, along with some other things: https://github.com/injinj/desh https://github.com/injinj/desh
ErotemeObelus 7y ago
This is really cool & sweet.

66 points by catern 7y ago

lolc 7y ago

Interesting! I keep getting frustrated with the arcane Bourne syntax and its limitations. I've toyed with a lot of shells. But when it comes down to it, I don't want to switch because the alternative is not ubiquitous.

ra88it 7y ago
I feel the same way. Would love to get behind a new shell that has a chance at ubiquity.
- chubot 7y ago
  The basic thesis behind my Oil project is that a new shell won't become ubiquitous unless it's compatible with bash. In other words, the goal of the project is to replace bash. Just like any new compiled language needs the ability to reuse C code, I think any new shell has to somehow "reuse" existing shell code, rather than starting from scratch. I'm even more convinced of that now than I was 3 years ago when I started the project. Even I didn't realize how deep the "shell ecosystem" is, i.e. how much inertia there is to overcome. Why Create a New Unix Shell? http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html I'm looking for people to test it now! I made a release a couple days ago and have gotten several good bug reports, but I need more. It runs many real bash scripts: (1) Thousands of lines of distro build scripts (unmodified) (2) Thousands of lines of bash completion scripts (with slight patches) (3) scripts generated by virtualenv, git-prompt.sh, etc. https://github.com/oilshell/oil/wiki/How-To-Test-OSH https://github.com/oilshell/oil/wiki/How-To-Test-OSH Testing it out is straightforward but maybe not super friendly. That's what I would like feedback on and help with :) Feel free to join https://oilshell.zulipchat.com/ https://oilshell.zulipchat.com/ (log in with Github) and give feedback. ---- EDIT: An important goal is to fill out this page of shell programs that Oil can run: https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run-Under-OSH https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run... Basically anything you use "in real life" is important!
  - ufo 7y ago
    Am I right to say that OSH is a subset of bash? If so, when does it differ from bash? I couldn't find this info on the blog. There are a lot of links :)
    - chubot 7y ago
      Roughly speaking, it's a "sane" subset. But it's also a large subset -- it aims to run all bash programs either unchanged or with trivial patches. So far that's been working out better than I expected. Occasionally a shell script will rely on something really odd, but that hasn't happened too often. For example, arithmetic in ksh/bash has some really horrible corner cases, but nobody relies on those. They mostly just increment counters and so forth. OSH also has some features that bash doesn't have, so it's not really a subset. Eventually there is supposed to be a legacy-free shell language called "Oil", but I've had my hands full with OSH, so I don't have much to say about that. But I think replacing bash is more important than creating an incompatible shell language that probably won't get used!
      - ufo 7y ago
        Do you know if there is a comprdhensive list of known differences somewhere?
      - ufo 7y ago
        Do you know if there is a comprehensive list of known differences somewhere?
        chubot 7y ago
        There are some difference listed here: https://github.com/oilshell/oil/blob/master/doc/osh-manual.md https://github.com/oilshell/oil/blob/master/doc/osh-manual.m... Those are the ones that I thought people might hit. There are surely some other differences, which may or may not matter in practice. This is why I'm looking for people to test OSH now. It's easier to change it sooner than later. If you want to use your favorite shell script unmodified with OSH, please test it :) Or let me know if you have difficulties testing it.
bryan2 7y ago
You could try zsh which is close enough to bash, but has a fair amount of modern features.
mamcx 7y ago
> I don't want to switch because the alternative is not ubiquitous. This is the sad truth about us, developers. "Disruptive" "Progress" "Tech advance" except, still suffering using C, C++, Shell, etc. Developers at large are so change averse that is not matched by any other group I think.
nielsbot 7y ago
I've switched to using python where possible.. but I guess there are reasons not to do that?
- vthriller 7y ago
  Python feels way too cumbersome for day to day use, unless you do a lot of data crunching. There are bash/python hybrids like [0], but good luck getting them on some legacy production server that still runs distro from 8 years ago, or more modern but isolated server, or... There are (were?) also some language quirks that I can't remember because it's been a while since I last tried xonsh. [0] https://xon.sh https://xon.sh (EDIT: changed URL from an outdated one.)
- claudiawerner 7y ago
  I've switched to Perl (and Perl 6) which seem to be better suited.
- lolc 7y ago
  When it comes to scripting, I do prefer Python over Bash.
codemac 7y ago
I use Byron's rc every day, and it's installable on all machines I use. Though I also use dvorak and other things where I control the computer I input on 99%+ of the time. Do you not have access to a compiler or package manager on machines you use?
- saagarjha 7y ago
  I may have access to them but lack the permissions or resources to install things.
- lolc 7y ago
  Thanks, I'll check it out. But the thing is, if only a fraction of the time I spend in shells is a different shell, I can't build productive habits.

TimTheTinker 7y ago

> We applied concepts from modern functional programming languages, such as Scheme and ML, to shells Note that ML here refers to a functional programming language[0], not machine learning. [0] https://en.wikipedia.org/wiki/ML_(programming_language) https://en.wikipedia.org/wiki/ML_(programming_language)

xelxebar 7y ago

First order functions and the ability to override builtins. Definitely giving this a try. I've tried out several shells as primary drivers---bash, mksh, zsh, fish. These got me thinking more generally in terms of "userspace tools" instead of just interfaces with coreutils etc, which made me wonder, can we use our favorite REPLs in lieu of a standard shell? I've tried using Haskell's ghci as a "shell" and, fixable ergonomics aside, it seems doable. We already have interfaces to kernel syscalls, so in principle it's "just" a libraries problem. File I/O is probably the easiest, and with /proc and /sys, that's enough to replace several info-gathering system tools, e.g. df, lscpu, mount (just listing). Really wish I could immerse myself in Haskell for a few years and see if I could learn enough to take this REPLshell idea somewhere.

injinj 7y ago

The problem with first class functions in a shell is that the main duty of a shell is to fork processes. Once the shell is forked, the only communication available are inter process. All of the environment is no longer shared. For example, in es: ; let (i = 0) { fn hello {echo $i; i = <={%sum $i 1}}} ; hello 0 ; hello 1 The closure variable i is incremented after every invocation of hello, but if hello in a subprocess, then i doesn't change: ; hello | wc -c 2 ; hello 2 This is because hello is a subprocess of the main shell when it is in a pipeline. When it is used without the pipe, then it is executed within the shell environment, not a child. I have been using a fork of es since Dec, mine has the %sum builtin above, along with some other things: https://github.com/injinj/desh https://github.com/injinj/desh

ErotemeObelus 7y ago

This is really cool & sweet.