11 ms·

Shell Has a Forth-Like Quality

136 points by chubot 10y ago

deleted 10y ago
[deleted]
ajkjk 10y ago
How hard would it to be to make an entirely new shell language that doesn't feel like it's 30+ years old? Why isn't this more common? I'm aware of, like, TermKit[0], but that didn't get finished and I don't recall hearing about any newer efforts. [0]: https://github.com/unconed/TermKit https://github.com/unconed/TermKit
- chubot 10y ago
  That's exactly what I'm doing :). The rest of the blog goes into some detail, although it's still in the early stages: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/ You can also see a list of other shells on the GitHub wiki page titled ExternalResources: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources I'm focusing on the shell as a programming language at first, in contrast to TermKit.
- mamcx 10y ago
  Build it is not the "hard" part. Anyway you can say a lot of languages are almost there (python, ruby, perl, forth, rebol). Is that developer are THE WORST. Them will refuse to switch to anything better if it means to a small change in his old ways. That is why C/C++/Js/Bash rule the world...
  - mamcx 10y ago
    BTW, I have thinking in build a shell/ipython hybrid. I think is worthless to try to emulate a shell (too much baggage), but instead build something that is alike a shell, but modern. However, still developers will not buy it. I think maybe it could work for a subset of the user that need to work with data-science and subset of system automation.
    - naringas 10y ago
      have you heard about xonsh? http://xon.sh/ http://xon.sh/
      - XorNot 10y ago
        I'm using xonsh at work, but currently the main thing I've noticed is that I still prefer iPython QT for exploring data and that xonsh starts up too slowly. Last I checked I think the thing I'm missing from using iPython purely as a shell is the ability to pipe?
  - hyperpape 10y ago
    It's somewhat difficult to come up with a good syntax. You need to support: unquoted strings, executing commands, pipes, file redirects, etc. Adoption would be even harder, but any resulting language is not going to look like Python or any other "mainstream" language, if it's intended to actually be usable.
    - chubot 10y ago
      Yeah this is a problem, but after writing a bash parser I think it's straightforward to solve. Oil is parsed line by line like any other shell, and there will be a simple rule to determine whether you're in command mode or expression mode. The "lexer modes" technique (previous referred to as "lexical state") should handle the two sublanguages easily. You basically need it to parse bash, and you will need much less of it to parse oil. My bash parser has 13 lexical states (up from 8), but I expect oil to have somewhere between 4 and 8. How it's used in the bash parser: http://www.oilshell.org/blog/2016/10/19.html http://www.oilshell.org/blog/2016/10/19.html
      - hyperpape 10y ago
        I'm not talking about implementation, but rather designing a clear and consistent syntax. It sounds like your intent is to hew pretty closely to existing shells, in which case you're probably doing more detail work.
        chubot 10y ago
        The strategy is to break compatibility with bash but provide an upgrade path, as mentioned here: https://news.ycombinator.com/item?id=13478291 https://news.ycombinator.com/item?id=13478291 It is a design issue to combine command and expression syntax (unquoted and vs. quoted literals). But I think I've figured it out ... we'll see in a few months! The blog will have updates on these kinds of issues.
        hyperpape 10y ago
        But are you maintaining POSIX compatibility? I thought I'd read that somewhere on your blog, but I might misremember.
  - StillBored 10y ago
    Beh, I guess i'm getting old then, because I spent ~10 years of my life (over 10 years ago) learning dozens of scripting and programming languages, and programming professionally in quite a number of them. The takeaway after all that was, rarely is one language better than another, its really about trade-offs. And frankly the older ones are a lot better at following the KISS principal and actually providing an uncluttered framework for writing _EFFICIENT_ code. For example, everyone knows that Java is a "better" and more productive language than COBOL, yet when you start talking to people trying to port old COBOL apps to java, what you discover is that it turns out that COBOL might actually do a better job in the business/transaction space. Same with python and Fortran in the HPC space, and strangely enough the core infrastructure all these more modern efficient languages are running on tends to be "C" based. Which despite the laundry list of issues that people like to parade around, seems to solve systems programming problems better than anything else. None of this would really be a problem except for the fact that the new up and coming language that is cool this year changes from year to year leaving a wasteland of abandoned poorly maintained projects using languages that no one really knew, and are despised by the people who are hired and have to learn them for that one job before moving on to the next one. So, yes bash is a POS, but its universal at this point, isn't really that bad at solving the core problems its designed to solve, and any developer with 1/2 a brain can write a sys V init script (or a lot of similar things) with little more than a couple hours and without the baggage of yet another 1/2GB language/framework/etc pulled in to sit alongside the dozen others already in the project. </rant> Now get off my yard.
- p10_user 10y ago
  What about fish? I use it on the terminal but still write my shell scripts in bash so they work across systems (including windows with minGW). I think a big issue with new shell adoption is the prevalence of bash basically everywhere.
  - kevin_thibedeau 10y ago
    > I think a big issue with new shell adoption is the prevalence of bash basically everywhere. Which is better than the days when csh and different Bourne flavors were common.
  - djsumdog 10y ago
    I've been using fish for over two years and love it. But the thing about fish is that is does totally break things that are expecting a sh/bash environment. There is more use out there and I've seen a number of shell tools/plugins include what to add to the fish config right under the bash and zsh sections (that's how I originally found out about fish). Fish does have bugs though. I've seen enough stack traces while running early builds to question the security (although I'm sure we all remember the bash shell-shcok bug that was laying around for years). If you haven't tried fish, I highly recommend it. It takes a little getting use to, but overall it adds a lot of really amazing functionality that I find really useful.
    - wtbob 10y ago
      > If you haven't tried fish, I highly recommend it. It takes a little getting use to, but overall it adds a lot of really amazing functionality that I find really useful. Serious question (that I've asked elsethread): what does fish offer than zsh & bash lack? A quick perusal of the docs reveals nothing obvious, but … I could be missing stuff.
      - ash 10y ago
        Other people indeed chimed in on your question in other comments. But here's my take about the most important fish features: * Instant auto suggestions, you just type stuff, like in a web browser. * Sane defaults, no need to bring around your own config. On a new machine I just install fish, run fish_config and customize the prompt with the web UI. * Scripting with simple syntax, especially for one-liners. Yes, fish is incompatible with bash, but it's less of issue in this case.
- taeric 10y ago
  Because there are many more than 30 years of learnings in the current shells that people ignore when building substitutes.
  - chubot 10y ago
    Yes I agree... that's why I took the trouble to write a very complete bash parser, which taught me a few things, and improved the Oil language design. Some features are plain mistakes, but some of them are useful things that newer shells aren't taking into account (particularly with regard to file descriptors). I'm aiming for a superset of bash semantics, but with different syntax. Importantly, it will be backward compatible, because bash can be converted to oil automatically. (I'm working on that now, not released yet.) http://www.oilshell.org/blog/2016/11/09.html http://www.oilshell.org/blog/2016/11/09.html The rest of the blog has a bunch of shell corner cases, like: http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/10/28.html http://www.oilshell.org/blog/2016/10/28.html Although I probably only blogged about 20% of the weird stuff I found.
    - taeric 10y ago
      Awesome posts, best of luck in this! Any suggestions/requests for users to help? And I realize I made an omission in my post. I meant to say not just learnings, but use. I would think the biggest hurdle for you will be building a user base.
      - chubot 10y ago
        Thanks a lot! I'm hoping to set things up for contributions in the next couple months. I'm want to publish the test matrix and get some help because filling out the corners takes a lot of time. The general architecture is pretty sound though -- I hit all the "major areas" of shell, although I may need to make another pass at globbing. If you are adventurous you should be able to clone it, follow the instructions for ./pybuild.sh. Then ./spec.sh install-shells, and then run './spec.sh smoke'. That test should pass, but there are like 15-20 other categories in spec.sh in various states of disarray! Of course this is in Python, but to me it makes sense to fill out a sketch before porting it to C++ (only the runtime; I'm hoping to bootstrap the parsers in oil itself). Not sure if that will inhibit or help contribution.
- kazinator 10y ago
  Unless you drop the requirement that the shell supports this type of usage: command arg1 arg2 ... then whatever is built to this requirement is going to feel decades old to some extent.
  - chubot 10y ago
    Why? There's nothing old about that syntax. On the contrary, I don't think anyone would use a shell WITHOUT the "cmd arg1 arg2" syntax. I certainly wouldn't want to. What feels ancient is the lack of good parse error messages and runtime error messages in bash. And syntax like a=X but not a = X or a = 'X' -- the difference being the spaces. And (echo hi) is OK for a subshell, but you need { echo hi; } for a group -- space and ; are required in the latter case, but not the former. The lack of proper functions in shell also feels ancient. For example, writing a completion function in bash is a pretty horrible experience. As another example, a simple function to escape HTML is either 'sed' or three statements like ${foo/&/&}, both of which are obtuse.
- nextos 10y ago
  I admire Unix, but I also feel things are getting old. I have a minimal Unix setup, with lots of little programs that do one thing and do it well. For example, mutt is my MUA (mail user agent). It just reads my emails, but fetching or sending is carried out by other programs. Fine. Now, the issue is that every little Unix utility that is non-trivial comes with its own little DSL for configuration. Most are inflexible, naive and different to each other. It's hard to script mutt. If you want to do some stuff the author hasn't planned for, you are better off patching the code. Flexibility, and composability, is not as good as they preach. For this reason, I am migrating everything to emacs. It offers me what Unix only promises but falls short at. Everything is an elisp function and thus easy to combine. Things are more organic. And still minimal.
  - pjmlp 10y ago
    > I admire Unix, but I also feel things are getting old. " We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy." Rob Pike, https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
    - nextos 10y ago
      Yes, it's a very good quote. It's sad most Plan 9 and Inferno innovations, just to name two further developments of Unix, have been ignored. But overall, I am more comfortable on live environments like Smalltalk or Emacs. There's just one fundamental abstraction, and things can be composed beautifully. In Unix, on the other hand, everything is not a file. And even if it was, little applications like mutt are silos. The code is static, it can't be easily changed or opened. A file interface, in the form of STDIN/STDOUT is too unstructured.
      - pjmlp 10y ago
- zeveb 10y ago
  I think that sweet-expressions (http://readable.sourceforge.net/ http://readable.sourceforge.net/) & CL-INTERPOL (http://weitz.de/cl-interpol/ http://weitz.de/cl-interpol/) could be the basis for a new kind of shell: let (foo bar) ls foo "bim/baz" ${foo} Might be turned into the equivalent of 'ls foo bim/baz bar' in sh. In fact, with a minimum of work I think this could be turned into a Lisp shell …
- equalunique 10y ago
  There's Microsoft PowerShell, an OO shell recently MIT-licensed and ported to Linux. Just throwing that one out there.
djsumdog 10y ago
I'm really excited about new shells. Fish has been mentioned here (which I've used for 2+ years). It's amazing and adds a lot of good functionality. I also like that the author mentioned runit. I wish there was more documentation for runit, but I've been using it on Void linux and it's an amazing, damn simply init system. If an init script is missing for any package in Void, they are trivial to write. Most are less than three lines. I also really like how the post author demonstrated how shell commands are composed in ways that traditional functions aren't composed; that is commands that change the environment for the command they run (environment variables, processor scheduling, security, input/output redirection). This is the very nature of an _implicit_.
- wtbob 10y ago
  > Fish has been mentioned here (which I've used for 2+ years). It's amazing and adds a lot of good functionality. What does it add exactly? As far as I can tell, it's needlessly incompatible with POSIX shells, and doesn't do as much as e.g. zsh (or even bash?). I'm not saying that zsh & bash are the end-all, be-all of shells, but I just don't see that fish improves on them. And honestly, for config files I think we could do a lot worse than an s-expression language.
  - shakna 10y ago
    Fish is similar to zsh in a lot of ways, but without configuration you get the best of zsh: * Syntax highlighting * Implicit cd to directory * 100,000 line history * History de-duplication * Inline auto-suggestions * Tab completion using man page data * Paginated completion * Copy/paste using X clipboard The only things I feel the need to configure on a new fish install, is PATH and my prompt. Because I'm pedantic and like things exactly so. You can get these with zsh, but it's more than apt-get install.
    - enriquto 10y ago
      Why the line history limit, though? I do not want that my lines get deleted silently. ever. I would rather prefer that random files on my disk are deleted, before that. Is there any shell without line history limit?
jasonhansel 10y ago
I think it would be interesting to have an init system and/or package manager configured using a logic programming language, such as a Prolog dialect. For instance, one could write "is_installed(X11) :- is_installed (KDE)" to specify that X11 must be installed if KDE is installed.
- iigs 10y ago
  I'm not sure how much of it is left today, but there is precedent for Prolog in OS startup: http://www.redditmirror.cc/cache/websites/web.archive.org_84624/web.archive.org/web/20040603192757/research.microsoft.com/research/dtg/davidhov/pap.htm http://www.redditmirror.cc/cache/websites/web.archive.org_84...
  - kazimuth 10y ago
    Are there two layers of caching in that URL?
  - jasonhansel 10y ago
    I wish that logic programming languages (and especially the one used in that article) had more friendly syntax. Even just replacing ":-" with "<-" would be nice...
    - OskarS 10y ago
      Actually, Prolog uses "-->" for to define grammars[1] for parsing. Not confusing at all! :) (actually, parsing is one of the things i really love about Prolog. Writing a simple grammars is much easier and much more pleasurable than using things like regular expressions. The fact that it's far more powerful and readable is just a bonus. ) [1] https://en.wikipedia.org/wiki/Definite_clause_grammar https://en.wikipedia.org/wiki/Definite_clause_grammar
- flukus 10y ago
  I'm sure the systemd people are working on one.
gwu78 10y ago
Easy way to do file descriptor passing ("Bernstein chaining") with any UNIX programs, not just ones written by djb: http://skarnet.org/software/execline/pipeline.html http://skarnet.org/software/execline/pipeline.html Execline resembles a "program launcher". We can make a braindead launcher in a few minutes using the sh language. This is just the first way that came to mind. #!/bin/sh test $# -ge 1|| exec echo usage: $0 program arguments test $# -le 5|| exec echo max 5 arguments { a=$#; b=$(command -v $1); echo 'main(){ char *b['$((a+1))'];' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[0]="'$1'";' # repeat a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[1]="'$2'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[2]="'$3'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[3]="'$4'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[4]="'$5'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[5]="'$6'";' } #| exec gcc -xc - -o run -static \ && exec run re: envdir, storing variables in the filesystem "no stinkin loops" or "stupid shell scripter" PoC, take 2 #!/bin/sh # e is a dir of "environmental variables" # e is mounted on tmpfs i.e., it's memory not disk # output: # ==> 0 # ==> 1 # ==> 2 # ==> 3 # ==> 4 # ==> 5 test -f e/$0||echo 0 > e/$0 read n < e/$0 test $n -le 5||exec rm e/$0 echo ==\> $n echo $((n+1)) > e/$0 test -f $0||exit # just in case $0
sigil 10y ago
Totally! Bernstein chaining is concatenative. So are shell pipelines, if you squint at them just right: function composition mediated by special stacks of records called stdin and stdout. Reminds me, I have a pretty neat tail-recursive shell pipeline trick I need to share with HN!
- stephancoral 10y ago
  Please share it!
- chubot 10y ago
  I'm interested in the tail recursive pipeline trick! The next post mentions that pipelines are also concatenative, although there is a slight difference and I think "point-free" is more accurate. "Pipelines Support Vectorized, Point-Free, and Imperative Style" http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
  - sigil 10y ago
    Nice. I'm loving this series, keep it up!
  - deleted 10y ago
    [deleted]
xiaq 10y ago
If you are interested in a shell with first-class function, list and map values and pipelines that pass such complex data (instead of structureless bytes), you might be interested in elvish: https://github.com/elves/elvish https://github.com/elves/elvish Disclaimer: I am the author of elvish.
- chubot 10y ago
  Hi xiaq I mentioned your shell here: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources If there is a better summary, let me know, or feel free to edit the wiki page yourself. (I think it should be editable.) I've been in contact with the authors of the oh shell, NGS shell, and mash shell, and exchanged some interesting ideas. The design of the oil language should be more concrete in 2-3 months and I'm hoping to get feedback from others who have thought about this space. I might ping you then :) Are the elvish structured data pipelines actually separate processes, or are they goroutines? Is there a format for piping them over say SSH? I have been thinking about this a little bit for oil.
  - xiaq 10y ago
    Hi chubot, I didn't realize that you are developing a shell as well. It's great to see competitions. Can you share some ideas that you learned from the authors of said shells? I would love to hear those, designing shells is not an easy task. Structured data pipelines run in the same process as elvish, in goroutines. For interacting with external commands you need to serialize and deserialize, like this: ~> put [foo bar lorem ipsum] | to-json | python3 -c 'import sys, json;print(json.dumps([x.upper() for x in json.load(sys.stdin)]))' | from-json ▶ [FOO BAR LOREM IPSUM]
    - chubot 10y ago
      OK cool, that makes sense. I had some good discussions with Michael MacInnis and recently Ilya Sher. I had actually read Michael's masters thesis in 2011 or 2012 and found it to be a very good historical overview of shells: https://scholar.google.com/scholar?cluster=9993440116444147516 https://scholar.google.com/scholar?cluster=99934401164441475... I agree with the goal of "taking shell seriously as a programming language", which I think all projects share. I disagreed on the need to make shell a variant of Lisp. I actually tried this a bit -- I hacked on femtolisp, which is used to bootstrap Julia, and I thought I could bootstrap a good shell with it. This didn't work out as nicely as I had hoped so I dropped the idea. Ilya was having some issues with the "two modes" of parsing -- command mode for shell-like constructs and expression mode for programming features (function calls, math, etc.). I thought this would be an issue too, but as mentioned in another comment, now that I've written a bash parser I think it's straightforward to solve, and Ilya agreed. The thing is that bash already has many different modes of parsing, which I implement using "lexer modes" or lexical state. This same technique can be applied to a new shell and then there should be a natural way of mixing commands and expressions (quoted and unquoted words, etc.): http://www.oilshell.org/blog/2016/10/19.html http://www.oilshell.org/blog/2016/10/19.html I'm also trying to make oil do what bash/dash/etc. do essentially syscall for syscall. I believe that is a little hard with Go because it has its own runtime. Although I think there is a way to pin a goroutine to a single thread which can mitigate those issues. I was planning to start a thread in maybe a month to review some of these issues and exchange feedback. Let me know if you are interested!
jacobolus 10y ago
Calling anything “oil shell” seems like a pretty good way to invite a trademark dispute from Shell Oil Company.
bloaf 10y ago
As someone not well versed in OS history / design: Why are terminals NOT just REPLs for regular programming languages with OS_Convenience_Functions #include'd automatically?
- skylerberg 10y ago
  General programming languages and shells have different priorities. Specifically a shell programming language should be designed to make launching external applications and processing their output the top priority. The language also needs to focus on easy of interactive input, so much so that the repl and the language or somewhat coupled. You wouldn't have a command in a general purpose programming language to execute the same line of code as you did 5 lines ago, that would be worse than goto. However, in a shell, such a thing sounds nice. The shell also needs to make it convenient to deal with some OS level objects, such as files. This is where glob syntax comes in handy. This all cascades to make a language that looks much different than you would want to write a full application in. One of the primary reasons is because of how shells need to handle strings. It is much more convenient to not type out quotes around string literals and it is nice to have syntax to reference environment variables, files, previous commands, etc. This leads to extremely complex quoting and expansion rules in most shells. It is not intuitive, but once learned it makes for a great interactive experience (compared with traditional languages).
- yumaikas 10y ago
  From a design standpoint: Because the ergonomics of shells and regular programming langauges are very different. 99% of your interactions with a shell is going to be single commands, or short piped chains. Economy of keystrokes is something that a lot of people want here, which is why cd and ls are both that short to type. Most programming languages require at least () to run commands, since not requiring that seems sloppy. This makes your two most common interactions twice as long to type. Shells have also been built to help glue applications together, and are more concise for the easy versions than most programming languages, sometimes heavily so. Most programming languages, (outside of maybe perl and ruby), don't make this use case anywhere close to their top priority, but shells do, trading off other things like a simple syntax for bitwise OR. In another sense, shells are exactly what you describe, it just happens that the most popular languages for them happen to be sh/bash/zsh/fish etc, which make a different set of tradeoffs than the 'traditional' programming languages. I'm only partially aware of the history standpoint. My understanding is that before awk, shells (sh, mostly, IIRC) were about the only scripting level languages on various Unixen.
GauntletWizard 10y ago
In addition, rather than docker, try sometime running a process with cgexec. It helps to know some of the underlying interfaces to get the full scope of what advanced compositions of those interfaces are doing.
JdeBP 10y ago
As I pointed out at https://news.ycombinator.com/item?id=13397381 https://news.ycombinator.com/item?id=13397381 , you forgot rtprio, idprio, chrt, and numactl; the s6, perp, daemontools-encore, and nosh toolsets; and execline. And TCL and the Thompson shell. * http://wiki.tcl.tk/15088 http://wiki.tcl.tk/15088 * http://jdebp.eu./FGA/daemontools-family.html http://jdebp.eu./FGA/daemontools-family.html * http://v6shell.org/ http://v6shell.org/
- chubot 10y ago
  Thanks, I did add s6 in response to your comment, but it wasn't meant to be exhaustive, so I left out the others. I didn't add Linux "perf" or a whole host of other such tools either. I'm not sure how TCL and the Thompson shell relate though.
  - JdeBP 10y ago
    Apart from the fact that you are re-using the Thompson shell's name, consider things like its if command. Yes, command; not reserved word. And my list is not exhaustive, either. But it makes the point to those that would erroneously dismiss this (as you yourself do) as something that "doesn't see much use today", that this, given how long the list of non-daemontools-family commands that operate this way actually turns out to be when one looks around, is not really a daemontools, or even a "Bernstein", thing.
SixSigma 10y ago
point free forth like does not make
aidenn0 10y ago
Is there a good place for discussing Oil with the author (mailing list, irc &c.)? I have thought a lot about many things in the blog posts, but there's no comment section, and I'd really like to get clarification as well as dive into a bit more detail on some of these things. [edit] Found it here: http://www.oilshell.org/blog/2016/12/29.html http://www.oilshell.org/blog/2016/12/29.html

136 points by chubot 10y ago

deleted 10y ago

[deleted]

ajkjk 10y ago

How hard would it to be to make an entirely new shell language that doesn't feel like it's 30+ years old? Why isn't this more common? I'm aware of, like, TermKit[0], but that didn't get finished and I don't recall hearing about any newer efforts. [0]: https://github.com/unconed/TermKit https://github.com/unconed/TermKit

chubot 10y ago
That's exactly what I'm doing :). The rest of the blog goes into some detail, although it's still in the early stages: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/ You can also see a list of other shells on the GitHub wiki page titled ExternalResources: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources I'm focusing on the shell as a programming language at first, in contrast to TermKit.
mamcx 10y ago
Build it is not the "hard" part. Anyway you can say a lot of languages are almost there (python, ruby, perl, forth, rebol). Is that developer are THE WORST. Them will refuse to switch to anything better if it means to a small change in his old ways. That is why C/C++/Js/Bash rule the world...
- mamcx 10y ago
  BTW, I have thinking in build a shell/ipython hybrid. I think is worthless to try to emulate a shell (too much baggage), but instead build something that is alike a shell, but modern. However, still developers will not buy it. I think maybe it could work for a subset of the user that need to work with data-science and subset of system automation.
  - naringas 10y ago
    have you heard about xonsh? http://xon.sh/ http://xon.sh/
    - XorNot 10y ago
      I'm using xonsh at work, but currently the main thing I've noticed is that I still prefer iPython QT for exploring data and that xonsh starts up too slowly. Last I checked I think the thing I'm missing from using iPython purely as a shell is the ability to pipe?
- hyperpape 10y ago
  It's somewhat difficult to come up with a good syntax. You need to support: unquoted strings, executing commands, pipes, file redirects, etc. Adoption would be even harder, but any resulting language is not going to look like Python or any other "mainstream" language, if it's intended to actually be usable.
  - chubot 10y ago
    Yeah this is a problem, but after writing a bash parser I think it's straightforward to solve. Oil is parsed line by line like any other shell, and there will be a simple rule to determine whether you're in command mode or expression mode. The "lexer modes" technique (previous referred to as "lexical state") should handle the two sublanguages easily. You basically need it to parse bash, and you will need much less of it to parse oil. My bash parser has 13 lexical states (up from 8), but I expect oil to have somewhere between 4 and 8. How it's used in the bash parser: http://www.oilshell.org/blog/2016/10/19.html http://www.oilshell.org/blog/2016/10/19.html
    - hyperpape 10y ago
      I'm not talking about implementation, but rather designing a clear and consistent syntax. It sounds like your intent is to hew pretty closely to existing shells, in which case you're probably doing more detail work.
      - chubot 10y ago
        The strategy is to break compatibility with bash but provide an upgrade path, as mentioned here: https://news.ycombinator.com/item?id=13478291 https://news.ycombinator.com/item?id=13478291 It is a design issue to combine command and expression syntax (unquoted and vs. quoted literals). But I think I've figured it out ... we'll see in a few months! The blog will have updates on these kinds of issues.
        hyperpape 10y ago
        But are you maintaining POSIX compatibility? I thought I'd read that somewhere on your blog, but I might misremember.
- StillBored 10y ago
  Beh, I guess i'm getting old then, because I spent ~10 years of my life (over 10 years ago) learning dozens of scripting and programming languages, and programming professionally in quite a number of them. The takeaway after all that was, rarely is one language better than another, its really about trade-offs. And frankly the older ones are a lot better at following the KISS principal and actually providing an uncluttered framework for writing _EFFICIENT_ code. For example, everyone knows that Java is a "better" and more productive language than COBOL, yet when you start talking to people trying to port old COBOL apps to java, what you discover is that it turns out that COBOL might actually do a better job in the business/transaction space. Same with python and Fortran in the HPC space, and strangely enough the core infrastructure all these more modern efficient languages are running on tends to be "C" based. Which despite the laundry list of issues that people like to parade around, seems to solve systems programming problems better than anything else. None of this would really be a problem except for the fact that the new up and coming language that is cool this year changes from year to year leaving a wasteland of abandoned poorly maintained projects using languages that no one really knew, and are despised by the people who are hired and have to learn them for that one job before moving on to the next one. So, yes bash is a POS, but its universal at this point, isn't really that bad at solving the core problems its designed to solve, and any developer with 1/2 a brain can write a sys V init script (or a lot of similar things) with little more than a couple hours and without the baggage of yet another 1/2GB language/framework/etc pulled in to sit alongside the dozen others already in the project. </rant> Now get off my yard.
p10_user 10y ago
What about fish? I use it on the terminal but still write my shell scripts in bash so they work across systems (including windows with minGW). I think a big issue with new shell adoption is the prevalence of bash basically everywhere.
- kevin_thibedeau 10y ago
  > I think a big issue with new shell adoption is the prevalence of bash basically everywhere. Which is better than the days when csh and different Bourne flavors were common.
- djsumdog 10y ago
  I've been using fish for over two years and love it. But the thing about fish is that is does totally break things that are expecting a sh/bash environment. There is more use out there and I've seen a number of shell tools/plugins include what to add to the fish config right under the bash and zsh sections (that's how I originally found out about fish). Fish does have bugs though. I've seen enough stack traces while running early builds to question the security (although I'm sure we all remember the bash shell-shcok bug that was laying around for years). If you haven't tried fish, I highly recommend it. It takes a little getting use to, but overall it adds a lot of really amazing functionality that I find really useful.
  - wtbob 10y ago
    > If you haven't tried fish, I highly recommend it. It takes a little getting use to, but overall it adds a lot of really amazing functionality that I find really useful. Serious question (that I've asked elsethread): what does fish offer than zsh & bash lack? A quick perusal of the docs reveals nothing obvious, but … I could be missing stuff.
    - ash 10y ago
      Other people indeed chimed in on your question in other comments. But here's my take about the most important fish features: * Instant auto suggestions, you just type stuff, like in a web browser. * Sane defaults, no need to bring around your own config. On a new machine I just install fish, run fish_config and customize the prompt with the web UI. * Scripting with simple syntax, especially for one-liners. Yes, fish is incompatible with bash, but it's less of issue in this case.
taeric 10y ago
Because there are many more than 30 years of learnings in the current shells that people ignore when building substitutes.
- chubot 10y ago
  Yes I agree... that's why I took the trouble to write a very complete bash parser, which taught me a few things, and improved the Oil language design. Some features are plain mistakes, but some of them are useful things that newer shells aren't taking into account (particularly with regard to file descriptors). I'm aiming for a superset of bash semantics, but with different syntax. Importantly, it will be backward compatible, because bash can be converted to oil automatically. (I'm working on that now, not released yet.) http://www.oilshell.org/blog/2016/11/09.html http://www.oilshell.org/blog/2016/11/09.html The rest of the blog has a bunch of shell corner cases, like: http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/11/06.html http://www.oilshell.org/blog/2016/10/28.html http://www.oilshell.org/blog/2016/10/28.html Although I probably only blogged about 20% of the weird stuff I found.
  - taeric 10y ago
    Awesome posts, best of luck in this! Any suggestions/requests for users to help? And I realize I made an omission in my post. I meant to say not just learnings, but use. I would think the biggest hurdle for you will be building a user base.
    - chubot 10y ago
      Thanks a lot! I'm hoping to set things up for contributions in the next couple months. I'm want to publish the test matrix and get some help because filling out the corners takes a lot of time. The general architecture is pretty sound though -- I hit all the "major areas" of shell, although I may need to make another pass at globbing. If you are adventurous you should be able to clone it, follow the instructions for ./pybuild.sh. Then ./spec.sh install-shells, and then run './spec.sh smoke'. That test should pass, but there are like 15-20 other categories in spec.sh in various states of disarray! Of course this is in Python, but to me it makes sense to fill out a sketch before porting it to C++ (only the runtime; I'm hoping to bootstrap the parsers in oil itself). Not sure if that will inhibit or help contribution.
kazinator 10y ago
Unless you drop the requirement that the shell supports this type of usage: command arg1 arg2 ... then whatever is built to this requirement is going to feel decades old to some extent.
- chubot 10y ago
  Why? There's nothing old about that syntax. On the contrary, I don't think anyone would use a shell WITHOUT the "cmd arg1 arg2" syntax. I certainly wouldn't want to. What feels ancient is the lack of good parse error messages and runtime error messages in bash. And syntax like a=X but not a = X or a = 'X' -- the difference being the spaces. And (echo hi) is OK for a subshell, but you need { echo hi; } for a group -- space and ; are required in the latter case, but not the former. The lack of proper functions in shell also feels ancient. For example, writing a completion function in bash is a pretty horrible experience. As another example, a simple function to escape HTML is either 'sed' or three statements like ${foo/&/&}, both of which are obtuse.
nextos 10y ago
I admire Unix, but I also feel things are getting old. I have a minimal Unix setup, with lots of little programs that do one thing and do it well. For example, mutt is my MUA (mail user agent). It just reads my emails, but fetching or sending is carried out by other programs. Fine. Now, the issue is that every little Unix utility that is non-trivial comes with its own little DSL for configuration. Most are inflexible, naive and different to each other. It's hard to script mutt. If you want to do some stuff the author hasn't planned for, you are better off patching the code. Flexibility, and composability, is not as good as they preach. For this reason, I am migrating everything to emacs. It offers me what Unix only promises but falls short at. Everything is an elisp function and thus easy to combine. Things are more organic. And still minimal.
- pjmlp 10y ago
  > I admire Unix, but I also feel things are getting old. " We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy." Rob Pike, https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
  - nextos 10y ago
    Yes, it's a very good quote. It's sad most Plan 9 and Inferno innovations, just to name two further developments of Unix, have been ignored. But overall, I am more comfortable on live environments like Smalltalk or Emacs. There's just one fundamental abstraction, and things can be composed beautifully. In Unix, on the other hand, everything is not a file. And even if it was, little applications like mutt are silos. The code is static, it can't be easily changed or opened. A file interface, in the form of STDIN/STDOUT is too unstructured.
    - pjmlp 10y ago
zeveb 10y ago
I think that sweet-expressions (http://readable.sourceforge.net/ http://readable.sourceforge.net/) & CL-INTERPOL (http://weitz.de/cl-interpol/ http://weitz.de/cl-interpol/) could be the basis for a new kind of shell: let (foo bar) ls foo "bim/baz" ${foo} Might be turned into the equivalent of 'ls foo bim/baz bar' in sh. In fact, with a minimum of work I think this could be turned into a Lisp shell …
equalunique 10y ago
There's Microsoft PowerShell, an OO shell recently MIT-licensed and ported to Linux. Just throwing that one out there.

djsumdog 10y ago

I'm really excited about new shells. Fish has been mentioned here (which I've used for 2+ years). It's amazing and adds a lot of good functionality. I also like that the author mentioned runit. I wish there was more documentation for runit, but I've been using it on Void linux and it's an amazing, damn simply init system. If an init script is missing for any package in Void, they are trivial to write. Most are less than three lines. I also really like how the post author demonstrated how shell commands are composed in ways that traditional functions aren't composed; that is commands that change the environment for the command they run (environment variables, processor scheduling, security, input/output redirection). This is the very nature of an _implicit_.

wtbob 10y ago
> Fish has been mentioned here (which I've used for 2+ years). It's amazing and adds a lot of good functionality. What does it add exactly? As far as I can tell, it's needlessly incompatible with POSIX shells, and doesn't do as much as e.g. zsh (or even bash?). I'm not saying that zsh & bash are the end-all, be-all of shells, but I just don't see that fish improves on them. And honestly, for config files I think we could do a lot worse than an s-expression language.
- shakna 10y ago
  Fish is similar to zsh in a lot of ways, but without configuration you get the best of zsh: * Syntax highlighting * Implicit cd to directory * 100,000 line history * History de-duplication * Inline auto-suggestions * Tab completion using man page data * Paginated completion * Copy/paste using X clipboard The only things I feel the need to configure on a new fish install, is PATH and my prompt. Because I'm pedantic and like things exactly so. You can get these with zsh, but it's more than apt-get install.
  - enriquto 10y ago
    Why the line history limit, though? I do not want that my lines get deleted silently. ever. I would rather prefer that random files on my disk are deleted, before that. Is there any shell without line history limit?

jasonhansel 10y ago

I think it would be interesting to have an init system and/or package manager configured using a logic programming language, such as a Prolog dialect. For instance, one could write "is_installed(X11) :- is_installed (KDE)" to specify that X11 must be installed if KDE is installed.

iigs 10y ago
I'm not sure how much of it is left today, but there is precedent for Prolog in OS startup: http://www.redditmirror.cc/cache/websites/web.archive.org_84624/web.archive.org/web/20040603192757/research.microsoft.com/research/dtg/davidhov/pap.htm http://www.redditmirror.cc/cache/websites/web.archive.org_84...
- kazimuth 10y ago
  Are there two layers of caching in that URL?
- jasonhansel 10y ago
  I wish that logic programming languages (and especially the one used in that article) had more friendly syntax. Even just replacing ":-" with "<-" would be nice...
  - OskarS 10y ago
    Actually, Prolog uses "-->" for to define grammars[1] for parsing. Not confusing at all! :) (actually, parsing is one of the things i really love about Prolog. Writing a simple grammars is much easier and much more pleasurable than using things like regular expressions. The fact that it's far more powerful and readable is just a bonus. ) [1] https://en.wikipedia.org/wiki/Definite_clause_grammar https://en.wikipedia.org/wiki/Definite_clause_grammar
flukus 10y ago
I'm sure the systemd people are working on one.

gwu78 10y ago

Easy way to do file descriptor passing ("Bernstein chaining") with any UNIX programs, not just ones written by djb: http://skarnet.org/software/execline/pipeline.html http://skarnet.org/software/execline/pipeline.html Execline resembles a "program launcher". We can make a braindead launcher in a few minutes using the sh language. This is just the first way that came to mind. #!/bin/sh test $# -ge 1|| exec echo usage: $0 program arguments test $# -le 5|| exec echo max 5 arguments { a=$#; b=$(command -v $1); echo 'main(){ char *b['$((a+1))'];' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[0]="'$1'";' # repeat a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[1]="'$2'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[2]="'$3'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[3]="'$4'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[4]="'$5'";' a=$((a-1)); test $a -ge 0|| exec echo 'b['$#']=(void *)0; execve("'$b'",b,(void *)0); }' echo 'b[5]="'$6'";' } #| exec gcc -xc - -o run -static \ && exec run re: envdir, storing variables in the filesystem "no stinkin loops" or "stupid shell scripter" PoC, take 2 #!/bin/sh # e is a dir of "environmental variables" # e is mounted on tmpfs i.e., it's memory not disk # output: # ==> 0 # ==> 1 # ==> 2 # ==> 3 # ==> 4 # ==> 5 test -f e/$0||echo 0 > e/$0 read n < e/$0 test $n -le 5||exec rm e/$0 echo ==\> $n echo $((n+1)) > e/$0 test -f $0||exit # just in case $0

sigil 10y ago

Totally! Bernstein chaining is concatenative. So are shell pipelines, if you squint at them just right: function composition mediated by special stacks of records called stdin and stdout. Reminds me, I have a pretty neat tail-recursive shell pipeline trick I need to share with HN!

stephancoral 10y ago
Please share it!
chubot 10y ago
I'm interested in the tail recursive pipeline trick! The next post mentions that pipelines are also concatenative, although there is a slight difference and I think "point-free" is more accurate. "Pipelines Support Vectorized, Point-Free, and Imperative Style" http://www.oilshell.org/blog/2017/01/15.html http://www.oilshell.org/blog/2017/01/15.html
- sigil 10y ago
  Nice. I'm loving this series, keep it up!
- deleted 10y ago
  [deleted]

xiaq 10y ago

If you are interested in a shell with first-class function, list and map values and pipelines that pass such complex data (instead of structureless bytes), you might be interested in elvish: https://github.com/elves/elvish https://github.com/elves/elvish Disclaimer: I am the author of elvish.

chubot 10y ago
Hi xiaq I mentioned your shell here: https://github.com/oilshell/oil/wiki/ExternalResources https://github.com/oilshell/oil/wiki/ExternalResources If there is a better summary, let me know, or feel free to edit the wiki page yourself. (I think it should be editable.) I've been in contact with the authors of the oh shell, NGS shell, and mash shell, and exchanged some interesting ideas. The design of the oil language should be more concrete in 2-3 months and I'm hoping to get feedback from others who have thought about this space. I might ping you then :) Are the elvish structured data pipelines actually separate processes, or are they goroutines? Is there a format for piping them over say SSH? I have been thinking about this a little bit for oil.
- xiaq 10y ago
  Hi chubot, I didn't realize that you are developing a shell as well. It's great to see competitions. Can you share some ideas that you learned from the authors of said shells? I would love to hear those, designing shells is not an easy task. Structured data pipelines run in the same process as elvish, in goroutines. For interacting with external commands you need to serialize and deserialize, like this: ~> put [foo bar lorem ipsum] | to-json | python3 -c 'import sys, json;print(json.dumps([x.upper() for x in json.load(sys.stdin)]))' | from-json ▶ [FOO BAR LOREM IPSUM]
  - chubot 10y ago
    OK cool, that makes sense. I had some good discussions with Michael MacInnis and recently Ilya Sher. I had actually read Michael's masters thesis in 2011 or 2012 and found it to be a very good historical overview of shells: https://scholar.google.com/scholar?cluster=9993440116444147516 https://scholar.google.com/scholar?cluster=99934401164441475... I agree with the goal of "taking shell seriously as a programming language", which I think all projects share. I disagreed on the need to make shell a variant of Lisp. I actually tried this a bit -- I hacked on femtolisp, which is used to bootstrap Julia, and I thought I could bootstrap a good shell with it. This didn't work out as nicely as I had hoped so I dropped the idea. Ilya was having some issues with the "two modes" of parsing -- command mode for shell-like constructs and expression mode for programming features (function calls, math, etc.). I thought this would be an issue too, but as mentioned in another comment, now that I've written a bash parser I think it's straightforward to solve, and Ilya agreed. The thing is that bash already has many different modes of parsing, which I implement using "lexer modes" or lexical state. This same technique can be applied to a new shell and then there should be a natural way of mixing commands and expressions (quoted and unquoted words, etc.): http://www.oilshell.org/blog/2016/10/19.html http://www.oilshell.org/blog/2016/10/19.html I'm also trying to make oil do what bash/dash/etc. do essentially syscall for syscall. I believe that is a little hard with Go because it has its own runtime. Although I think there is a way to pin a goroutine to a single thread which can mitigate those issues. I was planning to start a thread in maybe a month to review some of these issues and exchange feedback. Let me know if you are interested!

jacobolus 10y ago

Calling anything “oil shell” seems like a pretty good way to invite a trademark dispute from Shell Oil Company.

bloaf 10y ago

As someone not well versed in OS history / design: Why are terminals NOT just REPLs for regular programming languages with OS_Convenience_Functions #include'd automatically?

skylerberg 10y ago
General programming languages and shells have different priorities. Specifically a shell programming language should be designed to make launching external applications and processing their output the top priority. The language also needs to focus on easy of interactive input, so much so that the repl and the language or somewhat coupled. You wouldn't have a command in a general purpose programming language to execute the same line of code as you did 5 lines ago, that would be worse than goto. However, in a shell, such a thing sounds nice. The shell also needs to make it convenient to deal with some OS level objects, such as files. This is where glob syntax comes in handy. This all cascades to make a language that looks much different than you would want to write a full application in. One of the primary reasons is because of how shells need to handle strings. It is much more convenient to not type out quotes around string literals and it is nice to have syntax to reference environment variables, files, previous commands, etc. This leads to extremely complex quoting and expansion rules in most shells. It is not intuitive, but once learned it makes for a great interactive experience (compared with traditional languages).
yumaikas 10y ago
From a design standpoint: Because the ergonomics of shells and regular programming langauges are very different. 99% of your interactions with a shell is going to be single commands, or short piped chains. Economy of keystrokes is something that a lot of people want here, which is why cd and ls are both that short to type. Most programming languages require at least () to run commands, since not requiring that seems sloppy. This makes your two most common interactions twice as long to type. Shells have also been built to help glue applications together, and are more concise for the easy versions than most programming languages, sometimes heavily so. Most programming languages, (outside of maybe perl and ruby), don't make this use case anywhere close to their top priority, but shells do, trading off other things like a simple syntax for bitwise OR. In another sense, shells are exactly what you describe, it just happens that the most popular languages for them happen to be sh/bash/zsh/fish etc, which make a different set of tradeoffs than the 'traditional' programming languages. I'm only partially aware of the history standpoint. My understanding is that before awk, shells (sh, mostly, IIRC) were about the only scripting level languages on various Unixen.

GauntletWizard 10y ago

In addition, rather than docker, try sometime running a process with cgexec. It helps to know some of the underlying interfaces to get the full scope of what advanced compositions of those interfaces are doing.

JdeBP 10y ago

As I pointed out at https://news.ycombinator.com/item?id=13397381 https://news.ycombinator.com/item?id=13397381 , you forgot rtprio, idprio, chrt, and numactl; the s6, perp, daemontools-encore, and nosh toolsets; and execline. And TCL and the Thompson shell. * http://wiki.tcl.tk/15088 http://wiki.tcl.tk/15088 * http://jdebp.eu./FGA/daemontools-family.html http://jdebp.eu./FGA/daemontools-family.html * http://v6shell.org/ http://v6shell.org/

chubot 10y ago
Thanks, I did add s6 in response to your comment, but it wasn't meant to be exhaustive, so I left out the others. I didn't add Linux "perf" or a whole host of other such tools either. I'm not sure how TCL and the Thompson shell relate though.
- JdeBP 10y ago
  Apart from the fact that you are re-using the Thompson shell's name, consider things like its if command. Yes, command; not reserved word. And my list is not exhaustive, either. But it makes the point to those that would erroneously dismiss this (as you yourself do) as something that "doesn't see much use today", that this, given how long the list of non-daemontools-family commands that operate this way actually turns out to be when one looks around, is not really a daemontools, or even a "Bernstein", thing.

SixSigma 10y ago

point free forth like does not make

aidenn0 10y ago

Is there a good place for discussing Oil with the author (mailing list, irc &c.)? I have thought a lot about many things in the blog posts, but there's no comment section, and I'd really like to get clarification as well as dive into a bit more detail on some of these things. [edit] Found it here: http://www.oilshell.org/blog/2016/12/29.html http://www.oilshell.org/blog/2016/12/29.html