16 ms·

Four features that justify a new Unix shell

242 points by diegocg 6y ago

autoexec 6y ago
New shell or not I hope we see updates to existing shells to do better error handling too
- chubot 6y ago
  (author here) Yes, I actually proposed that in the comments [1]. Some should add command_sub_errexit to bash: echo $(date %x) # can you tell what's wrong here? echo 'script should fail before this' local d=$(date %x) echo 'script should fail before this' And process_sub_fail: diff <(sort left.txt) <(sort /oops/error) echo 'script should fail before this' As mentioned in the comment, bash doesn't even wait() on process subs! The 'run' builtin to fix the "if myfunc" problem should also be adopted by other shells. POSIX basically calcified broken language semantics and needs to be fixed. I also document stuff like shopt -s simple_word_eval for other shell implementers: https://www.oilshell.org/release/0.8.3/doc/simple-word-eval.html https://www.oilshell.org/release/0.8.3/doc/simple-word-eval.... ----- BTW Oil's code is very short (see comment), and I'm looking for help :) Let me know if you can't get a bin/osh working in 1 to 5 minutes. (If you have a Debian/Ubuntu-ish machine it should take about 1 minute.) This is a pure Python program that you can quickly modify/prototype, and then for the release it's translated into C++ for a 30-50x speedup [2]. https://github.com/oilshell/oil/wiki/Contributing https://github.com/oilshell/oil/wiki/Contributing [1] https://lobste.rs/s/qfiki1/four_features_justify_new_unix_shell#c_ncyhoh https://lobste.rs/s/qfiki1/four_features_justify_new_unix_sh... [2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html
breatheoften 6y ago
Are there any transpile to bash languages? I like the idea of a language that allows for a better experience in writing shell scripts -- and produces shell scripts with fewer bugs on the output side. But i wonder if a transpile approach wouldn't be better in the long run
- kccqzy 6y ago
  What do you think produces the traditional ./configure scripts that you run to configure an open source project prior to building it? Did some human write these ./configure scripts manually?
  - codetrotter 6y ago
    Autoconf isn’t flexible enough for general purpose scripting though, is it? So not really relevant I think.
  - viraptor 6y ago
    M4 doesn't really satisfy: > better experience in writing It will happily transform a small mistake into garbage. Then you have to debug both the transformed result and the original script to match up what you intended.
- chubot 6y ago
  (author here) Yeah I've been asked that before, and there are some things you can fix with a transpiler, but somethings you can't. That's why the tagline to Oil is now "our upgrade path from bash to a better language AND RUNTIME". The shell runtime needs to be fixed too, e.g. the error handling mentioned in the blog post and that I follow up on here: https://news.ycombinator.com/item?id=24872986 https://news.ycombinator.com/item?id=24872986 You might be able to compile that to some really long bash, but I don't think it works in general, because editing deployed scripts to fix stuff is useful (yes in production, but also on your own machine. Think of all the shell that people put on it, like virtualenv, nix-shell, rustup, etc.) You don't want to end up with the JavaScript problem: you have a dynamic language, but ALSO A BUILD PROCESS, which is basically the worst of both worlds.
  - breatheoften 6y ago
    So is the normal usage path to start using osh for interactive use and then use osh in a bash compatible mode to write scripts that are less buggy but can still be run on bash? Or is it more common to write scripts that tend to then require the osh runtime exist in whatever context they are deployed to?
xilni 6y ago
I love the direction this wants to go in. So many of these are reasons why we've banned new shell scripts on our projects and instead make python scripts.
- ritchiea 6y ago
  Are there ever good reasons to choose a shell script over a Python/Ruby/etc script?
  - chubot 6y ago
    Yes, a couple answers here: http://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts http://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-dis...
    - neolog 6y ago
      > It won't work. It would be like trying to convince people who are paid to write PHP not to write PHP. Many people have wasted breath on that, but important sites like Wikipedia are still written in hundreds of thousands of lines of PHP. If they're gonna keep writing bash, why would a new shell language help? > Even if a new line of shell never gets written, there will still be a huge installed base of shell scripts that you may need to understand (e.g. when they don't work). If the old scripts are in bash, why would a new shell language help? > Shell is still the best tool for many jobs. Most new "cloud" projects rely on Linux system images, in VMs or containers, and shell is the ideal language for creating such images. What's good about shell for this task?
      - chubot 6y ago
        Because you can run your bash scripts with Oil. The tagline is: It's our upgrade path from bash to a better language and runtime. [1] It's basically the same as JS -> TypeScript, or PHP -> Hack. It's a saner language (and runtime) that runs existing code. ----- Shell is good for creating Unix systems because of the tools it provides. Ones that deal with the file system and heterogeneous processes (i.e. stuff you didn't write in different languages). It's hard to explain, but if you work in that area, you'll very quickly see it. You could also do something like Linux From Scratch [2] and it will be very clear why shell is used. [1] http://www.oilshell.org/ http://www.oilshell.org/ [2] http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/
3np 6y ago
Two ergonomic issues in POSIX/bash not mentioned that I’d like to see a new shell improve on: - splitting pipes to separate flows (e.g one flow for stderr, one for stdout). Think of it as a graph. - native parallelization; I have to look in man every other time I use parallel.
- RMPR 6y ago
  > native parallelization; I have to look in man every other time I use parallel. Can relate, the syntax is very strange, at least to me. And I wonder where it came from.
- Splognosticus 6y ago
  > - splitting pipes to separate flows (e.g one flow for stderr, one for stdout). Think of it as a graph. Could you elaborate a little on what you mean by this? bash already has the ability to redirect STDERR and STDOUT independently, and when you introduce named pipes into the mix things can get really fancy (probably too fancy, this is about where your script stops being a script and starts being a program written in perhaps the most inconvenient language available.)
  - e12e 6y ago
    I imagine an easy/intuitive way to set up FIFOs for stdout/stderr with redirection, maybe? A way to say: for this following section, I want stderr to go trough "|sort -n|logger -t DEBUG -f -" and standard out to go through (... Other pipeline)?
    - folmar 6y ago
      command 2> >(sort -n|logger -t DEBUG -f -) | other pipeline
- titanomachy 6y ago
  The first is interesting. I have a hard time imagining what that syntax would look like. You'd need a way to refer to your graph nodes, so I don't think you would get much better than what named pipes (aka FIFOs) already provide. Totally agree about GNU parallel. The syntax is not badly-designed IMO, but I don't use it very frequently and it's different from everything else so I often get it wrong. Using it as a drop-in replacement for xargs is easy enough though.
abnry 6y ago
I am very sympathetic to this. I like the concept of a shell (there's a place for light scripting that doesn't involve python or perl or ruby), and appreciate what bash can do, but after encountering enough of the "gotchas" and just ugly behavior, I'm not really motivated to master it. The biggest reason to learn it seems to be the fact that everybody uses it and has done so for years. We need something akin to Jupyter notebooks for unix.
- chubot 6y ago
  Yes, I would like for that to happen: Shell as an engine for TUI or GUI https://github.com/oilshell/oil/issues/738 https://github.com/oilshell/oil/issues/738 C++ code should be embeddable in another program https://github.com/oilshell/oil/issues/822 https://github.com/oilshell/oil/issues/822 I think you can already use bash in Jupyter, but the integration is necessarily kind of coarse. I don't know exactly what a good integration is, but I'm looking for people interested in UIs to help me figure it out. At the very least, you can use Oil's parser to provide completions: http://www.oilshell.org/blog/tags.html?tag=interactive-shell#interactive-shell http://www.oilshell.org/blog/tags.html?tag=interactive-shell... (which is a separate integration point that running commands) Also, I mentioned in the blog post that there's some interest from maintainers in combining the fish shell and Oil. However someone has to do that work, and there's a lot of it! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback ---- Also, I use shell scripts as a productivity tool like a Jupyter notebook: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#semi-automation-with-runsh-scripts http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... I think we need to come up with a clever name for it -- some people call it "Taskfile", "go script", and I call it "run.sh". The basic idea is that I don't remember how to do things with computers, because there are too many things to do. I just remember WHERE I WROTE IT DOWN in a shell script. Some people write their notes in text files; I write them in executable shell scripts.
  - viraptor 6y ago
    > I don't know exactly what a good integration is, but I'm looking for people interested in UIs to help me figure it out. Ignoring the language bit, in write impressed with both AutoIT and PowerShell GUI integration. They really allow you to create script-like utilities.
solarkraft 6y ago
OSH is probably a good incremental improvement over bash, but I also enjoy using the significantly more tradition-breaking Powershell with its object oriented nature. It feels a lot more like programming and actually gives you useful suggestions right inside the terminal! On Linux the auto completion behavior is luckily less obnoxious than on Windows and it doesn't have the multi second startup delay either.
- klodolph 6y ago
  Agreed... PowerShell is actually interesting and it feels a bit more like you're inside a shell-like REPL rather than just a fancy shell.
  - 7thaccount 6y ago
    It's interesting until you run up against speed limitations both left and right. I still use it for some work and enjoy it, but it is by far the slowest technology I've used.
    - MaikuMori 6y ago
      Believe it or not you have to disable the progress bar to speed some things up. Downloads alone are at least 10x faster if you disable progress bar.
      - 7thaccount 6y ago
        File read and writes are also glacially slow by all the common cmdlets. Even if you use .NET in your code and make it far more verbose it CRAWLS. Someone on here previously showed a trick to get some speedups, but it is still slow.
- chubot 6y ago
  I should turn this into a FAQ, but PowerShell is natural on Windows, where the OS provides objects (either via the .NET VM, or COM and .DLLs, etc.) A Unix shell like bash or Oil is natural on Unix, where the OS uses text files. And in distributed systems where data is JSON, YAML, XML, protobuf, msgpack, etc. not objects. So basically shell is a "situated" language, and the easy of accomplishing any given task depends a lot on the environment. Windows is more tightly coupled and trying to provide something nice for you. Unix is messier but doesn't limit you, and it's what basically all big systems are made of these days. A major strength of shell is to glue things together that nobody thought should be glued together. --- Good example here from Paul Bucheit: http://www.oilshell.org/blog/2020/01/simplest-explanation.html#productive-programmers-secretly-use-shell http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... http://paulbuchheit.blogspot.com/2009/01/communicating-with-code.html http://paulbuchheit.blogspot.com/2009/01/communicating-with-... However, we needed a way for Gmail to make money, and Sanjeev Singh kept talking about using relevant ads, even though it was obviously a "bad idea". I remained skeptical, but thought that it might be a fun experiment, so I connected to that ads database (I assure you, random engineers can no longer do this!), copied out all of the ads+keywords, and did a little bit of sorting and filtering with some UNIX SHELL COMMANDS I then hacked up the "adult content" classifier that Matt Cutts and I had written for safe-search, linked that into the Gmail prototype, and then loaded the ads data into the classifier. My change to the classifier (which completely broke its original functionality, but this was a separate code branch) changed it from classifying pages as "adult", to classifying them according to which ad was most relevant. The resulting ad was then displayed in a little box on our Gmail prototype ui. The code was rather ugly and hackish, but more importantly, it only took a few hours to write! ---- Also, use whatever's best for you, but Unix people tend to hate PowerShell: https://medium.com/@octskyward/the-woes-of-powershell-8737e5346b1 https://medium.com/@octskyward/the-woes-of-powershell-8737e5... PowerShell feels like it was built by people who had heard about command lines a long time ago and tried to recreate one based on stories passed down through generations of their ancestors The fact that they cargo-culted operators like -eq, -le, -lt, while not maintaining compatibility is just silly to me. In Oil it's "x == y" for string equality.
l0b0 6y ago
There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char, it's QSN-encoded like 'multi-line \n name with NUL \0 byte' This is a bad idea. There is a solution which already works. Just terminate every filename with NUL and you're done. Trying to make something "sorta kinda human readable" is a mistake when dealing with what is effectively arbitrary byte sequences which can contain anything except for NUL. Consistency makes the consuming code much simpler. That said, QSN is a great idea for human input. [1] https://www.oilshell.org/blog/2020/10/osh-features.html#safe-processing-of-user-supplied-data-like-filenames https://www.oilshell.org/blog/2020/10/osh-features.html#safe...
- e12e 6y ago
  I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.
  - l0b0 6y ago
    For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.
    - chubot 6y ago
      I actually considered that, but it doesn't work. https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON... Single quoted strings in shell can't represent arbitrary strings either. And the %q format is actually different from the format ${x@Q} emits. QSN is a well-specified format.
- chubot 6y ago
  The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). [1] https://www.oilshell.org/blog/2017/09/29.html https://www.oilshell.org/blog/2017/09/29.html
oftenwrong 6y ago
The Simple Word Evaluation on its own is a significant improvement. bash and posix sh will continue to have their place where portability is necessary, but this seems like a better day-to-day tool.
ilaksh 6y ago
Just in case people are looking for something really new in a shell: besides Oil, check out fish shell. The killer feature for me is its autocompletion -- its basically psychic.
- clashmeifyoucan 6y ago
  can't recommend fish enough, the autocompletions and the intelligent history scrolling absolutely made the difference to me. probably an unpopular opinion but I really like fish scripting more than bash, the syntax is way more intuitive and I don't find myself missing bash at all.
- chubot 6y ago
  Yes, fish and Oil are very complementary (see the note about fish in the blog post)
- paleogizmo 6y ago
  Fish is amazing as an interactive shell. I've had it as my default for a couple years now. Oil isn't the only game in town for object shells, there is also NuShell and Elvish which follow the fish model of abandoning POSIX compatibility for ease of use. I think OP had mentioned it elsewhere on this thread, but I've found object shells mostly useful for scripting that involves a lot of calling and processing the output of programs that produce structured data. I mostly use PowerShell for this, though Nushell/Elvish/Oil seem to be actually usable now. I'm pretty excited that the Unix community seems to finally be moving on from defending the old unstructured text model of program interaction to embracing the benefits of object shells. Whatever you think of its implementation, PowerShell is a great idea.
quotemstr 6y ago
These things might justify making a non-Bourne shell, but they don't justify making a greenfield shell: the fish shell already has all these features, and since we all benefit by reducing shell fragmentation, I think the author ought to work on hacking on fish more before making yet another shell.
- oblio 6y ago
  Fish is incompatible with bash...
  - quotemstr 6y ago
    Every shell that changes how quoting works is incompatible with bash
    - oblio 6y ago
      There are different degrees of breakage and you should read his blog. One of the modes is compatible with 99% of the bash scripts used out in the wild.
      - quotemstr 6y ago
        Yes, you're right. A closer examination of Oil yields something much more interesting than I imagined.
- chubot 6y ago
  fish and Oil are very complementary (see the note about fish in the blog post)
- mypalmike 6y ago
  Shall I declare what you "ought to work on" in your free time too?
pierrebai 6y ago
I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.
- kbenson 6y ago
  I agree. Generally I find the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs. I use a shell for shell stuff, I use a scripting language for most other stuff, and I might use a systems level compiled language (or a scripting language that calls into a compiled library) for more performance specific needs. If you're already within a specific area and only need to venture into the other for a minimal aspect of the current project, it can be useful to stick with what you're in, but you quickly reach the point where it's better to choose a better tool because of diminishing returns from using a tool for something it's not good for. Maybe the hammer in the hand is fine for prying the single board off fence or wall, but if you're going to be doing it to ten or twenty boards, maybe walking over to the shed to get the crowbar will save a lot of time and effort in the end.
  - chubot 6y ago
    the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs. Exactly, that is the point of shell. Shell scripts are meant to invoke Python or Ruby programs. sibling comment: https://news.ycombinator.com/item?id=24875932 https://news.ycombinator.com/item?id=24875932 earlier comment: https://news.ycombinator.com/item?id=24083764 https://news.ycombinator.com/item?id=24083764
- R0b0t1 6y ago
  There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was literally made for this type of workload.
IggleSniggle 6y ago
Doesn't TCL (and tclsh) handle these top four features quite elegantly? I understand why people shudder at TCL, but they are mixing up the elegance of the language with the dysfunction of the situation it was pulled in to address. Yes, when you start gluing random stuff together it gets ugly. This is true in real-life too. That's the situation. TCL, from the "commands and strings" handles this quite elegantly with a kind of Haskell-like power. And you can just use it as your shell, because it is designed to run commands! And if you are on macOS or Linux, there's a good chance you even already have it installed. So go give it a try. I think perl comes at this from "the other end," making a general-use language into a shell-convenient package, but I don't have enough experience on it to comment. Nonetheless, I suspect the horrors of perl are somewhat similar in nature: Too many blackbox systems that must be glued together.
- qznc 6y ago
  TCL has a very simple model of a shell "REPL": interpret string as expression and evaluate. In contrast, a Unix shell has job control, stderr, pipes, etc.
  - IggleSniggle 6y ago
    I’m struggling to understand how you don’t get these features with tcl / tclsh. You get job control, familiar access to your file system and the usual conventions around command execution of binaries in your PATH (like ps, kill, etc), you have stderr, you have pipes... ...but you also have a language that is immediately polyglot because of the way it uses strings. Each string reference is analogous to a file-descriptor; a function pointer that can be redirected and executed in whatever context, while also feeling like native context scopes in any languages you might want to use to interpret/compile them due to their use of the {} delimiter.
    - indigo945 6y ago
      But you can't connect programs together via pipes from within TCL, except by doing a terrible hand-rolled implementation that does half the things that bash gives you for free. And arguably, the ability to connect programs together like that is the most powerful feature of any modern shell.
asiachick 6y ago
How about Powershell https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell I can't call myself a fan of powershell but if everyone switched I'd get used to it. sh and bash both seem like they should die in a fire. They are full of foot guns that end up costing millions in breaches and lost data. The space think even bit Apple back in the day, they had some OS upgrade script that ended up deleting your entire hard drive if there was a space in the volume name.
- Kuinox 6y ago
  Powershell is a bit better, but got it's own disadvantage: Despite the consistent naming, I can't remember correctly the modules names. Running Remove-NetFireWallRule without arguments delete all the firewalls rules despite a flag "-All" not being set. The learning curve is too hard, or it's too complex to learn on the job: It takes me at least half an hour to write semi-complex commands despite the number of hours I struggled trying to write commands with. And it still suck as a scripting language.
  - sedatk 6y ago
    > And it still suck as a scripting language. Why do you think that? I find PowerShell quite neat and more consistent than what bash and alike has to offer.
    - Kuinox 6y ago
      It does "resume on error" by default. It got script languages issues: The last line of your huge script may be syntaxically invalid. Here you start thinking "why I didn't wrote it in my favorite statically compiled language" Yes, it's still more consistent than what bash and alike has to offer, but it's still behind if you compare it to any language that you can write to a file. The only advantage it got against them, it's the huge API on windows. Sadly I wanted to use Powershell functions from C#, tought it would be easy because Powershell use .NET Core, but it's far from easy.
- boston_sre87 6y ago
yk42bb 6y ago
I still have high hopes for Lush, the Lisp Universal SHell[0]. [0]: http://lush.sourceforge.net/ http://lush.sourceforge.net/
mrtweetyhack 6y ago
a shell doesn't have to be a programming language as well. I just need it to get around the fs and execute a program
mariuolo 6y ago
Is this shell actually written in python2 in the year 2020? (see: https://github.com/oilshell/oil/tree/master/Python-2.7.13 https://github.com/oilshell/oil/tree/master/Python-2.7.13 )
- kzrdude 6y ago
  Some fun idiosyncrasy going on > It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python. Love it ;)

242 points by diegocg 6y ago

autoexec 6y ago

New shell or not I hope we see updates to existing shells to do better error handling too

chubot 6y ago
(author here) Yes, I actually proposed that in the comments [1]. Some should add command_sub_errexit to bash: echo $(date %x) # can you tell what's wrong here? echo 'script should fail before this' local d=$(date %x) echo 'script should fail before this' And process_sub_fail: diff <(sort left.txt) <(sort /oops/error) echo 'script should fail before this' As mentioned in the comment, bash doesn't even wait() on process subs! The 'run' builtin to fix the "if myfunc" problem should also be adopted by other shells. POSIX basically calcified broken language semantics and needs to be fixed. I also document stuff like shopt -s simple_word_eval for other shell implementers: https://www.oilshell.org/release/0.8.3/doc/simple-word-eval.html https://www.oilshell.org/release/0.8.3/doc/simple-word-eval.... ----- BTW Oil's code is very short (see comment), and I'm looking for help :) Let me know if you can't get a bin/osh working in 1 to 5 minutes. (If you have a Debian/Ubuntu-ish machine it should take about 1 minute.) This is a pure Python program that you can quickly modify/prototype, and then for the release it's translated into C++ for a 30-50x speedup [2]. https://github.com/oilshell/oil/wiki/Contributing https://github.com/oilshell/oil/wiki/Contributing [1] https://lobste.rs/s/qfiki1/four_features_justify_new_unix_shell#c_ncyhoh https://lobste.rs/s/qfiki1/four_features_justify_new_unix_sh... [2] http://www.oilshell.org/blog/2020/01/parser-benchmarks.html http://www.oilshell.org/blog/2020/01/parser-benchmarks.html

breatheoften 6y ago

Are there any transpile to bash languages? I like the idea of a language that allows for a better experience in writing shell scripts -- and produces shell scripts with fewer bugs on the output side. But i wonder if a transpile approach wouldn't be better in the long run

kccqzy 6y ago
What do you think produces the traditional ./configure scripts that you run to configure an open source project prior to building it? Did some human write these ./configure scripts manually?
- codetrotter 6y ago
  Autoconf isn’t flexible enough for general purpose scripting though, is it? So not really relevant I think.
- viraptor 6y ago
  M4 doesn't really satisfy: > better experience in writing It will happily transform a small mistake into garbage. Then you have to debug both the transformed result and the original script to match up what you intended.
chubot 6y ago
(author here) Yeah I've been asked that before, and there are some things you can fix with a transpiler, but somethings you can't. That's why the tagline to Oil is now "our upgrade path from bash to a better language AND RUNTIME". The shell runtime needs to be fixed too, e.g. the error handling mentioned in the blog post and that I follow up on here: https://news.ycombinator.com/item?id=24872986 https://news.ycombinator.com/item?id=24872986 You might be able to compile that to some really long bash, but I don't think it works in general, because editing deployed scripts to fix stuff is useful (yes in production, but also on your own machine. Think of all the shell that people put on it, like virtualenv, nix-shell, rustup, etc.) You don't want to end up with the JavaScript problem: you have a dynamic language, but ALSO A BUILD PROCESS, which is basically the worst of both worlds.
- breatheoften 6y ago
  So is the normal usage path to start using osh for interactive use and then use osh in a bash compatible mode to write scripts that are less buggy but can still be run on bash? Or is it more common to write scripts that tend to then require the osh runtime exist in whatever context they are deployed to?

xilni 6y ago

I love the direction this wants to go in. So many of these are reasons why we've banned new shell scripts on our projects and instead make python scripts.

ritchiea 6y ago
Are there ever good reasons to choose a shell script over a Python/Ruby/etc script?
- chubot 6y ago
  Yes, a couple answers here: http://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-discourage-people-from-writing-shell-scripts http://www.oilshell.org/blog/2018/01/28.html#shouldnt-we-dis...
  - neolog 6y ago
    > It won't work. It would be like trying to convince people who are paid to write PHP not to write PHP. Many people have wasted breath on that, but important sites like Wikipedia are still written in hundreds of thousands of lines of PHP. If they're gonna keep writing bash, why would a new shell language help? > Even if a new line of shell never gets written, there will still be a huge installed base of shell scripts that you may need to understand (e.g. when they don't work). If the old scripts are in bash, why would a new shell language help? > Shell is still the best tool for many jobs. Most new "cloud" projects rely on Linux system images, in VMs or containers, and shell is the ideal language for creating such images. What's good about shell for this task?
    - chubot 6y ago
      Because you can run your bash scripts with Oil. The tagline is: It's our upgrade path from bash to a better language and runtime. [1] It's basically the same as JS -> TypeScript, or PHP -> Hack. It's a saner language (and runtime) that runs existing code. ----- Shell is good for creating Unix systems because of the tools it provides. Ones that deal with the file system and heterogeneous processes (i.e. stuff you didn't write in different languages). It's hard to explain, but if you work in that area, you'll very quickly see it. You could also do something like Linux From Scratch [2] and it will be very clear why shell is used. [1] http://www.oilshell.org/ http://www.oilshell.org/ [2] http://www.linuxfromscratch.org/ http://www.linuxfromscratch.org/

3np 6y ago

Two ergonomic issues in POSIX/bash not mentioned that I’d like to see a new shell improve on: - splitting pipes to separate flows (e.g one flow for stderr, one for stdout). Think of it as a graph. - native parallelization; I have to look in man every other time I use parallel.

RMPR 6y ago
> native parallelization; I have to look in man every other time I use parallel. Can relate, the syntax is very strange, at least to me. And I wonder where it came from.
Splognosticus 6y ago
> - splitting pipes to separate flows (e.g one flow for stderr, one for stdout). Think of it as a graph. Could you elaborate a little on what you mean by this? bash already has the ability to redirect STDERR and STDOUT independently, and when you introduce named pipes into the mix things can get really fancy (probably too fancy, this is about where your script stops being a script and starts being a program written in perhaps the most inconvenient language available.)
- e12e 6y ago
  I imagine an easy/intuitive way to set up FIFOs for stdout/stderr with redirection, maybe? A way to say: for this following section, I want stderr to go trough "|sort -n|logger -t DEBUG -f -" and standard out to go through (... Other pipeline)?
  - folmar 6y ago
    command 2> >(sort -n|logger -t DEBUG -f -) | other pipeline
titanomachy 6y ago
The first is interesting. I have a hard time imagining what that syntax would look like. You'd need a way to refer to your graph nodes, so I don't think you would get much better than what named pipes (aka FIFOs) already provide. Totally agree about GNU parallel. The syntax is not badly-designed IMO, but I don't use it very frequently and it's different from everything else so I often get it wrong. Using it as a drop-in replacement for xargs is easy enough though.

abnry 6y ago

I am very sympathetic to this. I like the concept of a shell (there's a place for light scripting that doesn't involve python or perl or ruby), and appreciate what bash can do, but after encountering enough of the "gotchas" and just ugly behavior, I'm not really motivated to master it. The biggest reason to learn it seems to be the fact that everybody uses it and has done so for years. We need something akin to Jupyter notebooks for unix.

chubot 6y ago
Yes, I would like for that to happen: Shell as an engine for TUI or GUI https://github.com/oilshell/oil/issues/738 https://github.com/oilshell/oil/issues/738 C++ code should be embeddable in another program https://github.com/oilshell/oil/issues/822 https://github.com/oilshell/oil/issues/822 I think you can already use bash in Jupyter, but the integration is necessarily kind of coarse. I don't know exactly what a good integration is, but I'm looking for people interested in UIs to help me figure it out. At the very least, you can use Oil's parser to provide completions: http://www.oilshell.org/blog/tags.html?tag=interactive-shell#interactive-shell http://www.oilshell.org/blog/tags.html?tag=interactive-shell... (which is a separate integration point that running commands) Also, I mentioned in the blog post that there's some interest from maintainers in combining the fish shell and Oil. However someone has to do that work, and there's a lot of it! https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback https://github.com/oilshell/oil/wiki/Where-To-Send-Feedback ---- Also, I use shell scripts as a productivity tool like a Jupyter notebook: http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#semi-automation-with-runsh-scripts http://www.oilshell.org/blog/2020/02/good-parts-sketch.html#... I think we need to come up with a clever name for it -- some people call it "Taskfile", "go script", and I call it "run.sh". The basic idea is that I don't remember how to do things with computers, because there are too many things to do. I just remember WHERE I WROTE IT DOWN in a shell script. Some people write their notes in text files; I write them in executable shell scripts.
- viraptor 6y ago
  > I don't know exactly what a good integration is, but I'm looking for people interested in UIs to help me figure it out. Ignoring the language bit, in write impressed with both AutoIT and PowerShell GUI integration. They really allow you to create script-like utilities.

solarkraft 6y ago

OSH is probably a good incremental improvement over bash, but I also enjoy using the significantly more tradition-breaking Powershell with its object oriented nature. It feels a lot more like programming and actually gives you useful suggestions right inside the terminal! On Linux the auto completion behavior is luckily less obnoxious than on Windows and it doesn't have the multi second startup delay either.

klodolph 6y ago
Agreed... PowerShell is actually interesting and it feels a bit more like you're inside a shell-like REPL rather than just a fancy shell.
- 7thaccount 6y ago
  It's interesting until you run up against speed limitations both left and right. I still use it for some work and enjoy it, but it is by far the slowest technology I've used.
  - MaikuMori 6y ago
    Believe it or not you have to disable the progress bar to speed some things up. Downloads alone are at least 10x faster if you disable progress bar.
    - 7thaccount 6y ago
      File read and writes are also glacially slow by all the common cmdlets. Even if you use .NET in your code and make it far more verbose it CRAWLS. Someone on here previously showed a trick to get some speedups, but it is still slow.
chubot 6y ago
I should turn this into a FAQ, but PowerShell is natural on Windows, where the OS provides objects (either via the .NET VM, or COM and .DLLs, etc.) A Unix shell like bash or Oil is natural on Unix, where the OS uses text files. And in distributed systems where data is JSON, YAML, XML, protobuf, msgpack, etc. not objects. So basically shell is a "situated" language, and the easy of accomplishing any given task depends a lot on the environment. Windows is more tightly coupled and trying to provide something nice for you. Unix is messier but doesn't limit you, and it's what basically all big systems are made of these days. A major strength of shell is to glue things together that nobody thought should be glued together. --- Good example here from Paul Bucheit: http://www.oilshell.org/blog/2020/01/simplest-explanation.html#productive-programmers-secretly-use-shell http://www.oilshell.org/blog/2020/01/simplest-explanation.ht... http://paulbuchheit.blogspot.com/2009/01/communicating-with-code.html http://paulbuchheit.blogspot.com/2009/01/communicating-with-... However, we needed a way for Gmail to make money, and Sanjeev Singh kept talking about using relevant ads, even though it was obviously a "bad idea". I remained skeptical, but thought that it might be a fun experiment, so I connected to that ads database (I assure you, random engineers can no longer do this!), copied out all of the ads+keywords, and did a little bit of sorting and filtering with some UNIX SHELL COMMANDS I then hacked up the "adult content" classifier that Matt Cutts and I had written for safe-search, linked that into the Gmail prototype, and then loaded the ads data into the classifier. My change to the classifier (which completely broke its original functionality, but this was a separate code branch) changed it from classifying pages as "adult", to classifying them according to which ad was most relevant. The resulting ad was then displayed in a little box on our Gmail prototype ui. The code was rather ugly and hackish, but more importantly, it only took a few hours to write! ---- Also, use whatever's best for you, but Unix people tend to hate PowerShell: https://medium.com/@octskyward/the-woes-of-powershell-8737e5346b1 https://medium.com/@octskyward/the-woes-of-powershell-8737e5... PowerShell feels like it was built by people who had heard about command lines a long time ago and tried to recreate one based on stories passed down through generations of their ancestors The fact that they cargo-culted operators like -eq, -le, -lt, while not maintaining compatibility is just silly to me. In Oil it's "x == y" for string equality.

l0b0 6y ago

There are some good ideas in here, but `--qsn` is described[1] like this: > Print filenames ONE PER LINE. If a name contains a newline or other special char, it's QSN-encoded like 'multi-line \n name with NUL \0 byte' This is a bad idea. There is a solution which already works. Just terminate every filename with NUL and you're done. Trying to make something "sorta kinda human readable" is a mistake when dealing with what is effectively arbitrary byte sequences which can contain anything except for NUL. Consistency makes the consuming code much simpler. That said, QSN is a great idea for human input. [1] https://www.oilshell.org/blog/2020/10/osh-features.html#safe-processing-of-user-supplied-data-like-filenames https://www.oilshell.org/blog/2020/10/osh-features.html#safe...

e12e 6y ago
I think having both is a good idea. Sometimes maybe you get your list of path names from a Here-document, for example.
- l0b0 6y ago
  For sure we need both human- and machine-readable stream formats, but I don't really see how QSN has any advantage in that space over `printf '%q\n'` and NUL-terminated strings, respectively.
  - chubot 6y ago
    I actually considered that, but it doesn't work. https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON... Single quoted strings in shell can't represent arbitrary strings either. And the %q format is actually different from the format ${x@Q} emits. QSN is a well-specified format.
chubot 6y ago
The paragraph right below that mentions that Oil has "read -0", which consumes the find -print0 input. I also link to my Git Log in HTML post [1] from 3 years ago, which is ENTIRELY about the NUL byte solution :) ----- Shell scripts can use both formats, but the advantage to QSN is that it preserves the line-based nature of shell. Say I want to use wc -l, awk, or grep. Then the QSN-lines format is better than the NUL format. Also, you can transmit a series of say SHA256 checksums in binary format with QSN-lines, or some other binary format. Even entire JPG files, audio clips, wasm files, whatever. It's "8-bit clean" (and so are Oil strings). [1] https://www.oilshell.org/blog/2017/09/29.html https://www.oilshell.org/blog/2017/09/29.html

oftenwrong 6y ago

The Simple Word Evaluation on its own is a significant improvement. bash and posix sh will continue to have their place where portability is necessary, but this seems like a better day-to-day tool.

ilaksh 6y ago

Just in case people are looking for something really new in a shell: besides Oil, check out fish shell. The killer feature for me is its autocompletion -- its basically psychic.

clashmeifyoucan 6y ago
can't recommend fish enough, the autocompletions and the intelligent history scrolling absolutely made the difference to me. probably an unpopular opinion but I really like fish scripting more than bash, the syntax is way more intuitive and I don't find myself missing bash at all.
chubot 6y ago
Yes, fish and Oil are very complementary (see the note about fish in the blog post)
paleogizmo 6y ago
Fish is amazing as an interactive shell. I've had it as my default for a couple years now. Oil isn't the only game in town for object shells, there is also NuShell and Elvish which follow the fish model of abandoning POSIX compatibility for ease of use. I think OP had mentioned it elsewhere on this thread, but I've found object shells mostly useful for scripting that involves a lot of calling and processing the output of programs that produce structured data. I mostly use PowerShell for this, though Nushell/Elvish/Oil seem to be actually usable now. I'm pretty excited that the Unix community seems to finally be moving on from defending the old unstructured text model of program interaction to embracing the benefits of object shells. Whatever you think of its implementation, PowerShell is a great idea.

quotemstr 6y ago

These things might justify making a non-Bourne shell, but they don't justify making a greenfield shell: the fish shell already has all these features, and since we all benefit by reducing shell fragmentation, I think the author ought to work on hacking on fish more before making yet another shell.

oblio 6y ago
Fish is incompatible with bash...
- quotemstr 6y ago
  Every shell that changes how quoting works is incompatible with bash
  - oblio 6y ago
    There are different degrees of breakage and you should read his blog. One of the modes is compatible with 99% of the bash scripts used out in the wild.
    - quotemstr 6y ago
      Yes, you're right. A closer examination of Oil yields something much more interesting than I imagined.
chubot 6y ago
fish and Oil are very complementary (see the note about fish in the blog post)
mypalmike 6y ago
Shall I declare what you "ought to work on" in your free time too?

pierrebai 6y ago

I'm sorry to say to for me, shell scripting has lost. All my scripts are written in Python, which a thin shell script wrapper to launch it. Look at Oil. It tries hard to be bash compatible with optional buy-in to extension. Just use a superior language and give up on shell scripts, that's my advice.

kbenson 6y ago
I agree. Generally I find the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs. I use a shell for shell stuff, I use a scripting language for most other stuff, and I might use a systems level compiled language (or a scripting language that calls into a compiled library) for more performance specific needs. If you're already within a specific area and only need to venture into the other for a minimal aspect of the current project, it can be useful to stick with what you're in, but you quickly reach the point where it's better to choose a better tool because of diminishing returns from using a tool for something it's not good for. Maybe the hammer in the hand is fine for prying the single board off fence or wall, but if you're going to be doing it to ten or twenty boards, maybe walking over to the shed to get the crowbar will save a lot of time and effort in the end.
- chubot 6y ago
  the approach to have a few tools that work well in their specific niches to be superior to having one tool for all jobs. Exactly, that is the point of shell. Shell scripts are meant to invoke Python or Ruby programs. sibling comment: https://news.ycombinator.com/item?id=24875932 https://news.ycombinator.com/item?id=24875932 earlier comment: https://news.ycombinator.com/item?id=24083764 https://news.ycombinator.com/item?id=24083764
R0b0t1 6y ago
There is a better option and its name is Perl. Perl mixes the ins of shell scripts like easy argument and output passing with complex and easy to use control structures. You could use Python, but having been forced to use Perl extensively Perl is the superior choice for a complex shellscript-like workload. There's less boilerplate in Perl. There's a couple Python projects that come close like Fabric, but Perl was literally made for this type of workload.

IggleSniggle 6y ago

Doesn't TCL (and tclsh) handle these top four features quite elegantly? I understand why people shudder at TCL, but they are mixing up the elegance of the language with the dysfunction of the situation it was pulled in to address. Yes, when you start gluing random stuff together it gets ugly. This is true in real-life too. That's the situation. TCL, from the "commands and strings" handles this quite elegantly with a kind of Haskell-like power. And you can just use it as your shell, because it is designed to run commands! And if you are on macOS or Linux, there's a good chance you even already have it installed. So go give it a try. I think perl comes at this from "the other end," making a general-use language into a shell-convenient package, but I don't have enough experience on it to comment. Nonetheless, I suspect the horrors of perl are somewhat similar in nature: Too many blackbox systems that must be glued together.

qznc 6y ago
TCL has a very simple model of a shell "REPL": interpret string as expression and evaluate. In contrast, a Unix shell has job control, stderr, pipes, etc.
- IggleSniggle 6y ago
  I’m struggling to understand how you don’t get these features with tcl / tclsh. You get job control, familiar access to your file system and the usual conventions around command execution of binaries in your PATH (like ps, kill, etc), you have stderr, you have pipes... ...but you also have a language that is immediately polyglot because of the way it uses strings. Each string reference is analogous to a file-descriptor; a function pointer that can be redirected and executed in whatever context, while also feeling like native context scopes in any languages you might want to use to interpret/compile them due to their use of the {} delimiter.
  - indigo945 6y ago
    But you can't connect programs together via pipes from within TCL, except by doing a terrible hand-rolled implementation that does half the things that bash gives you for free. And arguably, the ability to connect programs together like that is the most powerful feature of any modern shell.

asiachick 6y ago

How about Powershell https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell I can't call myself a fan of powershell but if everyone switched I'd get used to it. sh and bash both seem like they should die in a fire. They are full of foot guns that end up costing millions in breaches and lost data. The space think even bit Apple back in the day, they had some OS upgrade script that ended up deleting your entire hard drive if there was a space in the volume name.

Kuinox 6y ago
Powershell is a bit better, but got it's own disadvantage: Despite the consistent naming, I can't remember correctly the modules names. Running Remove-NetFireWallRule without arguments delete all the firewalls rules despite a flag "-All" not being set. The learning curve is too hard, or it's too complex to learn on the job: It takes me at least half an hour to write semi-complex commands despite the number of hours I struggled trying to write commands with. And it still suck as a scripting language.
- sedatk 6y ago
  > And it still suck as a scripting language. Why do you think that? I find PowerShell quite neat and more consistent than what bash and alike has to offer.
  - Kuinox 6y ago
    It does "resume on error" by default. It got script languages issues: The last line of your huge script may be syntaxically invalid. Here you start thinking "why I didn't wrote it in my favorite statically compiled language" Yes, it's still more consistent than what bash and alike has to offer, but it's still behind if you compare it to any language that you can write to a file. The only advantage it got against them, it's the huge API on windows. Sadly I wanted to use Powershell functions from C#, tought it would be easy because Powershell use .NET Core, but it's far from easy.
boston_sre87 6y ago

yk42bb 6y ago

I still have high hopes for Lush, the Lisp Universal SHell[0]. [0]: http://lush.sourceforge.net/ http://lush.sourceforge.net/

mrtweetyhack 6y ago

a shell doesn't have to be a programming language as well. I just need it to get around the fs and execute a program

mariuolo 6y ago

Is this shell actually written in python2 in the year 2020? (see: https://github.com/oilshell/oil/tree/master/Python-2.7.13 https://github.com/oilshell/oil/tree/master/Python-2.7.13 )

kzrdude 6y ago
Some fun idiosyncrasy going on > It's written in Python, so the code is short and easy to change. But we automatically translate it to C++ with custom tools, to make it fast and small. The deployed executable doesn't depend on Python. Love it ;)