14 ms·

Unix Shell Programming: The Next 50 Years [pdf]

138 points by signa11 5y ago

flakiness 5y ago
It's a bit clickbity. In the introduction, the paper mostly talks about the ergonomics / maintainability (that resonates the audiencel like me), but the proposed solution, "jash", addresses the performance problem (who cares???). That's said, the brief survey in the first half of the paper is moderately interesting. It's not a real survey but an advertisement of the authors' work. But they've done something worth a quick glance like formalizing shell's semantics [1], which looks interesting if no useful. [1] http://shell.cs.pomona.edu/ http://shell.cs.pomona.edu/
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
dathinab 5y ago
I think what would be more use-full would be a new "shell" which is base on a modern context, incorporating lessens learned from years of cmd-line UX design (ok let's be honest many programs didn't learn anything wrt. UX). But adding jet another POSIX shell which maybe does some parts better but in the end has still not a grate UX because it's a POSIX shell seems kinda pointless IMHO.
- yoloClin 5y ago
  This. I want a shell that will contextually spit out plaintext in interactive mode, then a JSON object when scripted or piped. Pretty sure this is what Powershell does, but the UI just feels so damn unnatural.
  - zozbot234 5y ago
    Powershell does not use JSON. It's based on .NET objects.
  - kreetx 5y ago
    Perhaps IDEs and shells could converge?
  - hnlmorg 5y ago
    Murex does this. eg Take a plain text table, convert it into an SQL table and run an SQL query: » ps aux | select USER, count(*) GROUP BY USER USER count(*) _installcoordinationd 1 _locationd 4 _mdnsresponder 1 _netbios 1 _networkd 1 _nsurlsessiond 2 _reportmemoryexception 1 _softwareupdate 3 _spotlight 5 _timed 1 _usbmuxd 1 _windowserver 2 lmorg 349 root 134 The builtins usually print human readable output when STDOUT is a TTY, or JSON (or JSONlines) when the TTY is a pipe. » fid-list: FID Parent Scope State Run Mode BG Out Pipe Err Pipe Command Parameters 590 0 0 Executing Normal no out err fid-list (subject to change) » fid-list: | cat ["FID","Parent","Scope","State","RunMode","BG","OutPipe","ErrPipe","Command","Parameters"] [615,0,0,"Executing","Normal",false,"out","err",{},"(subject to change) "] [616,0,0,"Executing","Normal",false,"out","err",{},"cat"] and you can reformat to other data types, eg » fid-list: | format csv FID,Parent,Scope,State,RunMode,BG,OutPipe,ErrPipe,Command,Parameters 703,0,0,Executing,Normal,false,out,err,map[],(subject to change) 704,0,0,Executing,Normal,false,out,err,map[],csv and query data within those data structures using tools that are aware of that structural format. Eg Github's API returns a JSON object and we can filter through it to return just the issue ID's and titles: » open https://api.github.com/repos/lmorg/murex/issues | foreach issue { printf "%2s: %s\n" $issue[number] $issue[title] } 348: Potential regression bug in `fg` 347: Version 2.2 Release 342: Install on Fedora 34 fails (issue with `go get` + `bzr`) 340: `append` and `prepend` should `ReadArrayWithType` 318: Establish a testing framework that can work against the compiled executable, sending keystrokes to it 316: struct elements should alter data type to a primitive 311: No autocompletions for `openagent` currently exist 310: Supprt for code blocks in not: `! { code }` 308: `tabulate` leaks a zero length string entry when used against `rsync --help` Source: https://github.com/lmorg/murex https://github.com/lmorg/murex Website: https://murex.rocks https://murex.rocks
    - AnIdiotOnTheNet 5y ago
      As much as this is an improvement in many ways, using JSON for this feels like it doubles down on part of the problem with the current standard. Every tool is rendering JSON only for the next tool in the pipeline to parse it back out.
  - tmerse 5y ago
    I'd like most programs to implement a JSON interface, given right flags. Or maybe the existence of some env var like $DEFAULT_OUTPUT=JSON I like this approach: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc > CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts
- hnlmorg 5y ago
  This is the approach murex tries to take. It breaks POSIX compliance without remorse but retains compatibility wherever practical. This means it can have some of the flexibilities of Powershell (typed pipelines), the flexibilities of IDEs (events, better auto-completions, etc) but still works with all of your everyday command line tools. My ultra long term aim is to integrate the shell into a media rich terminal emulator so using the command line will have the speed and precision of a TUI, the power of an IDE but the rich content of a GUI. I'm a loooong way off achieving that but the shell is already usable and has been my primary shell for ~4 years now. And I welcome any and all feedback on the current and any future builds. At some point I'll publish a paper with my ideas. https://github.com/lmorg/murex https://github.com/lmorg/murex
  - cyberpunk 5y ago
    Just a FYI, murex is (also) a very widely used front of house trading platform, which costs an absolute fortune and has an army of lawyers, you may want to reconsider the name..
    - hnlmorg 5y ago
      Yeah, I get this a lot. I wasn't aware of the trading platform when I created the shell and was quite late into the development before I was made aware. The shell and the trading platform exist in different domains so as far as I'm concerned there isn't any trademark violation but if the trading platform (and their lawyers) disagree then I'm open to dialogue with them. I think HN is uniquely placed where it has a lot of trading topics as well as IT so it wasn't until my shell started trending on here that anyone had even noticed the naming conflict. But my shell had been around for a good few years before it started trending on here.
  - pxc 5y ago
    I'm surprised I hadn't heard of Murex yet! It looks pretty similar to Elvish but the type system is more foregrounded (and more complete?). Could you help me differentiate it from Elvish in terms of features and design?
- worble 5y ago
  I quite like elvish (https://elv.sh https://elv.sh) for a non-POSIX compatible shell. It's by no means perfect, and it's not 1.0 so breaking changes are pretty frequent, but I find the language structure and syntax make it worth it. The docs on the site are a little haphazard too, but once you get a feel for it, you probably won't need to refer to them too often.
js8 5y ago
Recently, I have been looking at the Factor language http://factorcode.org http://factorcode.org. I think I would like to have a shell based on that.
vbsteven 5y ago
The article recognizes that interop and backwards compatibility with existing systems is important for adoption and it does not try to impose a new clean room replacement. This is IMHO a good strategy that has proven itself in other contexts. For example Kotlin/Clojure/Scala all started out focusing on interop with the base Java/JVM technology. They provide a new layer of modern language features that is still compatible with all the existing code without forcing a rewrite. Over time these new languages get adopted, some older libraries get a modern equivalent and that helps these languages to branch out into other areas: e.g. Kotlin Native, multiplatform and ClojureScript. A similar thing can be seen with Typescript. It started as a type layer on top of JavaScript, making it easy to interop with old JS code and over time lots of pure Typescript library equivalents have popped up. Now that the Typescript ecosystem is flourishing we see technology like Deno that slowly moves Typescript away from its JS base. For a post-POSIX shell to be adopted widely I think it will need to use a similar strategy. Edit: re-reading my own comment, I might have just advocated for an Embrace, Extend, Extinguish strategy.
- pjmlp 5y ago
  That only seems to work for a short period of time, eventually the platform adopts the most interesting features of the candidates to replacement, and they fade away while the large majority of platform users keeps coding away on the language used to write the platform.
  - pxc 5y ago
    That's bad news for some new language authors, but not necessarily for programmers, right? These languages still succeed in pushing the norms language design forward, and that's pretty great for the users of programming languages.
    - pjmlp 5y ago
      True, the only issue is that dreams of taking over the world need to be refrained. The only platforms where new languages kind of took over, the move was forced by the company that owns the platform. However you are right, in every case even the system language for the platform got better.
lbayes 5y ago
This line was highly distracting for me, "To make matters worse, the shell has been mostly left for dead by both academia and industry, considering it an unsalvageable piece of junk that needs to be replaced at the first opportunity." Um, false? Many people in industry use and love using the shell. Please read, "In the beginning was the command line." The shell is by far one of the most powerful and expressive tools we have. Yes, there are many ways in which it could be improved, but JSON? That's not it.
- pjmlp 5y ago
  A REPL is more expressive, allows for graphical and structured output and regardless of UNIX culture books it precedes it by at least 10 years.
- hpoe 5y ago
  Actually that is the only time I started to find the shell limiting was working with JSON. But then I discovered jq and my issues are solved.
  - ur-whale 5y ago
    Yeah, except that jq, beyond 'jq .' is horribly unfriendly.
    - NegativeLatency 5y ago
      I had a similar complaint so I wrote this in ruby https://github.com/nburns/utilities/blob/master/rson https://github.com/nburns/utilities/blob/master/rson
  - s_dev 5y ago
    I started using jq as well -- was shocked to find the author was a former comp sci classmate of mine. It's very useful.
- hi41 5y ago
  I think you can invoke Perl in command line mode. Do people use Perl like that if they want to use Bash command line?
dale_glass 5y ago
The problem with the shell to me is that it started like a good idea, and then just stopped improving. Things that should be the bread and butter of scripting are oddly arcane. Some examples. How do you find where the script itself is? Because if the script comes with files, we want to find those when the script gets called from somewhere outside the current dir. Stack overflow suggests this as a first approach: SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" Then follows with 7 lines of arcane looking code containing a loop for a better solution. That's a ridiculous hoop to jump through for a common need. Dealing with filenames? Hope you enjoy pain. Spaces, newlines and quotes are a horrible pain to deal with properly, and regularly cause trouble. Dealing with correctly passing parameters? More of the same. Dealing with long file lists? You run into the command line length limit. Obtaining some incredibly common system parameter? Some insane combination of grep, awk and cut, probably. Personally I find it amazing that people invented this thing, then spent decades tripping on the same quoting and filename handling issues and not finding that to be a good reason to fix it. These days if it's more than 5 lines of code, or if arbitrary filenames are involved I go straight to Perl or Python, because the amount of completely preventable pain is drastically reduced.
- gnubison 5y ago
  > How do you find where the script itself is? No program can do this portably.
  - chubot 5y ago
    I think you might be mixing two things up: it's impossible for a native program to find where it is, because argv[0] can be anything. But it's possible for an interpreter to tell the script it's interpreting where it lives, i.e. argv[1]. For example, Python does this in a pretty straightforward way.
    - gnubison 5y ago
      You're right, my bad.
- 5y ago
dwpdwpdwpdwpdwp 5y ago
From Section 2.2 "The Bad" >> B1: Too arbitrary. The shell’s virtue of limitless composition (G1) is also its vice: the shell can compose arbitrary commands written in arbitrary languages. This is what I fundamentally love about the shell, and you can try to pry it from my cold, dead fingers >> B3: Too obscure. The semantics of the shell and common commands are documented in 300pp of standardese [7]. With respect to Bash, I couldn't agree more about the 'standardese' gripe. Man pages would be so, so much more useful if they prioritized showing examples of common/useful ways to use the command. I would love a 'man --examples $COMMAND' feature. Does something like that exist?
- revscat 5y ago
  Yes. tldr: https://tldr.sh/ https://tldr.sh/
  - ahub 5y ago
    Or as an alternative with no installation or dependence : `curl cht.sh tar`. Obviously, replace 'tar' by whatever command you see fit. Works on any machine online with curl available.
    - jbotz 5y ago
      That should probably be `curl cht.sh/tar`
- sn41 5y ago
  I don't recommend this ever, but I of course make this moonshine on my own computer: I usually edit the man page to add cookbook/bugfix sections. Most of them are cookbook-style recipes. Whenever I find out some obscure bug or gotcha out (spaces in file names passed through a pipe is one of my frequent issues), I go to the man page of cut/tr etc. and add it to the page. Next time I man for that, I remember what I did right a couple of years ago. Another thing I copiously modify is the "SEE ALSO" section for common commands, whenever I install new stuff from github etc. I confess that it is not a proper solution, but it works for me. Man pages have continued in the gloriously terse initial style of Ken Thompson. For mere mortals like I, I need a nicer explanation. My machine's manpages for some commands (find, grep being the most modified) tend to look like the old "HOWTO"s on linux.
ur-whale 5y ago
As much as I love shell ... there's no debugger worthy of the name. That makes it really hard sometimes.
- KirillPanov 5y ago
  With interpreted languages, "printf debugging" is really easy. It's one of their main advantages. I'd rather have "bash -x" and be able to edit the script than lose those features in order to gain gdb.
zoobab 5y ago
Did Ansible/Puppet, I can only say: "Go away with your shitty DSL!"
mroutput 5y ago
There's too many unsubstantiated statements in this paper to take it seriously.
hules 5y ago
Something that I particularly hate with shell scripting is that if you modify a shell script while it is executing, it will break the running instance: bash does not load the whole script on startup, it reads it line after line while executing it. I wonder why this stupid and dangerous behavior has never been changed.
- caymanjim 5y ago
  This is not true. I'm not sure what you're seeing, but neither bash nor zsh behave like this in Linux or MacOS.
  - effie 5y ago
    GP is correct. Bash does not read and parse whole script on start, that would be slow. It is done when needed.
- ptspts 5y ago
  Wrap the script with:if :; ...; fi; exit
chrisfinazzo 5y ago
I have said it several times on HN, but it bears repeating: As much as I find shell programming approachable and enjoyable, it really should be avoided if you can help it. Using `set -euo pipefail` and tools like Shellcheck help, but there's a limit. https://sipb.mit.edu/doc/safe-shell/ https://sipb.mit.edu/doc/safe-shell/ ^ Says the guy who forked a project an hour ago just to fix a couple errors, found a subshell and said, 'You're done, go no further' :)
- hpoe 5y ago
  15 seconds on the shell beats 5 minutes trying to debug a ruby or pythons script. That said I would agree as soon as something becomes complex or you need to do computation looking outside the shell is probably a good idea. The huerestic that I use is 1. Do I need to do math? Move out of the shell. 2. Is the logic starting to get complicated enough that I need consider creating modules? Move out of the shell. 3. Is there enough code to fill more than one screen? Out of the shell I go. 4. Is this going to be something that needs robust error handling and that will be used frequently outside of me? Out of the shell. I've found that as long as I don't hit one of those conditions hacking around in the shell is quite powerful and useful.
  - chrisfinazzo 5y ago
    For me (a Comms major/IS minor as an undergrad), most of what I'm manipulating is text and shell is just a tool to speed up the process and as such doesn't have those concerns. Still, I've found Rake and Make - as simple containers for shell rather useful and more flexible - newlines are sane, far less of this `\n` nonsense. ^ An odd complaint, but it just seems like the terminal or the environment should handle this without input from me. Now if only I could do something about Make syntax requiring tabs...
  - pjmlp 5y ago
    In what concerns UNIX shell, Python always wins for me, other than very basic scripts to automate launching applications, or a pipeline.
  - einpoklum 5y ago
vbsteven 5y ago
What if there was a standard way to describe inputs, outputs and side effects of programs? Think of it like writing/downloading type definitions for existing untyped code. With program descriptions like this shells could become smarter, generate warnings, abort before execution if types don’t match up, etc.
- toomanyducks 5y ago
  I think I like the theory of this but in practicality I'm pretty sure I'd just get irritated by it. In most usage, there is only one input and output format, being plain text. If I specify a given type further and send it to grep/awk/sed/cut, I'd need to explicitly or implicitly coerce the given output to plain text, at which point I'd either lose the original type information or require some unholy transformations to retrieve the original type. I think this misses how I use the shell: sending plain text information to and from files and executables. The whole point is that there is no type system and everything is just plain text. For something smarter that behaves more like a typed programming language, there's Perl, Python, etc.
pxc 5y ago
The researchers main concerns (making shell programs perform better, especially by better leveraging the parallel computing resources of supercomputing environments), but they do give a nod to stuff I care more about (usability and breaking out of the limits imposed by extant terminal emulator standards). I hope that this jump starts a bit more research into shell languages. Maybe the veneer of authority given by papers suggesting various improvements will help draw people to next-generation shells that take up those improvements and advertise that. If you were not aware, PowerShell has inspired a whole generation of next-generation shells built around structured pipelines, and they're very exciting. My personal favorite is Elvish, which also draws much inspiration from Fish, a lovely 'old-fashioned' (text-only pipeline) shell with a strong emphasis on interactivity and ease of use. One thing that makes Elvish special besides its ambition to be a pleasant and useful general-purpose programming language is its portability: it's typically distributed as a single static binary, and it even has native support for Windows. When I think of 'the future of the shell', I absolutely think of Elvish, namely with respect to: * better portability * high aims as a 'real' programming language * built-in static checks where possible (for now just syntax, I think, but a more complete type system is on the roadmap) * sophisticated and responsive interactive experience out of the box * structured pipelines, structured pipelines, structured pipelines To me, the performance issues the authors address are secondary to what I've enumerated above. But they're right that the fundamentals of the shell are great as well as overdue for some innovation.
- effie 5y ago
  I'm skeptical about academia being capable to help shell evolve. The paper is a good example why - they focus on justifying themselves and then on obscure academic ideas they like to pursue (POSIX formalization, highly parallel data processing) rather than usability for most users. Sometimes these align, but most of the time they don't. Your description of Elvish is much better in that respect. I'd love more capable shell with modern advancements like types and static checking. However, I don't get why structured data over pipelines is useful. You can already exchange any kind of data over pipeline as string of bytes. Sender and receiver have to be aware of the format in any case.
  - 5y ago
deleted 5y ago
[deleted]
usr1106 5y ago
> Unfortunately, this means that the behavior of a shell program cannot be known statically: a simple grep $PWD -in ~/.*shrc This is a weird claim. Every program's bahavior depends on its inputs in the general case. If you want to process static input you can do that using the shell. If your program should work on external inputs you cannot statically determine its output regardless of the language.
usr1106 5y ago
Everybody comments on shell programming, but nobody comments on the main message in their paper PaSH, POSH and JIT. I must admit I did not understand their message at the first quick reading. Formal methods? Well, I'm not whether anybody tries to prove their installation scripts (or whatever people would script) correct. Parallelization? Well, if you try to high-performance computing, is the shell really the place to start?
- effie 5y ago
  Well they start with "arguments" for why shell programming is deficient, and these arguments seem vague and not well expressed or incorrect. Most people just lose interest in further reading. And when you do read further, it's a lot of research field building with buzzwords that nobody who uses shell cares about.
einpoklum 5y ago
I have a bunch of reservations, already when reading just the paper's abstract: > The Unix shell is a powerful, ubiquitous, and reviled tool for managing computer systems. Reviled? By those who use it often, it is more loved than reviled. > The shell has been largely ignored by academia and industry. Seems like a bit of an exaggeration, but it's true that corporations have a thing for creating custom proprietary software rather than providing shell-script-based solutions. > While many replacement shells have been proposed, the Unix shell persists. There is no single "the" Unix shell. bash is probably the most popular, but it isn't "the" shell, singular. > Two recent threads of formal and practical research on the shell enable new approaches. We can help manage the shell’s essential shortcomings (dynamism, power, and abstruseness) Again with "the" shell. Anyway, a shall should be dynamic and powerful; and I wouldn't say bash is _terribly_ abstruse. If you apply yourself, your code could actually be rather readable. and address its inessential ones. Improving the shell holdsmuch promise for development, ops, and data processing.
- nomoreusernames 5y ago
  the fact that you can make your own functions and aliases makes me really doubt that people really understand the point of a shell. sure treating everything like a string may seem really dumb. but its easy peasy. because, well everything is a string. once you accept that its not that hard. anytime i can predict i need to repeat a task i can just create a function like work_helpers__reset_dans_password_again, and then you have it there in your tab complete. or why not just run: work_helpers__reset_password dan i think its because people dont want to rtfm. if they would rtfm, they would learn most problems have a solution that is not insane like the one they are about to start developing. you know how often i see this: cat somefile |grep thething instead of just: grep thething somefile
  - effie 5y ago
    The useless cat is actually very useful in practice - it is more readable and easier to modify.
deleted 5y ago
[deleted]
effie 5y ago
> the shell’s semantics is black magic, specified in a 119page impenetrable document that is the POSIX shell specification (with an extra 160pp on utilities!). That's academia going pedantic and mystifying things right there. Strict POSIX compliance was never that important to most people. I never read the POSIX shell specification and never needed it. Writers of shells like bash or ksh probably did and apparently managed this "black magic" well enough to make their shells work well. You can learn shell programming by yourself, from bash manpage and examples used in Linux systems. It's not rocket science. Computer languages get adopted not because they have nice and simple spec, but despite not having a neat and simple spec. It's like with natural languages. There is no authority and no simple spec. And it's not an important problem.
- cryptonector 5y ago
  Thank you. You've saved me bothering with TFA.
effie 5y ago
Authors use some vague and unconvincing arguments about deficiencies of well established tool (shell) in academic research to boost relevance of their obscure research (formal methods and performance of shell regarding multiprocessing). Without giving details of their work in the paper! Just tell me the damn problem your tool solves regarding unix shell programming and why it's cool. Don't waste our time with "rehabilitating shell" in academia.
haecceity 5y ago
Future is powershell
- deleted 5y ago
  [deleted]
effie 5y ago
One good (maybe the only one?) point in the article: > Systemd uses its own vari-able expansion regime, slightly different from the shell’s...encouraging you to runsh -cif you have an actual pipelineto run. Dockerfiles and CI config files arealmostshell scripts,but there’s no convenient way to just execute theRUNcom-mands orscriptlists they contain. And few are the Vagrant-files that don’t call out to someprovision.shto set up de-pendencies. Giving up on the shell means that each ‘modern’system tool will have its own janky quasi-shell language:a decidedly worse situation.
abnercoimbre 5y ago
> Many projects focus on better interactive shells, at some cost to programmability [23]. Innovation in terminal emulators (like Fig and iTerm2) improve user experience, too It's still early / unreleased but I'd like to add my own terminal [0] which lets the emulator have an understanding of user input -- rather than simply delegating to the shell. I'm finding that the interactive pathways that open up are mostly-uncharted territory. [0] https://media.handmade-seattle.com/terminal-click https://media.handmade-seattle.com/terminal-click

138 points by signa11 5y ago

It's a bit clickbity. In the introduction, the paper mostly talks about the ergonomics / maintainability (that resonates the audiencel like me), but the proposed solution, "jash", addresses the performance problem (who cares???). That's said, the brief survey in the first half of the paper is moderately interesting. It's not a real survey but an advertisement of the authors' work. But they've done something worth a quick glance like formalizing shell's semantics [1], which looks interesting if no useful. [1] http://shell.cs.pomona.edu/ http://shell.cs.pomona.edu/

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

dathinab 5y ago

I think what would be more use-full would be a new "shell" which is base on a modern context, incorporating lessens learned from years of cmd-line UX design (ok let's be honest many programs didn't learn anything wrt. UX). But adding jet another POSIX shell which maybe does some parts better but in the end has still not a grate UX because it's a POSIX shell seems kinda pointless IMHO.

yoloClin 5y ago
This. I want a shell that will contextually spit out plaintext in interactive mode, then a JSON object when scripted or piped. Pretty sure this is what Powershell does, but the UI just feels so damn unnatural.
- zozbot234 5y ago
  Powershell does not use JSON. It's based on .NET objects.
- kreetx 5y ago
  Perhaps IDEs and shells could converge?
- hnlmorg 5y ago
  Murex does this. eg Take a plain text table, convert it into an SQL table and run an SQL query: » ps aux | select USER, count(*) GROUP BY USER USER count(*) _installcoordinationd 1 _locationd 4 _mdnsresponder 1 _netbios 1 _networkd 1 _nsurlsessiond 2 _reportmemoryexception 1 _softwareupdate 3 _spotlight 5 _timed 1 _usbmuxd 1 _windowserver 2 lmorg 349 root 134 The builtins usually print human readable output when STDOUT is a TTY, or JSON (or JSONlines) when the TTY is a pipe. » fid-list: FID Parent Scope State Run Mode BG Out Pipe Err Pipe Command Parameters 590 0 0 Executing Normal no out err fid-list (subject to change) » fid-list: | cat ["FID","Parent","Scope","State","RunMode","BG","OutPipe","ErrPipe","Command","Parameters"] [615,0,0,"Executing","Normal",false,"out","err",{},"(subject to change) "] [616,0,0,"Executing","Normal",false,"out","err",{},"cat"] and you can reformat to other data types, eg » fid-list: | format csv FID,Parent,Scope,State,RunMode,BG,OutPipe,ErrPipe,Command,Parameters 703,0,0,Executing,Normal,false,out,err,map[],(subject to change) 704,0,0,Executing,Normal,false,out,err,map[],csv and query data within those data structures using tools that are aware of that structural format. Eg Github's API returns a JSON object and we can filter through it to return just the issue ID's and titles: » open https://api.github.com/repos/lmorg/murex/issues | foreach issue { printf "%2s: %s\n" $issue[number] $issue[title] } 348: Potential regression bug in `fg` 347: Version 2.2 Release 342: Install on Fedora 34 fails (issue with `go get` + `bzr`) 340: `append` and `prepend` should `ReadArrayWithType` 318: Establish a testing framework that can work against the compiled executable, sending keystrokes to it 316: struct elements should alter data type to a primitive 311: No autocompletions for `openagent` currently exist 310: Supprt for code blocks in not: `! { code }` 308: `tabulate` leaks a zero length string entry when used against `rsync --help` Source: https://github.com/lmorg/murex https://github.com/lmorg/murex Website: https://murex.rocks https://murex.rocks
  - AnIdiotOnTheNet 5y ago
    As much as this is an improvement in many ways, using JSON for this feels like it doubles down on part of the problem with the current standard. Every tool is rendering JSON only for the next tool in the pipeline to parse it back out.
- tmerse 5y ago
  I'd like most programs to implement a JSON interface, given right flags. Or maybe the existence of some env var like $DEFAULT_OUTPUT=JSON I like this approach: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc > CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts
hnlmorg 5y ago
This is the approach murex tries to take. It breaks POSIX compliance without remorse but retains compatibility wherever practical. This means it can have some of the flexibilities of Powershell (typed pipelines), the flexibilities of IDEs (events, better auto-completions, etc) but still works with all of your everyday command line tools. My ultra long term aim is to integrate the shell into a media rich terminal emulator so using the command line will have the speed and precision of a TUI, the power of an IDE but the rich content of a GUI. I'm a loooong way off achieving that but the shell is already usable and has been my primary shell for ~4 years now. And I welcome any and all feedback on the current and any future builds. At some point I'll publish a paper with my ideas. https://github.com/lmorg/murex https://github.com/lmorg/murex
- cyberpunk 5y ago
  Just a FYI, murex is (also) a very widely used front of house trading platform, which costs an absolute fortune and has an army of lawyers, you may want to reconsider the name..
  - hnlmorg 5y ago
    Yeah, I get this a lot. I wasn't aware of the trading platform when I created the shell and was quite late into the development before I was made aware. The shell and the trading platform exist in different domains so as far as I'm concerned there isn't any trademark violation but if the trading platform (and their lawyers) disagree then I'm open to dialogue with them. I think HN is uniquely placed where it has a lot of trading topics as well as IT so it wasn't until my shell started trending on here that anyone had even noticed the naming conflict. But my shell had been around for a good few years before it started trending on here.
- pxc 5y ago
  I'm surprised I hadn't heard of Murex yet! It looks pretty similar to Elvish but the type system is more foregrounded (and more complete?). Could you help me differentiate it from Elvish in terms of features and design?
worble 5y ago
I quite like elvish (https://elv.sh https://elv.sh) for a non-POSIX compatible shell. It's by no means perfect, and it's not 1.0 so breaking changes are pretty frequent, but I find the language structure and syntax make it worth it. The docs on the site are a little haphazard too, but once you get a feel for it, you probably won't need to refer to them too often.

js8 5y ago

Recently, I have been looking at the Factor language http://factorcode.org http://factorcode.org. I think I would like to have a shell based on that.

vbsteven 5y ago

The article recognizes that interop and backwards compatibility with existing systems is important for adoption and it does not try to impose a new clean room replacement. This is IMHO a good strategy that has proven itself in other contexts. For example Kotlin/Clojure/Scala all started out focusing on interop with the base Java/JVM technology. They provide a new layer of modern language features that is still compatible with all the existing code without forcing a rewrite. Over time these new languages get adopted, some older libraries get a modern equivalent and that helps these languages to branch out into other areas: e.g. Kotlin Native, multiplatform and ClojureScript. A similar thing can be seen with Typescript. It started as a type layer on top of JavaScript, making it easy to interop with old JS code and over time lots of pure Typescript library equivalents have popped up. Now that the Typescript ecosystem is flourishing we see technology like Deno that slowly moves Typescript away from its JS base. For a post-POSIX shell to be adopted widely I think it will need to use a similar strategy. Edit: re-reading my own comment, I might have just advocated for an Embrace, Extend, Extinguish strategy.

pjmlp 5y ago
That only seems to work for a short period of time, eventually the platform adopts the most interesting features of the candidates to replacement, and they fade away while the large majority of platform users keeps coding away on the language used to write the platform.
- pxc 5y ago
  That's bad news for some new language authors, but not necessarily for programmers, right? These languages still succeed in pushing the norms language design forward, and that's pretty great for the users of programming languages.
  - pjmlp 5y ago
    True, the only issue is that dreams of taking over the world need to be refrained. The only platforms where new languages kind of took over, the move was forced by the company that owns the platform. However you are right, in every case even the system language for the platform got better.

lbayes 5y ago

This line was highly distracting for me, "To make matters worse, the shell has been mostly left for dead by both academia and industry, considering it an unsalvageable piece of junk that needs to be replaced at the first opportunity." Um, false? Many people in industry use and love using the shell. Please read, "In the beginning was the command line." The shell is by far one of the most powerful and expressive tools we have. Yes, there are many ways in which it could be improved, but JSON? That's not it.

pjmlp 5y ago
A REPL is more expressive, allows for graphical and structured output and regardless of UNIX culture books it precedes it by at least 10 years.
hpoe 5y ago
Actually that is the only time I started to find the shell limiting was working with JSON. But then I discovered jq and my issues are solved.
- ur-whale 5y ago
  Yeah, except that jq, beyond 'jq .' is horribly unfriendly.
  - NegativeLatency 5y ago
    I had a similar complaint so I wrote this in ruby https://github.com/nburns/utilities/blob/master/rson https://github.com/nburns/utilities/blob/master/rson
- s_dev 5y ago
  I started using jq as well -- was shocked to find the author was a former comp sci classmate of mine. It's very useful.
hi41 5y ago
I think you can invoke Perl in command line mode. Do people use Perl like that if they want to use Bash command line?

dale_glass 5y ago

The problem with the shell to me is that it started like a good idea, and then just stopped improving. Things that should be the bread and butter of scripting are oddly arcane. Some examples. How do you find where the script itself is? Because if the script comes with files, we want to find those when the script gets called from somewhere outside the current dir. Stack overflow suggests this as a first approach: SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" Then follows with 7 lines of arcane looking code containing a loop for a better solution. That's a ridiculous hoop to jump through for a common need. Dealing with filenames? Hope you enjoy pain. Spaces, newlines and quotes are a horrible pain to deal with properly, and regularly cause trouble. Dealing with correctly passing parameters? More of the same. Dealing with long file lists? You run into the command line length limit. Obtaining some incredibly common system parameter? Some insane combination of grep, awk and cut, probably. Personally I find it amazing that people invented this thing, then spent decades tripping on the same quoting and filename handling issues and not finding that to be a good reason to fix it. These days if it's more than 5 lines of code, or if arbitrary filenames are involved I go straight to Perl or Python, because the amount of completely preventable pain is drastically reduced.

gnubison 5y ago
> How do you find where the script itself is? No program can do this portably.
- chubot 5y ago
  I think you might be mixing two things up: it's impossible for a native program to find where it is, because argv[0] can be anything. But it's possible for an interpreter to tell the script it's interpreting where it lives, i.e. argv[1]. For example, Python does this in a pretty straightforward way.
  - gnubison 5y ago
    You're right, my bad.
5y ago

dwpdwpdwpdwpdwp 5y ago

From Section 2.2 "The Bad" >> B1: Too arbitrary. The shell’s virtue of limitless composition (G1) is also its vice: the shell can compose arbitrary commands written in arbitrary languages. This is what I fundamentally love about the shell, and you can try to pry it from my cold, dead fingers >> B3: Too obscure. The semantics of the shell and common commands are documented in 300pp of standardese [7]. With respect to Bash, I couldn't agree more about the 'standardese' gripe. Man pages would be so, so much more useful if they prioritized showing examples of common/useful ways to use the command. I would love a 'man --examples $COMMAND' feature. Does something like that exist?

revscat 5y ago
Yes. tldr: https://tldr.sh/ https://tldr.sh/
- ahub 5y ago
  Or as an alternative with no installation or dependence : `curl cht.sh tar`. Obviously, replace 'tar' by whatever command you see fit. Works on any machine online with curl available.
  - jbotz 5y ago
    That should probably be `curl cht.sh/tar`
sn41 5y ago
I don't recommend this ever, but I of course make this moonshine on my own computer: I usually edit the man page to add cookbook/bugfix sections. Most of them are cookbook-style recipes. Whenever I find out some obscure bug or gotcha out (spaces in file names passed through a pipe is one of my frequent issues), I go to the man page of cut/tr etc. and add it to the page. Next time I man for that, I remember what I did right a couple of years ago. Another thing I copiously modify is the "SEE ALSO" section for common commands, whenever I install new stuff from github etc. I confess that it is not a proper solution, but it works for me. Man pages have continued in the gloriously terse initial style of Ken Thompson. For mere mortals like I, I need a nicer explanation. My machine's manpages for some commands (find, grep being the most modified) tend to look like the old "HOWTO"s on linux.

ur-whale 5y ago

As much as I love shell ... there's no debugger worthy of the name. That makes it really hard sometimes.

KirillPanov 5y ago
With interpreted languages, "printf debugging" is really easy. It's one of their main advantages. I'd rather have "bash -x" and be able to edit the script than lose those features in order to gain gdb.

zoobab 5y ago

Did Ansible/Puppet, I can only say: "Go away with your shitty DSL!"

mroutput 5y ago

There's too many unsubstantiated statements in this paper to take it seriously.

hules 5y ago

Something that I particularly hate with shell scripting is that if you modify a shell script while it is executing, it will break the running instance: bash does not load the whole script on startup, it reads it line after line while executing it. I wonder why this stupid and dangerous behavior has never been changed.

caymanjim 5y ago
This is not true. I'm not sure what you're seeing, but neither bash nor zsh behave like this in Linux or MacOS.
- effie 5y ago
  GP is correct. Bash does not read and parse whole script on start, that would be slow. It is done when needed.
ptspts 5y ago
Wrap the script with:if :; ...; fi; exit

chrisfinazzo 5y ago

I have said it several times on HN, but it bears repeating: As much as I find shell programming approachable and enjoyable, it really should be avoided if you can help it. Using `set -euo pipefail` and tools like Shellcheck help, but there's a limit. https://sipb.mit.edu/doc/safe-shell/ https://sipb.mit.edu/doc/safe-shell/ ^ Says the guy who forked a project an hour ago just to fix a couple errors, found a subshell and said, 'You're done, go no further' :)

hpoe 5y ago
15 seconds on the shell beats 5 minutes trying to debug a ruby or pythons script. That said I would agree as soon as something becomes complex or you need to do computation looking outside the shell is probably a good idea. The huerestic that I use is 1. Do I need to do math? Move out of the shell. 2. Is the logic starting to get complicated enough that I need consider creating modules? Move out of the shell. 3. Is there enough code to fill more than one screen? Out of the shell I go. 4. Is this going to be something that needs robust error handling and that will be used frequently outside of me? Out of the shell. I've found that as long as I don't hit one of those conditions hacking around in the shell is quite powerful and useful.
- chrisfinazzo 5y ago
  For me (a Comms major/IS minor as an undergrad), most of what I'm manipulating is text and shell is just a tool to speed up the process and as such doesn't have those concerns. Still, I've found Rake and Make - as simple containers for shell rather useful and more flexible - newlines are sane, far less of this `\n` nonsense. ^ An odd complaint, but it just seems like the terminal or the environment should handle this without input from me. Now if only I could do something about Make syntax requiring tabs...
- pjmlp 5y ago
  In what concerns UNIX shell, Python always wins for me, other than very basic scripts to automate launching applications, or a pipeline.
- einpoklum 5y ago

vbsteven 5y ago

What if there was a standard way to describe inputs, outputs and side effects of programs? Think of it like writing/downloading type definitions for existing untyped code. With program descriptions like this shells could become smarter, generate warnings, abort before execution if types don’t match up, etc.

toomanyducks 5y ago
I think I like the theory of this but in practicality I'm pretty sure I'd just get irritated by it. In most usage, there is only one input and output format, being plain text. If I specify a given type further and send it to grep/awk/sed/cut, I'd need to explicitly or implicitly coerce the given output to plain text, at which point I'd either lose the original type information or require some unholy transformations to retrieve the original type. I think this misses how I use the shell: sending plain text information to and from files and executables. The whole point is that there is no type system and everything is just plain text. For something smarter that behaves more like a typed programming language, there's Perl, Python, etc.

pxc 5y ago

The researchers main concerns (making shell programs perform better, especially by better leveraging the parallel computing resources of supercomputing environments), but they do give a nod to stuff I care more about (usability and breaking out of the limits imposed by extant terminal emulator standards). I hope that this jump starts a bit more research into shell languages. Maybe the veneer of authority given by papers suggesting various improvements will help draw people to next-generation shells that take up those improvements and advertise that. If you were not aware, PowerShell has inspired a whole generation of next-generation shells built around structured pipelines, and they're very exciting. My personal favorite is Elvish, which also draws much inspiration from Fish, a lovely 'old-fashioned' (text-only pipeline) shell with a strong emphasis on interactivity and ease of use. One thing that makes Elvish special besides its ambition to be a pleasant and useful general-purpose programming language is its portability: it's typically distributed as a single static binary, and it even has native support for Windows. When I think of 'the future of the shell', I absolutely think of Elvish, namely with respect to: * better portability * high aims as a 'real' programming language * built-in static checks where possible (for now just syntax, I think, but a more complete type system is on the roadmap) * sophisticated and responsive interactive experience out of the box * structured pipelines, structured pipelines, structured pipelines To me, the performance issues the authors address are secondary to what I've enumerated above. But they're right that the fundamentals of the shell are great as well as overdue for some innovation.

effie 5y ago
I'm skeptical about academia being capable to help shell evolve. The paper is a good example why - they focus on justifying themselves and then on obscure academic ideas they like to pursue (POSIX formalization, highly parallel data processing) rather than usability for most users. Sometimes these align, but most of the time they don't. Your description of Elvish is much better in that respect. I'd love more capable shell with modern advancements like types and static checking. However, I don't get why structured data over pipelines is useful. You can already exchange any kind of data over pipeline as string of bytes. Sender and receiver have to be aware of the format in any case.
- 5y ago

deleted 5y ago

[deleted]

usr1106 5y ago

> Unfortunately, this means that the behavior of a shell program cannot be known statically: a simple grep $PWD -in ~/.*shrc This is a weird claim. Every program's bahavior depends on its inputs in the general case. If you want to process static input you can do that using the shell. If your program should work on external inputs you cannot statically determine its output regardless of the language.

usr1106 5y ago

Everybody comments on shell programming, but nobody comments on the main message in their paper PaSH, POSH and JIT. I must admit I did not understand their message at the first quick reading. Formal methods? Well, I'm not whether anybody tries to prove their installation scripts (or whatever people would script) correct. Parallelization? Well, if you try to high-performance computing, is the shell really the place to start?

effie 5y ago
Well they start with "arguments" for why shell programming is deficient, and these arguments seem vague and not well expressed or incorrect. Most people just lose interest in further reading. And when you do read further, it's a lot of research field building with buzzwords that nobody who uses shell cares about.

einpoklum 5y ago

I have a bunch of reservations, already when reading just the paper's abstract: > The Unix shell is a powerful, ubiquitous, and reviled tool for managing computer systems. Reviled? By those who use it often, it is more loved than reviled. > The shell has been largely ignored by academia and industry. Seems like a bit of an exaggeration, but it's true that corporations have a thing for creating custom proprietary software rather than providing shell-script-based solutions. > While many replacement shells have been proposed, the Unix shell persists. There is no single "the" Unix shell. bash is probably the most popular, but it isn't "the" shell, singular. > Two recent threads of formal and practical research on the shell enable new approaches. We can help manage the shell’s essential shortcomings (dynamism, power, and abstruseness) Again with "the" shell. Anyway, a shall should be dynamic and powerful; and I wouldn't say bash is _terribly_ abstruse. If you apply yourself, your code could actually be rather readable. and address its inessential ones. Improving the shell holdsmuch promise for development, ops, and data processing.

nomoreusernames 5y ago
the fact that you can make your own functions and aliases makes me really doubt that people really understand the point of a shell. sure treating everything like a string may seem really dumb. but its easy peasy. because, well everything is a string. once you accept that its not that hard. anytime i can predict i need to repeat a task i can just create a function like work_helpers__reset_dans_password_again, and then you have it there in your tab complete. or why not just run: work_helpers__reset_password dan i think its because people dont want to rtfm. if they would rtfm, they would learn most problems have a solution that is not insane like the one they are about to start developing. you know how often i see this: cat somefile |grep thething instead of just: grep thething somefile
- effie 5y ago
  The useless cat is actually very useful in practice - it is more readable and easier to modify.

deleted 5y ago

[deleted]

effie 5y ago

> the shell’s semantics is black magic, specified in a 119page impenetrable document that is the POSIX shell specification (with an extra 160pp on utilities!). That's academia going pedantic and mystifying things right there. Strict POSIX compliance was never that important to most people. I never read the POSIX shell specification and never needed it. Writers of shells like bash or ksh probably did and apparently managed this "black magic" well enough to make their shells work well. You can learn shell programming by yourself, from bash manpage and examples used in Linux systems. It's not rocket science. Computer languages get adopted not because they have nice and simple spec, but despite not having a neat and simple spec. It's like with natural languages. There is no authority and no simple spec. And it's not an important problem.

cryptonector 5y ago
Thank you. You've saved me bothering with TFA.

effie 5y ago

Authors use some vague and unconvincing arguments about deficiencies of well established tool (shell) in academic research to boost relevance of their obscure research (formal methods and performance of shell regarding multiprocessing). Without giving details of their work in the paper! Just tell me the damn problem your tool solves regarding unix shell programming and why it's cool. Don't waste our time with "rehabilitating shell" in academia.

haecceity 5y ago

Future is powershell

deleted 5y ago
[deleted]

effie 5y ago

One good (maybe the only one?) point in the article: > Systemd uses its own vari-able expansion regime, slightly different from the shell’s...encouraging you to runsh -cif you have an actual pipelineto run. Dockerfiles and CI config files arealmostshell scripts,but there’s no convenient way to just execute theRUNcom-mands orscriptlists they contain. And few are the Vagrant-files that don’t call out to someprovision.shto set up de-pendencies. Giving up on the shell means that each ‘modern’system tool will have its own janky quasi-shell language:a decidedly worse situation.

abnercoimbre 5y ago

> Many projects focus on better interactive shells, at some cost to programmability [23]. Innovation in terminal emulators (like Fig and iTerm2) improve user experience, too It's still early / unreleased but I'd like to add my own terminal [0] which lets the emulator have an understanding of user input -- rather than simply delegating to the shell. I'm finding that the interactive pathways that open up are mostly-uncharted territory. [0] https://media.handmade-seattle.com/terminal-click https://media.handmade-seattle.com/terminal-click