3 ms·

Thanks for the reply. I think you make a fair point regarding shells as programming languages. I guess what I meant regarding missing the point is that, from m

Thanks for the reply. I think you make a fair point regarding shells as programming languages. I guess what I meant regarding missing the point is that, from my perspective, the shell was and is a basic way to interface with your computer system. Basic is a keyword here -- there are other ways to do it (native programs, syscalls, etc), but as a basic interface into operating and running programs on a computer, I think most shells do a pretty good job. As I said in my original comment, maybe they could be doing a better job and I'm just not seeing it, but this is in essence my thought. If Bash is too messy, there are a wealth of other highly portable and maintainable languages to do cross platform scripting on. Python comes to mind, as does Rust, and even older things such as Perl. I'm rambling a bit, I'm not sure my thoughts on the matter are that coherent. I guess in summation I'd just say, I think it's fine if shells are shells, and if they can't be stretched to accommodate hyper specific workflows and patterns, maybe consider writing it another language? Do shells languages have to be as capable as general purpose programming languages, and if so, why?

pxc 5y ago
> I'm rambling a bit, I'm not sure my thoughts on the matter are that coherent. [...] Do shells languages have to be as capable as general purpose programming languages, and if so, why? I think your thoughts here are definitely coherent, and in fact you're getting to the heart of the issue. This exact question is pretty central in motivating many next-generation shells. I'll refer to the documentation of a few of them below. Feel free to follow the associated link to tug on any of the threads and see if the overall argument holds in your opinion. > Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. — Oil Shell: Why create a new shell?: Shouldn't scripts over 100 lines be rewritten in Ruby or Python? ( https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#s... ) > Traditional shells use strings for all kinds of data. They can be stored in variables, used as function arguments, written to output and read from input. Strings are very simple to use, but they fall short if your data has an inherent structure. A common solution is using pseudo-structures of “each line representing a record, and each (whitespace-separated) field represents a property”, which is fine as long as your data do not contain whitespaces. If they do, you will quickly run into problems with escaping and quotation and find yourself doing black magics with strings. […] Elvish offers first-class support for data structures such as lists and maps. — Elvish: Some Unique Semantics: Structureful IO: Motivation ( https://elv.sh/learn/unique-semantics.html#motivation https://elv.sh/learn/unique-semantics.html#motivation ) > bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception). > > Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops. For example, try to check every status code of every program you run, see how your code looks like. Sure you can import 3rd party library for that. Is that as convenient as having automatic checking by default + list of known programs which don’t return zero + convenient syntax for specifying/overriding expected exit code? I guess not. — Next Generation Shell: bash or Python? The Square Pegs and a Round Hole Situation: Both are Inadequate for Ops ( https://ilya-sher.org/2020/10/31/bash-or-python-the-square-pegs-and-a-round-hole-situation/ https://ilya-sher.org/2020/10/31/bash-or-python-the-square-p... ) For me, the great thing about a shell language is that it's a programming language I get to _live_ in. Scripts can just grow organically from commands I chain together at my shell prompt, including simple loops and blocks and stuff. The better I get at the programming language, the better I get at navigating my computer and performing routine tasks. The better I get at navigating my computer and performing routine tasks, the better I get at the programming language. This is a really wonderful kind of thing imo, but its usefulness is undermined when one dimension of the shell (interactivity or programming capability) holds back the combination of the two. The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs. There definitely are shells that (imo) have tried to improve programmability (usually be embedding in a full-fledged programming language) in a way that undermines the simplicity and freeform character of shell languages, for example Rush (embedded in Ruby), Ammonite (embedded in Scala) and Xonsh (embedded in Python). But other next-generation shells, mostly following the examples of PowerShell and Fish (and lots of programming languages), (imo) do a remarkable job of improving programmability without making simple, everyday shell navigation workflows feel any weirder or more laborious. In particular, Elvish, Nushell, and Oil Shell strike me as very thoughtful efforts whose very small teams of contributors have been hacking away at them for quite a long time. One important thing to keep in mind for this particular breed of next-gen shells is that their novelty is mostly in the synthesis of a collection of minor and conservative innovations— things that we've seen in some shells (like PowerShell) and some programming languages (especially Lisps) for a long time. Each of these shells is a substantial leap forward beyond shells like Bash only because the language designs of shells like Bash have been largely frozen for decades. For a concrete example of this perspective: > Nu takes cues from a lot of familiar territory: traditional shells like bash, advanced shells like PowerShell, functional programming, systems programming, and more. But rather than trying to be the jack of all trades, Nu focuses its energy on doing a few things well: > > * Create a flexible cross-platform shell with a modern feel > * Allow you to mix and match commandline applications with a shell that understands the structure of your data > * Have the level of UX polish that modern CLI apps provide — The Nushell Book: Introduction ( https://www.nushell.sh/book/#introduction https://www.nushell.sh/book/#introduction ) That phrase at the end of the second bullet point is key: ‘a shell that understands the structure of your data’. We can see a clear call for shells like that in the growing crop of fancy new CLI tools designed for chopping up, querying, and modifying ‘structured’ (not merely textual or stringly-typed) data. Tools like that frequently show up here on HN: jq for JSON, yq for YAML, xsv for CSV, etc. Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other. The pipelines we use to plug various tools like this into each other, meanwhile, only know about raw text/byte streams. The natural thing to want, when you're faced with this, is to be able to just ingest data of all of these forms into variables in your shell which retain that structure, and query them in a uniform way (that your shell may even be able to assist you with as you type, with some static checking!). The Elvish shell has a nice example of using this to query the latest Elvish issues from GitHub on their website: curl -s https://api.github.com/repos/elves/elvish/issues | from-json | all (one) | each [issue]{ echo $issue[number]: $issue[title] } | head -n 11 which prints, on my system right now: 1396: Exception control: “try” should rethrow flow control exceptions 1391: Document forking behaviour 1385: Feature request: command to find location of rc.elv file 1381: `use edit` creates a crash situation for elvish 1380: Issues running `wal -i` 1377: exit builtin does not trigger program's defer statements 1376: buildin cmd source ! 1374: Feed stdin to all code blocks in run-parallel 1373: Recent performance/speed benchmarks? 1372: Octal Format Specifier 1371: "autoview", tables, new data formats... Imo that's pretty neat and concise! Now that JSON is increasingly a de facto standard interchange format, not just for web apps but often (optionally) for CLI programs, I think having idioms like this in your shell makes a ton of sense. And we don't have to give up the convenience of a simple, fast, filesystem-oriented, interactive shell experience to get it!
- spsesk117 5y ago
  Wow. Thanks for all of this info and for your very interesting perspective. I have a passing familiarity from HN and elsewhere with some of these new shells, but am not well informed enough to really comment on their usefulness/etc. With that said, you've given an excellent primer in your comment -- thanks very much! A few things you said stuck out to me: > For me, the great thing about a shell language is that it's a programming language I get to _live_ in. > The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs. Generally, I agree, and I actually use `eshell` in Emacs for this kind of thing a lot. Being in a pseudo-shell LISP layer allows me to get arbitrarily manipulate text I get back from the command line with elisp and allows for some really flexible workflows. With that said, eshell does of course have many trade offs and limitations, which I won't get into here. I think where these things always fall apart for me though is in the Elvish shell example you provided. To me, that is not functionally better than using Bash and jq, it's just different. Maybe it's better, I'm not sure, but my first impression is that it's neither more concise or more readable, it's just different syntax. I think there is a credible argument to be made for "batteries included" type shells, with something like native json parsing, but at what point does it then tip into being like Powershell or Python, where once again you've gotten away from the native/accessible experience because you need to support `n` kinds of structured data inputs/outputs on `n` platforms? As I was reading your comment, the thought that occurred to me was: Why not just make a python library that can be loaded into the REPL that abstracts over some of the more cumbersome parts of interacting with the OS/filesystem in a shell like way? That seems to be the best of both worlds, and from what I gather that seems like what Rush and Xonsh are doing? I need to look into them further. > Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other. I agree re: structured data manipulation, but I think the solution that has naturally emerged exists for a reason. I can use pipes, and programs like jq to push and pull data into and out of whatever format I need, and each layer of that ecosystem can be maintained in parallel, adapting to changes in the overall landscape. In a sentence, it's the core of the Unix philosophy. One thing and one thing well and all that. I dunno. I've thought a lot about it while typing this comment out. I think it sounds like I'm disagreeing with you, but I'm not. I don't even really think we're debating. I think these new ideas for shells are good things, and I will definitely investigate them more and see if they make my life easier. I just can't seem to shake this feeling that we can't have our cake and eat it too. Maybe I'm looking at it the wrong way though. Maybe in 20 years time shells like Oil and Elvish will be the norm, and we'll be complaining about how they don't handle quantum data structures well without lots of pipes and fd's :D Either way, this has been a very interesting digression. Thanks again for your thoughtful comment and insight.

pxc 5y ago

> I'm rambling a bit, I'm not sure my thoughts on the matter are that coherent. [...] Do shells languages have to be as capable as general purpose programming languages, and if so, why? I think your thoughts here are definitely coherent, and in fact you're getting to the heart of the issue. This exact question is pretty central in motivating many next-generation shells. I'll refer to the documentation of a few of them below. Feel free to follow the associated link to tug on any of the threads and see if the overall argument holds in your opinion. > Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on. — Oil Shell: Why create a new shell?: Shouldn't scripts over 100 lines be rewritten in Ruby or Python? ( https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#s... ) > Traditional shells use strings for all kinds of data. They can be stored in variables, used as function arguments, written to output and read from input. Strings are very simple to use, but they fall short if your data has an inherent structure. A common solution is using pseudo-structures of “each line representing a record, and each (whitespace-separated) field represents a property”, which is fine as long as your data do not contain whitespaces. If they do, you will quickly run into problems with escaping and quotation and find yourself doing black magics with strings. […] Elvish offers first-class support for data structures such as lists and maps. — Elvish: Some Unique Semantics: Structureful IO: Motivation ( https://elv.sh/learn/unique-semantics.html#motivation https://elv.sh/learn/unique-semantics.html#motivation ) > bash does not meet any modern expectations for syntax, error handling nor has ability to work with structured data (beyond arrays and associative arrays which can not be nested). Let it go. You are not usually coding in assembly, FORTRAN, C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your life harder than it should be. Let it go. (Let’s not make it a blanket statement. Use your own judgement when to make an exception). > > Python along with many other languages are general purpose programming languages which were not intended to solve specifically Ops problems. The consequence is longer and less readable scripts when dealing with files or running external programs, which are both pretty common for Ops. For example, try to check every status code of every program you run, see how your code looks like. Sure you can import 3rd party library for that. Is that as convenient as having automatic checking by default + list of known programs which don’t return zero + convenient syntax for specifying/overriding expected exit code? I guess not. — Next Generation Shell: bash or Python? The Square Pegs and a Round Hole Situation: Both are Inadequate for Ops ( https://ilya-sher.org/2020/10/31/bash-or-python-the-square-pegs-and-a-round-hole-situation/ https://ilya-sher.org/2020/10/31/bash-or-python-the-square-p... ) For me, the great thing about a shell language is that it's a programming language I get to _live_ in. Scripts can just grow organically from commands I chain together at my shell prompt, including simple loops and blocks and stuff. The better I get at the programming language, the better I get at navigating my computer and performing routine tasks. The better I get at navigating my computer and performing routine tasks, the better I get at the programming language. This is a really wonderful kind of thing imo, but its usefulness is undermined when one dimension of the shell (interactivity or programming capability) holds back the combination of the two. The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs. There definitely are shells that (imo) have tried to improve programmability (usually be embedding in a full-fledged programming language) in a way that undermines the simplicity and freeform character of shell languages, for example Rush (embedded in Ruby), Ammonite (embedded in Scala) and Xonsh (embedded in Python). But other next-generation shells, mostly following the examples of PowerShell and Fish (and lots of programming languages), (imo) do a remarkable job of improving programmability without making simple, everyday shell navigation workflows feel any weirder or more laborious. In particular, Elvish, Nushell, and Oil Shell strike me as very thoughtful efforts whose very small teams of contributors have been hacking away at them for quite a long time. One important thing to keep in mind for this particular breed of next-gen shells is that their novelty is mostly in the synthesis of a collection of minor and conservative innovations— things that we've seen in some shells (like PowerShell) and some programming languages (especially Lisps) for a long time. Each of these shells is a substantial leap forward beyond shells like Bash only because the language designs of shells like Bash have been largely frozen for decades. For a concrete example of this perspective: > Nu takes cues from a lot of familiar territory: traditional shells like bash, advanced shells like PowerShell, functional programming, systems programming, and more. But rather than trying to be the jack of all trades, Nu focuses its energy on doing a few things well: > > * Create a flexible cross-platform shell with a modern feel > * Allow you to mix and match commandline applications with a shell that understands the structure of your data > * Have the level of UX polish that modern CLI apps provide — The Nushell Book: Introduction ( https://www.nushell.sh/book/#introduction https://www.nushell.sh/book/#introduction ) That phrase at the end of the second bullet point is key: ‘a shell that understands the structure of your data’. We can see a clear call for shells like that in the growing crop of fancy new CLI tools designed for chopping up, querying, and modifying ‘structured’ (not merely textual or stringly-typed) data. Tools like that frequently show up here on HN: jq for JSON, yq for YAML, xsv for CSV, etc. Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other. The pipelines we use to plug various tools like this into each other, meanwhile, only know about raw text/byte streams. The natural thing to want, when you're faced with this, is to be able to just ingest data of all of these forms into variables in your shell which retain that structure, and query them in a uniform way (that your shell may even be able to assist you with as you type, with some static checking!). The Elvish shell has a nice example of using this to query the latest Elvish issues from GitHub on their website: curl -s https://api.github.com/repos/elves/elvish/issues | from-json | all (one) | each [issue]{ echo $issue[number]: $issue[title] } | head -n 11 which prints, on my system right now: 1396: Exception control: “try” should rethrow flow control exceptions 1391: Document forking behaviour 1385: Feature request: command to find location of rc.elv file 1381: `use edit` creates a crash situation for elvish 1380: Issues running `wal -i` 1377: exit builtin does not trigger program's defer statements 1376: buildin cmd source ! 1374: Feed stdin to all code blocks in run-parallel 1373: Recent performance/speed benchmarks? 1372: Octal Format Specifier 1371: "autoview", tables, new data formats... Imo that's pretty neat and concise! Now that JSON is increasingly a de facto standard interchange format, not just for web apps but often (optionally) for CLI programs, I think having idioms like this in your shell makes a ton of sense. And we don't have to give up the convenience of a simple, fast, filesystem-oriented, interactive shell experience to get it!

spsesk117 5y ago
Wow. Thanks for all of this info and for your very interesting perspective. I have a passing familiarity from HN and elsewhere with some of these new shells, but am not well informed enough to really comment on their usefulness/etc. With that said, you've given an excellent primer in your comment -- thanks very much! A few things you said stuck out to me: > For me, the great thing about a shell language is that it's a programming language I get to _live_ in. > The idea with these new shells is to try to find a way to enhance the programming capabilities of shells without making them any less convenient for navigating the filesystem and performing simple tasks with external programs. Generally, I agree, and I actually use `eshell` in Emacs for this kind of thing a lot. Being in a pseudo-shell LISP layer allows me to get arbitrarily manipulate text I get back from the command line with elisp and allows for some really flexible workflows. With that said, eshell does of course have many trade offs and limitations, which I won't get into here. I think where these things always fall apart for me though is in the Elvish shell example you provided. To me, that is not functionally better than using Bash and jq, it's just different. Maybe it's better, I'm not sure, but my first impression is that it's neither more concise or more readable, it's just different syntax. I think there is a credible argument to be made for "batteries included" type shells, with something like native json parsing, but at what point does it then tip into being like Powershell or Python, where once again you've gotten away from the native/accessible experience because you need to support `n` kinds of structured data inputs/outputs on `n` platforms? As I was reading your comment, the thought that occurred to me was: Why not just make a python library that can be loaded into the REPL that abstracts over some of the more cumbersome parts of interacting with the OS/filesystem in a shell like way? That seems to be the best of both worlds, and from what I gather that seems like what Rush and Xonsh are doing? I need to look into them further. > Developers and sysadmins want to be able to pull in structured data from whatever source and query it and manipulate it right there inside their shells. But each format gets its own utilities for that kind of querying and manipulation, and they don't necessarily know how to talk to each other. I agree re: structured data manipulation, but I think the solution that has naturally emerged exists for a reason. I can use pipes, and programs like jq to push and pull data into and out of whatever format I need, and each layer of that ecosystem can be maintained in parallel, adapting to changes in the overall landscape. In a sentence, it's the core of the Unix philosophy. One thing and one thing well and all that. I dunno. I've thought a lot about it while typing this comment out. I think it sounds like I'm disagreeing with you, but I'm not. I don't even really think we're debating. I think these new ideas for shells are good things, and I will definitely investigate them more and see if they make my life easier. I just can't seem to shake this feeling that we can't have our cake and eat it too. Maybe I'm looking at it the wrong way though. Maybe in 20 years time shells like Oil and Elvish will be the norm, and we'll be complaining about how they don't handle quantum data structures well without lots of pipes and fd's :D Either way, this has been a very interesting digression. Thanks again for your thoughtful comment and insight.