7 ms·

NGS is programming-language-first while other modern shells are typically shell-first. Multiple dispatch would probably be the most prominent manifestation of t

by ilyash 5y ago

NGS is programming-language-first while other modern shells are typically shell-first. Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.

laumars 5y ago
> NGS is programming-language-first while other modern shells are typically shell-first. Not always no. Even if your point were true, being programmer-first is not always a desirable feature. The vast majority of shell work is basic and repetitive. Most of the time people just want something that functions a little like Bash but less shitty for scripting. The fact that Powershell, LISP nor Python haven’t taken over the world for shell usage proves that a higher level language REPL generally makes for a shit daily shell. So usually you end up with just a small few purists using it. And that’s not really good enough. Whereas Oil, Elvish, Murex and even Fish (to a less dramatic extent) are looking at what makes a good shell and then fixing shell scripting within that shell. Having gone through the REPL phase myself with a great many different languages and found them all painful for daily use, I’m inclined to agree with shell-first approach. > Multiple dispatch would probably be the most prominent manifestation of this approach in NGS. Again, NGS isn’t unique in that regard. Murex has methods, Powershell has methods. I’ve not seen anything new in NGSs “Multiple Dispatch” nor “MultiMethod” docs that other alt shells aren’t also doing. Don’t take this the wrong way, it looks very impressive what you’re doing. But it’s not unique these days. And I say this having tried a great many options out there.
- ilyash 5y ago
  > being programmer-first is not always a desirable feature. programming-language-first allows convenient scripting. I could not think of a way to make anything shell-first be convenient for scripting (anything beyond tiny scale). > higher level language REPL generally makes for a shit daily shell. I think it proves that general purpose languages, where using them as a shell was afterthought "makes for a shit daily shell". NGS, on the other hand, has somewhat-bash-like (read: easily run external programs) syntax at the top level, which should be good for CLI. Want to use more advanced features that are typically associated with "real" programming languages? OK, pay the price, switch syntax with { ... } and have full blown programming language at your disposal, in the shell. > NGS isn’t unique in that regard. I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? > Murex has methods, Powershell has methods. Just looked at Murex docs. No multiple dispatch mentioned. I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html https://murex.rocks/docs/commands/function.html There is a huge difference in having methods and multiple dispatch. > not unique these days NGS is a mixture of "borrowed" features and unique ones. Multiple dispatch is an old concept and has been implemented in other programming languages. Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes. Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. Edit: and some other features around handling exit codes such as short syntax to provide expected exit code (any other exit code throws exception) Edit: clarification - NGS knows that some external programs have exit code 1 which does not signify an error.
  - laumars 5y ago
    > I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? Murex does this but in a slightly different way. Rather than method overloading (which is a bad feature in any language in my opinion but I get it has its fans) murex has APIs that allow for writing data type agnostic methods. in practice that means the same tools to query a length of an array, or grab an item in a map (etc) work irrespective of whether the data type is a JSON array or map, YAML, CSV, S-Expressions or even just ‘ps’ (etc) output…and so on and so forth. This expands out to all methods, meaning you basically have a ‘jq’ like toolset that works exactly the same - same commands etc - for most data formats (certainly the ones I use daily). Which in effect is the same end result as what you’re describing except without the method overloading and instead allowing me as a shell script writer not to worry less about the data type or format of the piped data. There are a few specific methods that do allow for overloading though, but that’s where functionality is a little more complex (these are usually event handlers though. I don’t use them much so I can’t say how well they work) > NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. Murex does this too. I’m pretty sure I’ve seen other shells do that as well (possibly Elvish?) Iconically it was actually the error handling that drew me to murex and the smart handling of data types within methods that kept me there. Basically all the same features you’re promoting in NGS. Which is why I’m saying there is a lot out there that does the same. Anyhow, it’s been interesting reading about your work on NGS. Good luck
  - hnlmorg 5y ago
    murex author here. Hopefully I can answer some questions: > I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? I've not heard of the term "multiple dispatch" but reading the thread it sounds like you're describing function overloading. Powershell does support this with classes[0]. murex does it's overloading at the API level[1]. The reason behind that decision is to keep the methods simple (eg a pipeline might contain JSON or CSV data) because you as a shell user don't want to run into situations where you've written a function that supports one data type but not another, you just want it to work first time. So murex automatically abstracts that part away for you. In addition to the point the other poster mentioned about consistent `jq`-like methods that are data type agnostic, murex allows for easy iteration through objects agnostic of their data type (eg `open somedata -> foreach { do stuff }` -- where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way). There are some specific builtins that support "overloading" of sorts but instead of overloading the function they have handlers that call user defined functions[2][3]. This removes the mystery behind function overloading because you can easily trace which handlers exist for which data types. It's also worth adding that function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority. There are also private[4] functions (which are namespaced) so you have additional controls to avoid accidental overloading when writing modules too. And the interactive shell expands out any command (eg displays a hint text stating if a command is a function, alias, external command, etc) so there's a reduced risk of being unaware if `ls` is an alias, function, `/bin/ls`, or even a symlink to something else. > I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html https://murex.rocks/docs/commands/function.html Named parameters are optional because neither Windows nor POSIX have any understanding of named arguments in their processes. I did consider abstracting that into murex's functions regardless like Python et al would but I couldn't design a way that wasn't jarring nor cumbersome to write in a hurry nor worked transparently with Windows and POSIX ARGS[]. So I've come up with an optional builtin called `args`[5] which allows you to define flags, which are effectively the same thing as named parameters except they're supported natively by Windows and POSIX ARGS[], so you can write a murex script as a native shell script without leaving the user to write another abstraction layer themselves interpreting an indexed argument into an named arguments. Thus giving you the flexibility to write really quick Bash-like functions or more verbose scripting style functions depending on your need. > Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes. Both of these are baked into murex as well. The run-command-and-parse-output is the API stuff mentioned above. It's where the overloading happens. As for error handling: any command is considered a failure if there is either a non-zero exit code, STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across. Additionally STDERR is highlighted red by default and when a process fails you're given a mini-stack trace showing where in the script the error happened. You have try/catch blocks, saner `if` syntax etc that all use the same API for detecting if a process has failed too. Keeping everything consistent. The other thing murex has to help catch bugs and error is a testing framework baked into the shell language. The docs for test[7] need expanding but in essence: - you can write proper unit tests - you can intercept STDOUT (even if it's mid pipeline) and test that output. This works around instances where (1) you can't have full on unit tests due to side effects in a function that can't be easily mocked (2) you want to add your own debugging routines (rather than just printing values to STDOUT or running commands manually to see their output -- like one would normally have to do with shell scripting) - you can add state watches. Except in murex the watches are Turing complete so you're not just adding noise to your terminal output but rather putting meaningful debug messages and scripts. And all of the debugging stuff can be written straight into your normal shell routines and cause no additional execution overhead unless you've purposely enabled test mode. So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too. > Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. If you're describing that as a "two branch" approach then technically murex could be argued as having "three branches" because it checks exit code, STDERR contents, and payload size too. Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables. I guess where NGS and murex really differ is NGS likes to expose its smart features whereas in murex they're abstracted away a little (they can still be altered, customised, etc) to keep the daily mundane shell usage as KISS (keep it simple stupid) as possible. eg you can overload function calls if you really wanted but that can often cause unforeseen complications or other unexpected annoyances right when you least want it to. So murex keeps that stuff around for when you need it but finds ways to avoid people needing to rely on it and furthermore unwraps the covers of what a routine does in the interactive terminal. It's all about offering abstractions but removing surprises. > NGS knows that some external programs have exit code 1 which does not signify an error. Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided. I do completely understand the incentive behind wanting to do this and wouldn't criticise others for doing that but given external programs can change without the shell being aware, they can be overloaded with aliases, functions, etc, and they might even just differ between Linux, BSD, Windows, etc -- well it just seemed like hard coding executable behaviour causes more potential issues than it solves. You also then run into problems where users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions. Ultimately it creates a kind of special magic that is unpredictable and outside the control of the shell itself. So instead I've relied on having foundational logic that is consistent. It's the one side of shell programming where I've placed the responsibility onto the developer to get right rather than "automagically" doing what I think they are expecting. This of course does mean I have to focus even harder on ensuring all other aspects of the shell are predictable and low maintenance so that the developer can cover any specific edge cases with ease. Which goes a long way to explaining why I've chosen the path I've chosen (ie reducing the amount of syntax sugar, overloading, etc needed for daily use that one might rely on in a more traditional programming language). Links: [0] https://stackoverflow.com/a/57294942 https://stackoverflow.com/a/57294942 [1] https://murex.rocks/docs/GUIDE.apis.html https://murex.rocks/docs/GUIDE.apis.html [2] https://murex.rocks/docs/commands/openagent.html https://murex.rocks/docs/commands/openagent.html && https://murex.rocks/docs/commands/open.html https://murex.rocks/docs/commands/open.html [3] https://murex.rocks/docs/commands/event.html https://murex.rocks/docs/commands/event.html (I need to write a lot more documentation around event driving programming in murex) [4] https://murex.rocks/docs/commands/private.html https://murex.rocks/docs/commands/private.html [5] https://murex.rocks/docs/commands/args.html https://murex.rocks/docs/commands/args.html [6] https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d229cc32d68404c061/lang/types/types.go#L38 https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d2... [7] https://murex.rocks/docs/commands/test.html https://murex.rocks/docs/commands/test.html
hnlmorg 5y ago
Coincidentally I've recently written a blog post about just this topic: https://murex.rocks/docs/blog/split_personalities.html https://murex.rocks/docs/blog/split_personalities.html

by ilyash 5y ago

NGS is programming-language-first while other modern shells are typically shell-first. Multiple dispatch would probably be the most prominent manifestation of this approach in NGS.

laumars 5y ago

> NGS is programming-language-first while other modern shells are typically shell-first. Not always no. Even if your point were true, being programmer-first is not always a desirable feature. The vast majority of shell work is basic and repetitive. Most of the time people just want something that functions a little like Bash but less shitty for scripting. The fact that Powershell, LISP nor Python haven’t taken over the world for shell usage proves that a higher level language REPL generally makes for a shit daily shell. So usually you end up with just a small few purists using it. And that’s not really good enough. Whereas Oil, Elvish, Murex and even Fish (to a less dramatic extent) are looking at what makes a good shell and then fixing shell scripting within that shell. Having gone through the REPL phase myself with a great many different languages and found them all painful for daily use, I’m inclined to agree with shell-first approach. > Multiple dispatch would probably be the most prominent manifestation of this approach in NGS. Again, NGS isn’t unique in that regard. Murex has methods, Powershell has methods. I’ve not seen anything new in NGSs “Multiple Dispatch” nor “MultiMethod” docs that other alt shells aren’t also doing. Don’t take this the wrong way, it looks very impressive what you’re doing. But it’s not unique these days. And I say this having tried a great many options out there.

ilyash 5y ago
> being programmer-first is not always a desirable feature. programming-language-first allows convenient scripting. I could not think of a way to make anything shell-first be convenient for scripting (anything beyond tiny scale). > higher level language REPL generally makes for a shit daily shell. I think it proves that general purpose languages, where using them as a shell was afterthought "makes for a shit daily shell". NGS, on the other hand, has somewhat-bash-like (read: easily run external programs) syntax at the top level, which should be good for CLI. Want to use more advanced features that are typically associated with "real" programming languages? OK, pay the price, switch syntax with { ... } and have full blown programming language at your disposal, in the shell. > NGS isn’t unique in that regard. I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? > Murex has methods, Powershell has methods. Just looked at Murex docs. No multiple dispatch mentioned. I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html https://murex.rocks/docs/commands/function.html There is a huge difference in having methods and multiple dispatch. > not unique these days NGS is a mixture of "borrowed" features and unique ones. Multiple dispatch is an old concept and has been implemented in other programming languages. Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes. Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. Edit: and some other features around handling exit codes such as short syntax to provide expected exit code (any other exit code throws exception) Edit: clarification - NGS knows that some external programs have exit code 1 which does not signify an error.
- laumars 5y ago
  > I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? Murex does this but in a slightly different way. Rather than method overloading (which is a bad feature in any language in my opinion but I get it has its fans) murex has APIs that allow for writing data type agnostic methods. in practice that means the same tools to query a length of an array, or grab an item in a map (etc) work irrespective of whether the data type is a JSON array or map, YAML, CSV, S-Expressions or even just ‘ps’ (etc) output…and so on and so forth. This expands out to all methods, meaning you basically have a ‘jq’ like toolset that works exactly the same - same commands etc - for most data formats (certainly the ones I use daily). Which in effect is the same end result as what you’re describing except without the method overloading and instead allowing me as a shell script writer not to worry less about the data type or format of the piped data. There are a few specific methods that do allow for overloading though, but that’s where functionality is a little more complex (these are usually event handlers though. I don’t use them much so I can’t say how well they work) > NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. Murex does this too. I’m pretty sure I’ve seen other shells do that as well (possibly Elvish?) Iconically it was actually the error handling that drew me to murex and the smart handling of data types within methods that kept me there. Basically all the same features you’re promoting in NGS. Which is why I’m saying there is a lot out there that does the same. Anyhow, it’s been interesting reading about your work on NGS. Good luck
- hnlmorg 5y ago
  murex author here. Hopefully I can answer some questions: > I do scan alternative shells from time to time. I could have missed but I didn't see multiple dispatch in any of them. Which ones have it? Just to clarify: In which shell you can define several methods with the same name and when called, the method to invoke is selected based on the types of the arguments? I've not heard of the term "multiple dispatch" but reading the thread it sounds like you're describing function overloading. Powershell does support this with classes[0]. murex does it's overloading at the API level[1]. The reason behind that decision is to keep the methods simple (eg a pipeline might contain JSON or CSV data) because you as a shell user don't want to run into situations where you've written a function that supports one data type but not another, you just want it to work first time. So murex automatically abstracts that part away for you. In addition to the point the other poster mentioned about consistent `jq`-like methods that are data type agnostic, murex allows for easy iteration through objects agnostic of their data type (eg `open somedata -> foreach { do stuff }` -- where you don't need to think about data types, file formats, etc, murex does the heavy lifting for you in a consistent and predictable way). There are some specific builtins that support "overloading" of sorts but instead of overloading the function they have handlers that call user defined functions[2][3]. This removes the mystery behind function overloading because you can easily trace which handlers exist for which data types. It's also worth adding that function overloading is supported in the same way that its also supported in Bash too where you might have a function but if there is a alias of the same name that will take priority. There are also private[4] functions (which are namespaced) so you have additional controls to avoid accidental overloading when writing modules too. And the interactive shell expands out any command (eg displays a hint text stating if a command is a function, alias, external command, etc) so there's a reduced risk of being unaware if `ls` is an alias, function, `/bin/ls`, or even a symlink to something else. > I did not even see how to define named parameters - https://murex.rocks/docs/commands/function.html https://murex.rocks/docs/commands/function.html Named parameters are optional because neither Windows nor POSIX have any understanding of named arguments in their processes. I did consider abstracting that into murex's functions regardless like Python et al would but I couldn't design a way that wasn't jarring nor cumbersome to write in a hurry nor worked transparently with Windows and POSIX ARGS[]. So I've come up with an optional builtin called `args`[5] which allows you to define flags, which are effectively the same thing as named parameters except they're supported natively by Windows and POSIX ARGS[], so you can write a murex script as a native shell script without leaving the user to write another abstraction layer themselves interpreting an indexed argument into an named arguments. Thus giving you the flexibility to write really quick Bash-like functions or more verbose scripting style functions depending on your need. > Examples of things in NGS that I have not seen anywhere else: syntax for run-command-and-parse-output, proper handling of exit codes. Both of these are baked into murex as well. The run-command-and-parse-output is the API stuff mentioned above. It's where the overloading happens. As for error handling: any command is considered a failure if there is either a non-zero exit code, STDERR contains a failed message (eg "false", "failed", etc) or STDERR is > STDOUT[6]. This has covered every use case I've come across. Additionally STDERR is highlighted red by default and when a process fails you're given a mini-stack trace showing where in the script the error happened. You have try/catch blocks, saner `if` syntax etc that all use the same API for detecting if a process has failed too. Keeping everything consistent. The other thing murex has to help catch bugs and error is a testing framework baked into the shell language. The docs for test[7] need expanding but in essence: - you can write proper unit tests - you can intercept STDOUT (even if it's mid pipeline) and test that output. This works around instances where (1) you can't have full on unit tests due to side effects in a function that can't be easily mocked (2) you want to add your own debugging routines (rather than just printing values to STDOUT or running commands manually to see their output -- like one would normally have to do with shell scripting) - you can add state watches. Except in murex the watches are Turing complete so you're not just adding noise to your terminal output but rather putting meaningful debug messages and scripts. And all of the debugging stuff can be written straight into your normal shell routines and cause no additional execution overhead unless you've purposely enabled test mode. So its fair to say I've spent a significant amount of time designing smarter ways of handling failures than your average shell scripting language. As I'm sure you have too. > Regarding exit codes. Typical approach to exit codes varies. Python - "I don't care, the programmer should handle it". Some Python libraries and other places - "Non-zero is an error". That's simplistic and does not reflect the reality in which some utilities return 1 for "false". bash (and probably other shells too) is unable to handle in a straightforward manner situation where external command can return exit codes for "true", "false" and "error". It just doesn't fit in the "if" with two branches. NGS does handle it with "if" with two branches + possible exception thrown for "error" exit code. If you're describing that as a "two branch" approach then technically murex could be argued as having "three branches" because it checks exit code, STDERR contents, and payload size too. Personally I just describe it as "error handling" because in my view this is how peoples expectations are rather than the reality of handling forked executables. I guess where NGS and murex really differ is NGS likes to expose its smart features whereas in murex they're abstracted away a little (they can still be altered, customised, etc) to keep the daily mundane shell usage as KISS (keep it simple stupid) as possible. eg you can overload function calls if you really wanted but that can often cause unforeseen complications or other unexpected annoyances right when you least want it to. So murex keeps that stuff around for when you need it but finds ways to avoid people needing to rely on it and furthermore unwraps the covers of what a routine does in the interactive terminal. It's all about offering abstractions but removing surprises. > NGS knows that some external programs have exit code 1 which does not signify an error. Having different behaviours for different executables hard coded into the shell is one behaviour I purposely avoided. I do completely understand the incentive behind wanting to do this and wouldn't criticise others for doing that but given external programs can change without the shell being aware, they can be overloaded with aliases, functions, etc, and they might even just differ between Linux, BSD, Windows, etc -- well it just seemed like hard coding executable behaviour causes more potential issues than it solves. You also then run into problems where users expect the shell to understand all external executables but there are some you haven't anticipated thus breaking expectation / assumptions. Ultimately it creates a kind of special magic that is unpredictable and outside the control of the shell itself. So instead I've relied on having foundational logic that is consistent. It's the one side of shell programming where I've placed the responsibility onto the developer to get right rather than "automagically" doing what I think they are expecting. This of course does mean I have to focus even harder on ensuring all other aspects of the shell are predictable and low maintenance so that the developer can cover any specific edge cases with ease. Which goes a long way to explaining why I've chosen the path I've chosen (ie reducing the amount of syntax sugar, overloading, etc needed for daily use that one might rely on in a more traditional programming language). Links: [0] https://stackoverflow.com/a/57294942 https://stackoverflow.com/a/57294942 [1] https://murex.rocks/docs/GUIDE.apis.html https://murex.rocks/docs/GUIDE.apis.html [2] https://murex.rocks/docs/commands/openagent.html https://murex.rocks/docs/commands/openagent.html && https://murex.rocks/docs/commands/open.html https://murex.rocks/docs/commands/open.html [3] https://murex.rocks/docs/commands/event.html https://murex.rocks/docs/commands/event.html (I need to write a lot more documentation around event driving programming in murex) [4] https://murex.rocks/docs/commands/private.html https://murex.rocks/docs/commands/private.html [5] https://murex.rocks/docs/commands/args.html https://murex.rocks/docs/commands/args.html [6] https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d229cc32d68404c061/lang/types/types.go#L38 https://github.com/lmorg/murex/blob/19d2aeb27e814a4c6fec77d2... [7] https://murex.rocks/docs/commands/test.html https://murex.rocks/docs/commands/test.html

hnlmorg 5y ago

Coincidentally I've recently written a blog post about just this topic: https://murex.rocks/docs/blog/split_personalities.html https://murex.rocks/docs/blog/split_personalities.html