3 ms·

This looks great! I'm highly interested about the error handling and testing baked into it. Moving away from the most common shell syntax is a bold move, but

by linkdd 6y ago

This looks great! I'm highly interested about the error handling and testing baked into it. Moving away from the most common shell syntax is a bold move, but not necessarily a bad one, shell scripting needs some fresh air :)

hnlmorg 6y ago
> I'm highly interested about the error handling and testing baked into it. The big two things with error handling is try[0] and trypipe[1] which work much like `-e` and `-o pipefail` (respectively) in bash. The difference is the syntax is structured more like Java's `try` and `catch` blocks so you can gracefully handle errors without having to check the exit code of each command. The big divergence from your traditional shell is that STDERR is monitored and if the number of bytes written to STDERR is greater than STDOUT then the process will be treated as if it had a non-zero exit code (while not altering the actual exit code just in case `0` is meaningful for that process). Conditional blocks like `if`, `switch` etc also treat the exit code and STDOUT/STDERR output as part of the boolean conditional, eg !if { which foobar } then { out "foobar could not be found" } As for testing, I badly need to add more detail to the `test` docs[2] because they explain the syntax but not how nor why you might want to use it. Essentially there are a few different tests: - inlined test: this is where you write tests inside your function and attach them to a process via a special "test" pipe. When that function is executed the tests are either ignored (normally), or logged (when test mode is enabled). If the test mode is enabled then the tests attached to the process check that the process is in the expected state (eg the right type of content written to STDOUT et al, exit code, etc). This is a good way of monitoring processes in scripts and helps massively with debugging. - state: this really should have been called "watch" since that's the crux of what it is, albeit it's not a simple variable watch, you can write shell scripts as part of the state checks so you can have greater flexibility into the runtime inspection. This is also attached to processes via a special pipe. - unit tests: this is much closer to the kind of testing one typically thinks about as developer. Unit tests, when run, will also output the inlined and state results too and the reporting format can be changed too. There is also a debug mode which is more verbose. It prints, unsurprisingly, debug messages I've written into the shell. But it also prints error messages that might normally be suppressed (eg in the `if` example above, you wouldn't normally want `which` to print it's error message since you only care about whether it passed or failed. But with debug turned on those error messages would also be printed). A few other features that help with day to day operation with regards to errors: - STDERR is printed in red text (this can be optionally turned off if it's not to your tastes) - Builtins print not just the application raising an error but also the line numbers. This is particularly handy given the functional nature of shells (ie every structure, such as `if` etc, is a builtin command). - Every script that's loaded, function/test/autocomplete/etc stored also logs where it was loaded from and when (all visible in the `runtime` command) so you can clearly inspect every bit of running code in murex and where it was loaded from). [0] https://murex.rocks/docs/commands/try.html https://murex.rocks/docs/commands/try.html [1] https://murex.rocks/docs/commands/trypipe.html https://murex.rocks/docs/commands/trypipe.html [2] https://murex.rocks/docs/commands/test.html https://murex.rocks/docs/commands/test.html > Moving away from the most common shell syntax is a bold move, but not necessarily a bad one, shell scripting needs some fresh air :) Yeah. That happened more by chance than design. Happy to discuss more about this if you wish but man this post is already lengthy hehe
- linkdd 6y ago
  > The difference is the syntax is structured more like Java's `try` and `catch` blocks Which is a big plus, but the most interesting part to me is the "if" behavior you implemented. !if { foobar } then { ... } Which basically translate to Bash (without STDOUT/STDERR handling): foobar if [ $? -ne 0 ] then ... fi This is like 99% of my use cases when I'm writing scripts... > As for testing, [...] they explain the syntax but not how nor why you might want to use it. When I saw the examples, I immediately thought of "validation scripts" that would assert that an environment respects some specification (like the configure script from autotools ?) > but man this post is already lengthy hehe I enjoyed the details :)
  - hhvn 6y ago
    This works the same as your example in bash. foobar || { echo fail; }
- ridiculous_fish 6y ago
  Can you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? Have you run into problems with commands that tailor their output to the tty, like clang does? Also how does it work with job control?
  - hnlmorg 6y ago
    > Can you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? STDERR is read by murex and then printed to the TTY. It's a massive cheat but it does mean I can also colourise easily errors in red. However you can disable this interception and have processes write directly to the TTY with the following command config set proc force-tty true > Have you run into problems with commands that tailor their output to the tty, like clang does? Generally no because STDOUT is linked to the TTY (as long as the process isn't piped). But Bash does get upset if launched from within murex. Hence the following function is hardcoded in murex to wrap around `bash`: # Wrapper script around GNU bash config: set proc force-tty true if { $ARGS -> len -> = -1 } then { exec bash @{ $ARGS -> @[1..] } } else { exec bash } > Also how does it work with job control? It does, but only with external commands (builtins aren't forked processes so adding job control to them is harder - a manual job rather than something given for free from POSIX). Additionally when you stop a running process (^Z) murex outputs some status information on said process. eg ~ » exec sleep 10; echo hello ^Z STDIN: 0 bytes read / 0 bytes written STDOUT: 0 bytes read / 0 bytes written STDERR: 0 bytes read / 0 bytes written FID 429 has been stopped. Use `fg 429` / `bg 429` to manage the FID or `jobs` or `fid-list` to see a list of processes running on this shell. ~ » jobs PID State Background Process Parameters 429 Stopped false exec sleep 10 ~ » fg 429 hello You can also write your own shell scripts that get invoked whenever you do ^Z too (eg if you want to output additional information on said PID). NB A bit of background on the FID vs PID. PID is obviously the UNIX process ID. But since murex builtins aren't forked they're not given a PID yet you might still want to manage that function. So murex has a "function ID" (FID) which is an additional layer of process management. It's a bit of a kludge to get around the shortcomings of not forking murex but it kind of works (the reason murex doesn't fork builtins is itself a shortcut to allow richer pipelines).
midasuni 6y ago
What task do you use with a shell that could be performed better by a different type of interactive shell? If it’s more than a few commands I’m going to be writing a script in Perl or Python. If it’s a one off I’m going to be using basic commands like cat, sed, cut, grep, sort, jq, curl etc to investigate But what is the problem is is simple and unique enough not to write a proper utility to tackle it, but complex enough that normal bash isn’t good enough?
- hnlmorg 6y ago
  Not the OP but maybe I can answer this by describing the history of murex and how the shell evolved: This shell actually started out as a non-REPL tool (like grep, sort, jq, etc) for better log parsing were logs were structured data rather than flat test file. You can see some of the earlier influences with my Apache log parsers also on Github (eg https://github.com/lmorg/firesword https://github.com/lmorg/firesword). But I increasingly wanted to group, count, etc records and not just Apache logs but other structured data. I wrote a few tools that used sqlite3 but I wanted something a little more generic. So I wrote a tool that could parse structured data and supported smarter iteration syntax (`foreach` to iterate through items in an array rather than just lines in a file, `formap` to iterate through items in a hash/map). A lot of this stuff could have been written in Perl, but Perl is slow and I'd often want to parse through several million records of data. Before I knew it, I'd written enough functionality for it to be a shell in it's own right. So the next problem I thought to solve was just how crappy bash is from a UI/UX perspective. I mean I'd had ~20 years of experience using bash so I wasn't scared nor confused by it. But I wanted something that could: - parse man pages to populate auto-completions - something that was easier to template out basic completions - provided meaningful descriptions alongside the auto-completion suggestions - had meaningful context sensitive hints on every keystroke - syntax highlighting - something that could perform fzf-like searches through the completions without having to install and configure numerous additional tools - something with sane defaults Basically the convenience of an IDE in a shell. I've since added other features like: - spell checking - colourised errors (STDERR is written in red so they stand out) - events (so you can define hotkeys, trigger scripts on file system changes, etc) - plugins for smarter tmux integration. eg if I hit "F1" inside tmux the murex will open another pane to the side with the man page of the particular command or murex builtin plus a cheatsheet doc for common usages. Or if it is a function, it will print the source code of the function. So you can easily see references without leaving the interactive prompt. The reason this has expanded into a fully fledged scripting language is because naturally you'd want to write dynamic auto-completions, events, the prompt message and other dynamic components in the same language as the interactive prompt. Thus I then added features like - smarter error handling (eg `try` / `catch`) - testing framework for debugging, catching regressions, etc - namespacing so code can be written as smaller functional components without filling up the global namespace with functions that aren't intended to be called directly - modules so that I can redistribute sane defaults as a base but have an easy way to import and redistribute extended features that might be seen as bloat to other users. I'd still recommend Perl or Python for solving problems that need more powerful / flexible semantics - even if that problem is still a "write many read once" kind of problem. But this shell does bridge the gap between bash and Python while offering a convenient experience inspired by popular IDEs. It's evolved from every pet peeve I've had while working in the command line (if something bothered me then I'd write some code to fix it). So it might not be to everyone's tastes but it does solve real world problems for me as it's been my primary shell for several years now. YMMV

by linkdd 6y ago

hnlmorg 6y ago

> I'm highly interested about the error handling and testing baked into it. The big two things with error handling is try[0] and trypipe[1] which work much like `-e` and `-o pipefail` (respectively) in bash. The difference is the syntax is structured more like Java's `try` and `catch` blocks so you can gracefully handle errors without having to check the exit code of each command. The big divergence from your traditional shell is that STDERR is monitored and if the number of bytes written to STDERR is greater than STDOUT then the process will be treated as if it had a non-zero exit code (while not altering the actual exit code just in case `0` is meaningful for that process). Conditional blocks like `if`, `switch` etc also treat the exit code and STDOUT/STDERR output as part of the boolean conditional, eg !if { which foobar } then { out "foobar could not be found" } As for testing, I badly need to add more detail to the `test` docs[2] because they explain the syntax but not how nor why you might want to use it. Essentially there are a few different tests: - inlined test: this is where you write tests inside your function and attach them to a process via a special "test" pipe. When that function is executed the tests are either ignored (normally), or logged (when test mode is enabled). If the test mode is enabled then the tests attached to the process check that the process is in the expected state (eg the right type of content written to STDOUT et al, exit code, etc). This is a good way of monitoring processes in scripts and helps massively with debugging. - state: this really should have been called "watch" since that's the crux of what it is, albeit it's not a simple variable watch, you can write shell scripts as part of the state checks so you can have greater flexibility into the runtime inspection. This is also attached to processes via a special pipe. - unit tests: this is much closer to the kind of testing one typically thinks about as developer. Unit tests, when run, will also output the inlined and state results too and the reporting format can be changed too. There is also a debug mode which is more verbose. It prints, unsurprisingly, debug messages I've written into the shell. But it also prints error messages that might normally be suppressed (eg in the `if` example above, you wouldn't normally want `which` to print it's error message since you only care about whether it passed or failed. But with debug turned on those error messages would also be printed). A few other features that help with day to day operation with regards to errors: - STDERR is printed in red text (this can be optionally turned off if it's not to your tastes) - Builtins print not just the application raising an error but also the line numbers. This is particularly handy given the functional nature of shells (ie every structure, such as `if` etc, is a builtin command). - Every script that's loaded, function/test/autocomplete/etc stored also logs where it was loaded from and when (all visible in the `runtime` command) so you can clearly inspect every bit of running code in murex and where it was loaded from). [0] https://murex.rocks/docs/commands/try.html https://murex.rocks/docs/commands/try.html [1] https://murex.rocks/docs/commands/trypipe.html https://murex.rocks/docs/commands/trypipe.html [2] https://murex.rocks/docs/commands/test.html https://murex.rocks/docs/commands/test.html > Moving away from the most common shell syntax is a bold move, but not necessarily a bad one, shell scripting needs some fresh air :) Yeah. That happened more by chance than design. Happy to discuss more about this if you wish but man this post is already lengthy hehe

linkdd 6y ago
> The difference is the syntax is structured more like Java's `try` and `catch` blocks Which is a big plus, but the most interesting part to me is the "if" behavior you implemented. !if { foobar } then { ... } Which basically translate to Bash (without STDOUT/STDERR handling): foobar if [ $? -ne 0 ] then ... fi This is like 99% of my use cases when I'm writing scripts... > As for testing, [...] they explain the syntax but not how nor why you might want to use it. When I saw the examples, I immediately thought of "validation scripts" that would assert that an environment respects some specification (like the configure script from autotools ?) > but man this post is already lengthy hehe I enjoyed the details :)
- hhvn 6y ago
  This works the same as your example in bash. foobar || { echo fail; }
ridiculous_fish 6y ago
Can you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? Have you run into problems with commands that tailor their output to the tty, like clang does? Also how does it work with job control?
- hnlmorg 6y ago
  > Can you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? STDERR is read by murex and then printed to the TTY. It's a massive cheat but it does mean I can also colourise easily errors in red. However you can disable this interception and have processes write directly to the TTY with the following command config set proc force-tty true > Have you run into problems with commands that tailor their output to the tty, like clang does? Generally no because STDOUT is linked to the TTY (as long as the process isn't piped). But Bash does get upset if launched from within murex. Hence the following function is hardcoded in murex to wrap around `bash`: # Wrapper script around GNU bash config: set proc force-tty true if { $ARGS -> len -> = -1 } then { exec bash @{ $ARGS -> @[1..] } } else { exec bash } > Also how does it work with job control? It does, but only with external commands (builtins aren't forked processes so adding job control to them is harder - a manual job rather than something given for free from POSIX). Additionally when you stop a running process (^Z) murex outputs some status information on said process. eg ~ » exec sleep 10; echo hello ^Z STDIN: 0 bytes read / 0 bytes written STDOUT: 0 bytes read / 0 bytes written STDERR: 0 bytes read / 0 bytes written FID 429 has been stopped. Use `fg 429` / `bg 429` to manage the FID or `jobs` or `fid-list` to see a list of processes running on this shell. ~ » jobs PID State Background Process Parameters 429 Stopped false exec sleep 10 ~ » fg 429 hello You can also write your own shell scripts that get invoked whenever you do ^Z too (eg if you want to output additional information on said PID). NB A bit of background on the FID vs PID. PID is obviously the UNIX process ID. But since murex builtins aren't forked they're not given a PID yet you might still want to manage that function. So murex has a "function ID" (FID) which is an additional layer of process management. It's a bit of a kludge to get around the shortcomings of not forking murex but it kind of works (the reason murex doesn't fork builtins is itself a shortcut to allow richer pipelines).

midasuni 6y ago

What task do you use with a shell that could be performed better by a different type of interactive shell? If it’s more than a few commands I’m going to be writing a script in Perl or Python. If it’s a one off I’m going to be using basic commands like cat, sed, cut, grep, sort, jq, curl etc to investigate But what is the problem is is simple and unique enough not to write a proper utility to tackle it, but complex enough that normal bash isn’t good enough?

hnlmorg 6y ago
Not the OP but maybe I can answer this by describing the history of murex and how the shell evolved: This shell actually started out as a non-REPL tool (like grep, sort, jq, etc) for better log parsing were logs were structured data rather than flat test file. You can see some of the earlier influences with my Apache log parsers also on Github (eg https://github.com/lmorg/firesword https://github.com/lmorg/firesword). But I increasingly wanted to group, count, etc records and not just Apache logs but other structured data. I wrote a few tools that used sqlite3 but I wanted something a little more generic. So I wrote a tool that could parse structured data and supported smarter iteration syntax (`foreach` to iterate through items in an array rather than just lines in a file, `formap` to iterate through items in a hash/map). A lot of this stuff could have been written in Perl, but Perl is slow and I'd often want to parse through several million records of data. Before I knew it, I'd written enough functionality for it to be a shell in it's own right. So the next problem I thought to solve was just how crappy bash is from a UI/UX perspective. I mean I'd had ~20 years of experience using bash so I wasn't scared nor confused by it. But I wanted something that could: - parse man pages to populate auto-completions - something that was easier to template out basic completions - provided meaningful descriptions alongside the auto-completion suggestions - had meaningful context sensitive hints on every keystroke - syntax highlighting - something that could perform fzf-like searches through the completions without having to install and configure numerous additional tools - something with sane defaults Basically the convenience of an IDE in a shell. I've since added other features like: - spell checking - colourised errors (STDERR is written in red so they stand out) - events (so you can define hotkeys, trigger scripts on file system changes, etc) - plugins for smarter tmux integration. eg if I hit "F1" inside tmux the murex will open another pane to the side with the man page of the particular command or murex builtin plus a cheatsheet doc for common usages. Or if it is a function, it will print the source code of the function. So you can easily see references without leaving the interactive prompt. The reason this has expanded into a fully fledged scripting language is because naturally you'd want to write dynamic auto-completions, events, the prompt message and other dynamic components in the same language as the interactive prompt. Thus I then added features like - smarter error handling (eg `try` / `catch`) - testing framework for debugging, catching regressions, etc - namespacing so code can be written as smaller functional components without filling up the global namespace with functions that aren't intended to be called directly - modules so that I can redistribute sane defaults as a base but have an easy way to import and redistribute extended features that might be seen as bloat to other users. I'd still recommend Perl or Python for solving problems that need more powerful / flexible semantics - even if that problem is still a "write many read once" kind of problem. But this shell does bridge the gap between bash and Python while offering a convenient experience inspired by popular IDEs. It's evolved from every pet peeve I've had while working in the command line (if something bothered me then I'd write some code to fix it). So it might not be to everyone's tastes but it does solve real world problems for me as it's been my primary shell for several years now. YMMV