8 ms·
Alternative shell with native support for JSON, YAML, TOML, CSV, etc.
- usr1106 6y agoLooks very promising. Does anybody use for in real work? Writing scripts for something that could be buggy sounds frightening.
- linkdd 6y agoThis looks great! I'm highly interested about the error handling and testing baked into it. Moving away from the most common shell syntax is a bold move, but not necessarily a bad one, shell scripting needs some fresh air :)
- hnlmorg 6y ago> I'm highly interested about the error handling and testing baked into it. The big two things with error handling is try[0] and trypipe[1] which work much like `-e` and `-o pipefail` (respectively) in bash. The difference is the syntax is structured more like Java's `try` and `catch` blocks so you can gracefully handle errors without having to check the exit code of each command. The big divergence from your traditional shell is that STDERR is monitored and if the number of bytes written to STDERR is greater than STDOUT then the process will be treated as if it had a non-zero exit code (while not altering the actual exit code just in case `0` is meaningful for that process). Conditional blocks like `if`, `switch` etc also treat the exit code and STDOUT/STDERR output as part of the boolean conditional, eg !if { which foobar } then { out "foobar could not be found" } As for testing, I badly need to add more detail to the `test` docs[2] because they explain the syntax but not how nor why you might want to use it. Essentially there are a few different tests: - inlined test: this is where you write tests inside your function and attach them to a process via a special "test" pipe. When that function is executed the tests are either ignored (normally), or logged (when test mode is enabled). If the test mode is enabled then the tests attached to the process check that the process is in the expected state (eg the right type of content written to STDOUT et al, exit code, etc). This is a good way of monitoring processes in scripts and helps massively with debugging. - state: this really should have been called "watch" since that's the crux of what it is, albeit it's not a simple variable watch, you can write shell scripts as part of the state checks so you can have greater flexibility into the runtime inspection. This is also attached to processes via a special pipe. - unit tests: this is much closer to the kind of testing one typically thinks about as developer. Unit tests, when run, will also output the inlined and state results too and the reporting format can be changed too. There is also a debug mode which is more verbose. It prints, unsurprisingly, debug messages I've written into the shell. But it also prints error messages that might normally be suppressed (eg in the `if` example above, you wouldn't normally want `which` to print it's error message since you only care about whether it passed or failed. But with debug turned on those error messages would also be printed). A few other features that help with day to day operation with regards to errors: - STDERR is printed in red text (this can be optionally turned off if it's not to your tastes) - Builtins print not just the application raising an error but also the line numbers. This is particularly handy given the functional nature of shells (ie every structure, such as `if` etc, is a builtin command). - Every script that's loaded, function/test/autocomplete/etc stored also logs where it was loaded from and when (all visible in the `runtime` command) so you can clearly inspect every bit of running code in murex and where it was loaded from). [0] https://murex.rocks/docs/commands/try.html https://murex.rocks/docs/commands/try.html [1] https://murex.rocks/docs/commands/trypipe.html https://murex.rocks/docs/commands/trypipe.html [2] https://murex.rocks/docs/commands/test.html https://murex.rocks/docs/commands/test.html > Moving away from the most common shell syntax is a bold move, but not necessarily a bad one, shell scripting needs some fresh air :) Yeah. That happened more by chance than design. Happy to discuss more about this if you wish but man this post is already lengthy hehe
- linkdd 6y ago> The difference is the syntax is structured more like Java's `try` and `catch` blocks Which is a big plus, but the most interesting part to me is the "if" behavior you implemented. !if { foobar } then { ... } Which basically translate to Bash (without STDOUT/STDERR handling): foobar if [ $? -ne 0 ] then ... fi This is like 99% of my use cases when I'm writing scripts... > As for testing, [...] they explain the syntax but not how nor why you might want to use it. When I saw the examples, I immediately thought of "validation scripts" that would assert that an environment respects some specification (like the configure script from autotools ?) > but man this post is already lengthy hehe I enjoyed the details :)
- hhvn 6y agoThis works the same as your example in bash. foobar || { echo fail; }
- ridiculous_fish 6y agoCan you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? Have you run into problems with commands that tailor their output to the tty, like clang does? Also how does it work with job control?
- hnlmorg 6y ago> Can you say more about how stderr is monitored? Are you capturing it with a pipe, or do you have another technique? STDERR is read by murex and then printed to the TTY. It's a massive cheat but it does mean I can also colourise easily errors in red. However you can disable this interception and have processes write directly to the TTY with the following command config set proc force-tty true > Have you run into problems with commands that tailor their output to the tty, like clang does? Generally no because STDOUT is linked to the TTY (as long as the process isn't piped). But Bash does get upset if launched from within murex. Hence the following function is hardcoded in murex to wrap around `bash`: # Wrapper script around GNU bash config: set proc force-tty true if { $ARGS -> len -> = -1 } then { exec bash @{ $ARGS -> @[1..] } } else { exec bash } > Also how does it work with job control? It does, but only with external commands (builtins aren't forked processes so adding job control to them is harder - a manual job rather than something given for free from POSIX). Additionally when you stop a running process (^Z) murex outputs some status information on said process. eg ~ » exec sleep 10; echo hello ^Z STDIN: 0 bytes read / 0 bytes written STDOUT: 0 bytes read / 0 bytes written STDERR: 0 bytes read / 0 bytes written FID 429 has been stopped. Use `fg 429` / `bg 429` to manage the FID or `jobs` or `fid-list` to see a list of processes running on this shell. ~ » jobs PID State Background Process Parameters 429 Stopped false exec sleep 10 ~ » fg 429 hello You can also write your own shell scripts that get invoked whenever you do ^Z too (eg if you want to output additional information on said PID). NB A bit of background on the FID vs PID. PID is obviously the UNIX process ID. But since murex builtins aren't forked they're not given a PID yet you might still want to manage that function. So murex has a "function ID" (FID) which is an additional layer of process management. It's a bit of a kludge to get around the shortcomings of not forking murex but it kind of works (the reason murex doesn't fork builtins is itself a shortcut to allow richer pipelines).
- midasuni 6y agoWhat task do you use with a shell that could be performed better by a different type of interactive shell? If it’s more than a few commands I’m going to be writing a script in Perl or Python. If it’s a one off I’m going to be using basic commands like cat, sed, cut, grep, sort, jq, curl etc to investigate But what is the problem is is simple and unique enough not to write a proper utility to tackle it, but complex enough that normal bash isn’t good enough?
- hnlmorg 6y agoNot the OP but maybe I can answer this by describing the history of murex and how the shell evolved: This shell actually started out as a non-REPL tool (like grep, sort, jq, etc) for better log parsing were logs were structured data rather than flat test file. You can see some of the earlier influences with my Apache log parsers also on Github (eg https://github.com/lmorg/firesword https://github.com/lmorg/firesword). But I increasingly wanted to group, count, etc records and not just Apache logs but other structured data. I wrote a few tools that used sqlite3 but I wanted something a little more generic. So I wrote a tool that could parse structured data and supported smarter iteration syntax (`foreach` to iterate through items in an array rather than just lines in a file, `formap` to iterate through items in a hash/map). A lot of this stuff could have been written in Perl, but Perl is slow and I'd often want to parse through several million records of data. Before I knew it, I'd written enough functionality for it to be a shell in it's own right. So the next problem I thought to solve was just how crappy bash is from a UI/UX perspective. I mean I'd had ~20 years of experience using bash so I wasn't scared nor confused by it. But I wanted something that could: - parse man pages to populate auto-completions - something that was easier to template out basic completions - provided meaningful descriptions alongside the auto-completion suggestions - had meaningful context sensitive hints on every keystroke - syntax highlighting - something that could perform fzf-like searches through the completions without having to install and configure numerous additional tools - something with sane defaults Basically the convenience of an IDE in a shell. I've since added other features like: - spell checking - colourised errors (STDERR is written in red so they stand out) - events (so you can define hotkeys, trigger scripts on file system changes, etc) - plugins for smarter tmux integration. eg if I hit "F1" inside tmux the murex will open another pane to the side with the man page of the particular command or murex builtin plus a cheatsheet doc for common usages. Or if it is a function, it will print the source code of the function. So you can easily see references without leaving the interactive prompt. The reason this has expanded into a fully fledged scripting language is because naturally you'd want to write dynamic auto-completions, events, the prompt message and other dynamic components in the same language as the interactive prompt. Thus I then added features like - smarter error handling (eg `try` / `catch`) - testing framework for debugging, catching regressions, etc - namespacing so code can be written as smaller functional components without filling up the global namespace with functions that aren't intended to be called directly - modules so that I can redistribute sane defaults as a base but have an easy way to import and redistribute extended features that might be seen as bloat to other users. I'd still recommend Perl or Python for solving problems that need more powerful / flexible semantics - even if that problem is still a "write many read once" kind of problem. But this shell does bridge the gap between bash and Python while offering a convenient experience inspired by popular IDEs. It's evolved from every pet peeve I've had while working in the command line (if something bothered me then I'd write some code to fix it). So it might not be to everyone's tastes but it does solve real world problems for me as it's been my primary shell for several years now. YMMV
- bonzini 6y agoThis looks great. I am not sure about the different handling of globbing, because globbing seems more common on the command line's everyday use than complex indexing; double quotes are manageable and they could be the default outside top-level context. Zsh's "don't expand words but expand wildcards" seems like a nice middle ground.
- hnlmorg 6y agoAuthor here: Yeah I'm not entirely sold on my approach to globbing either. From an academic perspective I stand by my decision to make globbing functional. But from a usability perspective it kind of sucks. The introduction of the `@g` prefix to allow inlined globbing does make things a little less painful but I'm honestly not sold on that solution either. I'm open to suggestions / changing things up wrt globbing and will take another look at Zsh to see how they've addressed the problem. Thanks for the feedback.
- wodenokoto 6y agoI agree on the concept being brilliant, especially as it gives easy access to use regex instead of glob, which is rather clumsy to do in normal bash. It also prevents a lot of accidental globbing. I would need to use it for a while to decide if the extra syntax outweighs those benefits, but I like the way of thinking!
- Pxtl 6y agoThe worst "accidental glob" is Windows Powershell, where square-brackets in filenames means you can't pipe from "ls" into other cmdlets because it interprets the square-brackets as filter commands even when they read them from the filesystem itself.
- uranusjr 6y agoThis is pretty cool! I’ve been searching for a nice shell to unify my workflows on Mac, Windows, and Linux (WSL). Choices are few and far between, Elvish is the only thing that has worked out (which is fortunately more than decent for me), and I’ll probably give this one a spin.
- xiaq 6y agoElvish author here, glad that you like it, and thanks for the plug :) To save people a search, here's Elvish's homepage: https://elv.sh https://elv.sh
- hnlmorg 6y agoI have a huge amount of respect for Elvish. There's a lot of parallels in design, goals, etc between what I've built and Elvish. I think we even started our projects around the same time too. And I love the fact that you've already got a community using it. If truth be told, seeing the success of Elvish and the number of people who've found it useful has helped inspire me to keep at the boring stuff of my own shell (like writing documentation) just in case anyone else happens to find pleasure in using my humble project too. So I want to thank you for all your work on Elvish and for it's wider contribution to the community :)
- electricEmu 6y agoPowerShell also runs on Mac, Windows, and Linux. You pass objects instead of plaintext through the pipeline which provides a rich experience. The full .NET type and ecosystem are also available when you really need a full fledged class for a thing. There are good choices in 2021. https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7.1 https://docs.microsoft.com/en-us/powershell/scripting/instal...
- hnlmorg 6y agomurex also passes objects. The difference is the objects are passed as UNIX byte streams but with an additional data-type meta field. So processes written in murex can have PowerShell-like objects but at the same time standard POSIX utilities can still be added to the pipeline with no additional effort. That latter point is, in my opinion, the biggest hurdle to using PowerShell. Of course, it ultimately boils down to personal preference and for some, .NET support is a killer feature.
- XorNot 6y agoThis being written in Go is a huge positive. I am curious - if I can alias SSH to force this down the pipe to wherever I'm SSH'ing to, that's a huge adoption benefit.
- lathiat 6y agoThere are a few tools that work like this including sshuttle - transferring a local copy of the tool to the remote end. Mosh also kind of works like this but can’t recall if you technically need the software already installed at the other end. Sshuttle definitely doesn’t.
- theblazehen 6y agoYou may be thinking of https://github.com/danrabinowitz/sshrc https://github.com/danrabinowitz/sshrc?
- sneak 6y agoI don't really understand copying one's .vimrc to a remote machine via ssh to run a remote vim, while vim itself has open/save-over-scp support. It's an extra step, plus typing lag, plus my support tools needed by my vim plug-ins won't be there, et c.
- permalac 6y agoDo you mind giving an example of what do you aliases do?
- ape4 6y agoI wonder if bash could appropriate some of these good ideas.
- mondoshawan 6y agoCould be possible to write these extensions as bash plugin libraries -- I've long thought about adding such features this way, but haven't had the time or pressure enough to really need it yet.
- ilyash 6y agoTo the best of my understanding, probably no. * It's very hard to make changes to bash without breaking anything given the existing amount of code written in bash. * If I remember correctly, bash does not have garbage collection and something tells me, it will not be easy to agree on or to add. Hello, structured data...
- leokennis 6y agoEvery month a cool shell is posted on HN and I get all excited. Then I remember I need to work on hundreds of corporate servers, half of which I cannot install anything on. So the only common denominator shell is bash. So bash it is.
- jquast 6y agoWe could use pseudo pty’s to drive a remote bash through transpilation of a local shell if we wanted to fix that problem. It’s not unpossible.
- ithkuil 6y agoHow much of the complex processing (.e.g. selecting fields in json documents) could be done without having to pipe data back to the client (without relying on tools like jq on the server)?
- chubot 6y agoHm that is an interesting idea and something like it has crossed my mind before. Oil will be well-suited for that strategy since it's bash compatible. I don't know enough about terminals to judge how feasible or easy it is, but if anyone does, please chime in here: https://github.com/oilshell/oil/issues/908 https://github.com/oilshell/oil/issues/908
- dan-robertson 6y agoJust need to wait for a shell with native support for ssh
- gumby 6y agoWhat would that look like? I’ve never felt constrained by just calling ssh.
- dan-robertson 6y ago1. When you ssh you get to continue with the benefits and syntax of your current shell. The closest I know to this is eshell which gets some access to remote hosts through tramp rather than ssh. 2. There’s a small family of commands that take commands as arguments, eg env, xargs, parallel, find, ssh. Maybe they should get special support and the shell should allow you to write subshells to be passed to these commands in some first-class way, similar to the way bash magically converts <(...) into a special file name arg with a process running redirected to that file
- ducktective 6y agoHow is it so that we see some applications that languages like Rust (and C/C++) seem to be tailored for, but more and more cases pop up that the author have chosen Go? Is it a quality vs speed of dev thing? Or am I just wrong in assuming anything system-related should be written in C/C++/Rust?
- coldtea 6y agoThe author choses whatever they're familiar with and likes, and Rust has a larger barrier to learn, so less people use it than Go. Not some huge mystery... >Or am I just wrong in assuming anything system-related should be written in C/C++/Rust? For one, Go has a GC, so memory safety wise it should be as safe as Rust. Second "systems" is too broad a term to be very meaningful. There are excellent systems programs (databases, search engines, etc) written in Java for example...
- erik_seaberg 6y agoGo makes everything mutable with no synchronization, so concurrent slice appends can be lost and map reads and writes can panic. Rust makes you decide which reference is mutable, which takes work but it’s important for the same reasons as static typing.
- hnlmorg 6y agoWhile that's true, it's also true that just because the reference is mutable it doesn't mean the developer is stupid enough to pass mutable references between threads without mutexes to prevent the aforementioned race conditions from happening in the first place. Plus Go also has channels which are a thread safe way of message passing. I'm not taking anything away from Rust but a great many of us have been writing multithreaded code for years before languages like Rust made it safe. These aren't new problem I had to solve. But I'm also not complacent about the risks of multithreaded code and in fact murex's test suite does check for race conditions.
- hnlmorg 6y agoAuthor here: murex pre-dates Rust 1.0 and by the time Rust was stable there was already considerable code committed to murex. I have toyed with the idea of rewriting in Rust but every time I weigh up the pros and cons it becomes obvious the rewrite is just an academic exercise rather than a worthwhile venture (why rewrite something that already works?) The advantage of Go over C++ is that you get memory safety and cross compilation out of the box. Plus Go is a much faster language (for me -- YMMV) to prototype in; which matters when I have a day job and two young kids :D
- ramblerman 6y agoThis looks great, as a somewhat unix novice can someone explain to me the recurring fear I see where people say it wont be compatible for scripting. Can you not separate these worlds? I.e. use an interactive shell like this or fish for your day to day, and still write bash scripts where needed on your server? I'm probably missing something though.
- funkaster 6y agoAs long as your scripts have a valid shebang in text executables (pointing to a “classic” shell), you should be fine
- coldtea 6y ago>Can you not separate these worlds? I.e. use an interactive shell like this or fish for your day to day, and still write bash scripts where needed on your server? You can. Though you miss all kinds of ready to paste code, tutorials, tips, plugins, functions, autocompletions, etc. you can find for established shells...
- hnlmorg 6y agoAuthor here: Agreed. This is one of my biggest regrets for not following POSIX syntax more closely. So much so that I did at one point start writing in support for inlining Bash (and others) code into murex as a way of allowing users to make use of the rich content already out there. But I couldn't decide on a clean way of implementing it that wasn't any better than `bash -c 'some code'` so ended up removing the feature. As it is, I'm still half tempted to incorporate a bit more POSIX support to make code sharing easier. But there's a real risk that I could compromise the design goals of my own language in doing so. I think this is projects like Oil Shell (https://www.oilshell.org/ https://www.oilshell.org/) have the upper hand. It's gone for Bash compatibility plus new features. I have a lot of respect for what they do as it's a much harder undertaking than what I've done.
- gigatexal 6y agoI am super interested in these other shells. (I think they could add a lot to my productivity but I wonder at what cost?) The thing is I am reticent to learn/invest the time into them when I know for portability I will have to stick to bash or the more common shells found on many *nix systems.
- hnlmorg 6y agoI think the crux of it is how much time you spend in the command line on your local machine. If it is substantial then picking a shell you're productive in is little different to picking a graphical desktop environment (knowing that your colleagues could be running a different DE, or even OS). If the majority of your time in the command line is on other peoples machines and/or servers, then you're unlikely to see much, if any, gains picking an alternative shell on your local machine. With regards to murex specifically, there's nothing stopping it from running on servers but pragmatically that's an unrealistic domain to target. What I'm focusing more on is a better experience for running DevOps / developer tools on an engineers local machine. The cost of that is portability - which is, understandably, a deal breaker for lots of people.
- Galanwe 6y ago`->` used instead of `|`? How does that not severely destroy any kind of attempt of compatibility with bash? `-` is often used as parameter to mean "read from stdin" or "output to stdout", and `>` is a redirection. `cat ... ¦ myprog -> log.txt` would now call myprog with no parameter and try to pipe that to a file, instead of just having myprog read from stdin and write myprog's stdout to log.txt A workflow I often use is: `aws s3 cp s3://somewhere/file.txt -` That copies a file to stdout (i.e. it displays the file content from S3). I would then routinely just "arrow up" and add a `> file.txt` to save it. Something I wonder too is why writing a new shell while most of the features could be implemented as just regular programs that you would pipe to? `file.txt -> cast csv -> [ column_name ]` Why the need to create a special syntax with [] I stead of just writing a `filter` program that reads input optionally decorated by that type information added by `cast`?
- cixter 6y agoProbably like this: myprog - > file.txt
- hnlmorg 6y agoAuthor here: > `->` used instead of `|`? How does that not severely destroy any kind of attempt of compatibility with bash? `|` tokens are still supported. I just prefer `->` because I think it's more readable. But that's a personal preference and those who disagree are welcome to use pipe instead. > `cat ... ¦ myprog -> log.txt` would now call myprog with no parameter and try to pipe that to a file, instead of just having myprog read from stdin and write myprog's stdout to log.txt Fair point. That issue is somewhat mitigated by the REPL automatically inserting a space at the end of commands and parameters (ie `-$SPACE>` isn't a token) but I agree there is still a risk. The shell is highly configurable (via the `config` command) and I could easily add a config option to disable `->` tokens in the parser. But I'd wager those who complain about it the most are unlikely to be individuals who would ever convert to using the shell. Which would make pandering to those complaints a fools errand. However if you are sincere about giving this a try but put off by `->` tokens then create a Github issue requesting a `config` option to disable said token I'll gladly implement it. > Something I wonder too is why writing a new shell while most of the features could be implemented as just regular programs that you would pipe to? Because I wanted to :) In fairness though, this evolved into a shell. Originally the syntax was very different and the goal was better log parsing. It was an AWK / Javascript kind of hybrid. But I'm not a massive fan of either language and a shell just organically came from the ashes of the original project. However if you wanted to make use of murex as a command rather than as a shell then you can still do the following: cat example.json | murex -c 'cast json | [ Index ]' > Why the need to create a special syntax with [] I stead of just writing a `filter` program that reads input optionally decorated by that type information added by `cast`? Because murex pipelines aren't just dumb byte streams like in Bash. They fallback to dumb byte streams when the STDIN end is a typical POSIX command, but all other times they pass type information too. This means that `[]` doesn't need casting the majority of the time. It means all the other builtin commands can natively handle JSON arrays like plain text lists. And it means you can use TOML, YAML, S-Expressions, CSV files and so on and so forth without having to learn different tooling for each nor remember to manually cast your data correctly. I've been using this as my primary shell for over 3 years and bar the odd annoyance with how I've implemented globbing, I actually find this quite intuitive vs bash (baring in mind I had ~20 years of experience with bash previously so definitely wasn't a noob). But ultimately it boils down to personal preference and thus I'd never begrudge someone preferring `|` / bash / AWK / etc over any monstrosity I might build. I just make this project public in case anyone else might like it.
- Pxtl 6y agoReminds me a bit of PowerShell, which also tried the "hey, let's do shell but with object-streams instead of string-strings". Unfortunately PowersHell is less than the sum of its parts - it's very powerfull and expressive, but ungodly warty and inconsistent, mashing up a grab-bag of good ideas from POSIX, DOS, and C# to produce a complete mess.
- ilyash 6y agoPowerShell - bad syntax and quirky semantics. Looks like on the syntax front they took the bad parts from bash and Perl.
- ravi-delia 6y agoIn all fairness, bash is...bash. PowerShell does a really good job of striking a balance between the ease of creating horrible hacked together monstrosities in bash with the ease of terrifyingly converting between data structures in python. It obviously doesn't live up to what it could be, but there's definitely something there.
- sidpatil 6y agoPowerShell also seems to borrow from Tcl, especially the parameter syntax (Tcl calls them options, *sh calls them arguments) and the script block construct (which is simply a curly-brace-quoted string in Tcl).
- madhadron 6y agoAnd then the first time you actually systematically parse and edit a giant JSON blob with it you start going, "Okay, this is a serious improvement."
- sixstringtheory 6y agoI use jq to work with JSON in the terminal on macOS. Anyone used both that could compare the two?
- 6y ago
- antihacker_team 6y agoVulnerabilities research. OWASP, PTES, NIST SP800-115. You pay only for the found bug, depending on the criticality. Over than 6 years of experience. email: info@antihacker.pw
- ilyash 6y agoHi. Author of Next Generation Shell here. I had roughly the same motivation. The solution differs widely though. I noted that the main difference of NGS from other modern shells is the attention, prioritization and investment in the programming language. The language is now in a good shape and I'm starting to work on the UI. The first UI will be CLI. https://github.com/ngs-lang/ngs https://github.com/ngs-lang/ngs ( For the curious, readme includes references to other alternative shells )
- ridiculous_fish 6y agoDoes murex use threads? If so, how does it handle the fork/thread interaction?
- karmakaze 6y agoWhat happens if two communicating programs can 'speak' multiple formats--how do they choose one to use? Could we leverage http content negotiation?
- hnlmorg 6y agoMuch like any programming language that supports generics: a function will operate upon a variable based on the data type of that variable. With murex, data types are passed down the pipeline (albeit out of band). The data can be recast, reformatted, etc but by default processes that support multiple formats honour the data type received in STDOUT and thus output in the same format. The Content-Type HTTP head is used to determine data type when the pipeline originates from the web. So there is some awareness there. Likewise file extensions are used if content originates from a file system.