4 ms·

> I wish that shell would be more sane. Absolutely agreed, shell is in many respects terrible. >I don't think that a shell should be a complete programming la

> I wish that shell would be more sane. Absolutely agreed, shell is in many respects terrible. >I don't think that a shell should be a complete programming language. On the contrary, I think shell should be a more complete programming language! Drop the stringly typing and add actual types (hence eliminating 80% of bothersome awksedgrep magic; yes, no need to tell me it's a real tall order), add proper error handling... I truly value simplicity where possible, but I think that the primary interface that I use to communicate with my computer should be as powerful as possible. Numerous times I've built up a shell pipeline only to realize near the end that I need to do something that shell is horrible at, and had to redo the whole thing in Python from scratch. I cannot really see a reason to limit it. >Some of the most commonly used utils like awk '{ print $2}', sed, grep, sort, ... should be included out of the box to improve performance and to provide consistency across distributions. UNIX "philosophy" got a lot of things wrong, but not this one. These common utilities you mention are good especially when they're separate, because it's not the shell's job to improve performance, provide consistency, etc... of a dozen and a half different utilities, a number sure only to grow in time as people discover what commonly used thing they want in their shell. Though I'll give you that `awk '{ print $2 }'` should definitely be its own utility or a built-in.

alpaca128 6y ago
Keeping those small utilities separate is definitely the right choice as it keeps everything very flexible. I could just replace grep with ripgrep in my workflow and suddenly get a drastically quicker search without having to change anything else. If all those features were integrated in the shell the user is more or less stuck with the whole package. Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.
geophile 6y ago
Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly typed. Instead of bailing out and redoing everything in Python (when you reach the end of your pipeline), you can add a bit of Python code in your command. For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes: ls -fr \ | select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \ | read -l \ | select (path, line: line.startswith('class ')) read -l reads and labels a file, i.e., filename -> stream of (filename, line in file) (Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)
- geophile 6y ago
  Oh, and when you do bail out and go to Python, marcel's operators are available in a module (marcel.api), so that you can take your pipeline and move it into Python easily. E.g. here's the same example in Python: from marcel.api import * for path, line in (ls(file=True, recursive=True) | select(lambda f: f.suffix == '.py' and now() - f.mtime < days(1)) | read(label=True) | select(lambda path, line: line.startswith('class '))): print(f'({path}: {line}) read(label=True)
- enriquto 6y ago
  > Yes, this is doable with find and awk You don't even need awk for that one: grep ^class `find . -mtime -1 | grep py$` Maybe you could try to use an example where it really makes a difference? Some natural operation that would be very cumbersome with plain shell but is easy and direct with marcel. Otherwise many people may fail to see the point.
  - geophile 6y ago
    Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): read -c foo.csv | sort (*x: int(x[3]) + int(x[6])) I suppose we could argue about what is a "natural" operation. Marcel grew out of a set of tasks that were "natural" in the domain I was working in. An important part of that domain was operating on databases, and clusters of nodes, and databases on those nodes. So marcel has features for those capabilities. Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
    - enriquto 6y ago
      > Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.
ohthehugemanate 6y ago
This is exactly the idea behind Powershell. You don't pipe strings between commands, you pipe typed objects etc etc. I'm convinced that if it had better support on Linux it would usher in a new age if shells. As it is though, it would be hard to use it as a daily driver.[1] [1] https://code.joejag.com/2020/a-month-with-powershell.html https://code.joejag.com/2020/a-month-with-powershell.html
willghatch 6y ago
Since everybody else is peddling their better shells, I'll drop a plug for mine: https://rash-lang.org https://rash-lang.org It's a shell embedded in the Racket programming language. If you're familiar with Xonsh, it's similar, but both more powerful and less polished. It allows easy, recursive mixing of shell code (not posix compatible, but with a similar feel) and Racket. Its interactive mode is not polished (I need to write a better line editor), but it works, and it's great for programming. I haven't worked on it much lately (I need to wrap up other things to finish my PhD), but this weekend I added support for user-programmable substitutions with automatic cleanup. As an example, beyond common substitutions like process substitution (IE <() and >() in bash), I added a demo “closure substitution” form that allows you to send Racket functions to `find -exec`. Importantly, this is something that a user could add, because Rash is super extensible and malleable. Shell is a great DSL, but there are huge advantages to having an embedded DSL rather than a stand-alone DSL. Rash inherits all the cool features of Racket and can be mixed with other Racket languages (eg. Typed Racket, Honu, etc), has advanced features like first-class delimited continuations and the world's most advanced macro system (which makes Rash possible), and any shell script can import functions from Racket's catalog of third-party packages. Embedded shells like Rash allow a shell script to be copied from interactions like you do with Bash, but then grow past the “rewrite in Python” stage gradually with no rewrite -- just a gradual transition from being more shell code to being more “normal” code. Though I'm kicking myself for continually not getting around to rewriting all the documentation, which is still very poor.

alpaca128 6y ago

Keeping those small utilities separate is definitely the right choice as it keeps everything very flexible. I could just replace grep with ripgrep in my workflow and suddenly get a drastically quicker search without having to change anything else. If all those features were integrated in the shell the user is more or less stuck with the whole package. Which is also one of the reasons I prefer using Vim over Emacs; I can embed Vim in any terminal multiplexer of my choice and combine it with various terminal programs. Every single part of that environment is easy to understand and easy to replace with another program or script. It's less homogenous and consistent, but I am fine with that tradeoff. As one might guess I am not a big fan of IDEs.

geophile 6y ago

Check out my pandemic project, marcel: https://marceltheshell.org https://marceltheshell.org. It hits that spot you described: still a shell, but more strongly typed. Instead of bailing out and redoing everything in Python (when you reach the end of your pipeline), you can add a bit of Python code in your command. For example, find all the .py files under the current directory, find the ones that have changed in the last day, and then print the lines (with their filenames) that define classes: ls -fr \ | select (f: f.suffix == '.py' and now() - f.mtime < days(1)) \ | read -l \ | select (path, line: line.startswith('class ')) read -l reads and labels a file, i.e., filename -> stream of (filename, line in file) (Yes, this is doable with find and awk, but this is just an accessible example that gets across the idea of Python functions on the command line.)

geophile 6y ago
Oh, and when you do bail out and go to Python, marcel's operators are available in a module (marcel.api), so that you can take your pipeline and move it into Python easily. E.g. here's the same example in Python: from marcel.api import * for path, line in (ls(file=True, recursive=True) | select(lambda f: f.suffix == '.py' and now() - f.mtime < days(1)) | read(label=True) | select(lambda path, line: line.startswith('class '))): print(f'({path}: {line}) read(label=True)
enriquto 6y ago
> Yes, this is doable with find and awk You don't even need awk for that one: grep ^class `find . -mtime -1 | grep py$` Maybe you could try to use an example where it really makes a difference? Some natural operation that would be very cumbersome with plain shell but is easy and direct with marcel. Otherwise many people may fail to see the point.
- geophile 6y ago
  Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): read -c foo.csv | sort (*x: int(x[3]) + int(x[6])) I suppose we could argue about what is a "natural" operation. Marcel grew out of a set of tasks that were "natural" in the domain I was working in. An important part of that domain was operating on databases, and clusters of nodes, and databases on those nodes. So marcel has features for those capabilities. Partly, I was also scratching an itch: I prefer using Python to exploring the sublanguages of various shell commands. Much cleaner, in my opinion.
  - enriquto 6y ago
    > Read a CSV file, foo.csv, and sort by the sum of the 4th and 7th columns (which are integers): Yes, that's a much better example. Here the "obvious" shell solution involves using awk to compute the sum of the two columns and putting it as the first field, sorting by the first field, and then removing that field. I guess a "rosetta stone" of such examples (e.g. on the frontpage of the site) would make a strong case for the interest of marcel.

ohthehugemanate 6y ago

This is exactly the idea behind Powershell. You don't pipe strings between commands, you pipe typed objects etc etc. I'm convinced that if it had better support on Linux it would usher in a new age if shells. As it is though, it would be hard to use it as a daily driver.[1] [1] https://code.joejag.com/2020/a-month-with-powershell.html https://code.joejag.com/2020/a-month-with-powershell.html

willghatch 6y ago

Since everybody else is peddling their better shells, I'll drop a plug for mine: https://rash-lang.org https://rash-lang.org It's a shell embedded in the Racket programming language. If you're familiar with Xonsh, it's similar, but both more powerful and less polished. It allows easy, recursive mixing of shell code (not posix compatible, but with a similar feel) and Racket. Its interactive mode is not polished (I need to write a better line editor), but it works, and it's great for programming. I haven't worked on it much lately (I need to wrap up other things to finish my PhD), but this weekend I added support for user-programmable substitutions with automatic cleanup. As an example, beyond common substitutions like process substitution (IE <() and >() in bash), I added a demo “closure substitution” form that allows you to send Racket functions to `find -exec`. Importantly, this is something that a user could add, because Rash is super extensible and malleable. Shell is a great DSL, but there are huge advantages to having an embedded DSL rather than a stand-alone DSL. Rash inherits all the cool features of Racket and can be mixed with other Racket languages (eg. Typed Racket, Honu, etc), has advanced features like first-class delimited continuations and the world's most advanced macro system (which makes Rash possible), and any shell script can import functions from Racket's catalog of third-party packages. Embedded shells like Rash allow a shell script to be copied from interactions like you do with Bash, but then grow past the “rewrite in Python” stage gradually with no rewrite -- just a gradual transition from being more shell code to being more “normal” code. Though I'm kicking myself for continually not getting around to rewriting all the documentation, which is still very poor.