4 ms·

We've had this discussion before[1]. :) It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work

by imiric 3y ago

We've had this discussion before[1]. :) It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work in the exact way you expect it to. You could just as well implement a version of `rm` that parses stdin in a specific way by default, but that would likely make it work for that specific use case, and not in the generic and composable way the tools are meant to be used. The whole point of Unix is for tools to be independent of each other _and_ of the environment they run in, while giving the user the freedom to compose them in any way they need. What Nushell has built instead is a closed ecosystem of tools that interoperate well with each other, but not with the outside environment. This means that a) the ecosystem is not extensible unless one contributes to Nushell, or directly implements Nushell features, and b) external tools will always be second-class citizens, that might even be incompatible with Nushell. To give you an example, how would I use GNU ls within Nushell? Suddenly, I can't use any of the `where` fanciness, and my entire pipeline breaks. I would have to resort to another Nushell-specific helper to integrate the command, which needs to be written in a very generic way to support a wide range of use cases, or switch my entire pipeline to use external commands only, which defeats the purpose of using Nushell to begin with. This is a contrived example, but if you compound this with the amount of CLI tools that exist and have yet to be written, it's a usability and maintenance headache. So I'm glad that Nushell exists, as it challenges existing notions of what a shell can be, but let's not disparage existing shells or Unix to prove a point. The Unix ecosystem is so widespread and successful today _because_ of these early decisions, and will more than likely continue to exist because of them as well. That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem. [1]: https://news.ycombinator.com/item?id=36706617 https://news.ycombinator.com/item?id=36706617

nextaccountic 3y ago
You just can't parse the output of ls and every bash guide will tell you that. This means that ls has no place in shell scripts unfortunately
salohcin 3y ago
I think these are both orthogonal arguments. @p-e-w is arguing for tools that neatly and simply compose workflows, while you're arguing for independence of tools set. Let's explore further. > It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work in the exact way you expect it to. Fair statement, but it's never just one example. It's tonnes of examples that can be made. There are groups of tools whose sole function is just to help stitch other commands together (I.e. getting ls and rm into the same workflow). It works, but the point being made is that it could be better. > You could just as well implement a version of `rm` that parses stdin in a specific way by default, but that would likely make it work for that specific use case, and not in the generic and composable way the tools are meant to be used. I would argue this is _exactly_ what's happened - just that it isn't for only rm, but a whole common subset of tools to bootstrap a new way of working. Your exact argument works still for nushell, if someone wants a new tool for rm (or the legacy one) they could still plausibly do that - and your point above would still hold, it would completely defeat the purpose of nushell, as suddenly _where_ doesn't work. That same argument however applies for current shells... If I suddenly swap out rm, my scripts at some point are going to start failing. Unfortunately we're tightly bound to an environment and ecosystem based on legacy decisions. There are plenty of examples of how that's played out with technology in other ways (for good and bad) - the question should be, is this the right evolution? > That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem. I would point out that no good reason has been given for no contract in existing shells? Or maybe we should observe that there is actually a contract, but it's just a lot less structured. To the point that the contract is just an agreed historical structure that popular individual programs are expected to use. Regardless, a contract isn't a bad thing, especially when it's done correctly to balance flexibility and structure. I'm not convinced nushell is the way to go, but I would prefer the debate not to be muddled. Sure current shells have done fine. We expect future shells to do better. Will they imbue the Unix philosophy perfectly? Well like any good philosophy that will be up for debate.
- fiddlerwoaroof 3y ago
  > There are groups of tools whose sole function is just to help stitch other commands together This is an example of the Unix philosophy working: replace smarts in the tools with smart compositions of tools; i.e. don’t make programs have to determine whether to read file names from arguments or stdin, use arguments by default and `xargs` to convert stdin into arguments.
qazxcvbnm 3y ago
In that sense, perhaps that facet of capabilities of Nushell would be more comparable to a Foreign-data Wrapper for SQL rather than a shell with pipes.
follower 3y ago
> What Nushell has built instead is a closed ecosystem of tools that interoperate well with each other, but not with the outside environment. IMO "closed ecosystem" feels like a mischaracterization of the project--there's multiple features that are designed to support interoperability outside the nushell environment. > how would I use GNU ls within Nushell? To specifically answer this question there's at least 3 possible approaches (including basic parsing of output based on "Parse external ls command and combine columns for datetime" example[0])): # Option 1 (use `^ls`) $ nu --commands '^ls -lb | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB' # Option 2 (use `run-external`) $ nu --commands 'run-external --redirect-stdout "ls" "-lb" | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB' # Option 3 (use `--stdin`) $ ls -lb | nu --stdin --commands 'detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | to nuon' | nu --stdin --commands 'from nuon | where size > 10KiB' ``` You could also use `jc` to parse the ls output to JSON & pipe that into nushell. > I would have to resort to another Nushell-specific helper to integrate the command, which needs to be written in a very generic way to support a wide range of use cases Well, one person would need to write that, once (or just use `jc`). As a matter of interest, how would you do this example task ("display name & size for which of the last 5 files in a directory list are larger than 10KiB", I think) with bash/coreutils command pipeline? (Presumably some combo of `ls`, `cut`, `tail` and others?) [0] https://www.nushell.sh/commands/docs/detect_columns.html#examples https://www.nushell.sh/commands/docs/detect_columns.html#exa...
- imiric 3y ago
  You're right, I'm being a bit harsh towards Nushell, partly because I don't see the problem these tools are trying to fix as particularly significant, and because the legacy Unix design decisions are what helped make it successful today. My point is that an open ecosystem can't prosper if all tools are part of a single project, and they depend on each other to work. There's a good reason why even GNU tools, and even those part of coreutils, are not built with an assumption that they will be part of the same pipeline. All of them are independent, and "do one thing well", and how they integrate into a pipeline is entirely up to the user. All external tools that follow this same principle can generally be composed in the same way. The benefit of this is that the user doesn't depend on a single project, and each tool is easily replaceable by another. Thanks for those Nushell examples. At first glance, they don't look readable or intuitive to me, but maybe it's because you're parsing the output of `ls`, which is generally a bad idea. I only mentioned `ls` as a contrived example, but imagine you have to integrate one or more external commands, and not just at the start of the pipeline, but somewhere in the middle. Doing those conversion steps from/to Nushell would become very cumbersome. I wouldn't use `jc` or any structured exchange format for this task. `find` can do most of the legwork: find -type f -size +10k -printf '%k\t%P\n' | sort -n -k1 | tail -5 This doesn't show the results in a nice table as Nushell would probably do, but IMO it's much simpler, very clear what each tool does, and the pipeline is easily extensible at any point.
mongol 3y ago
> That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem. I don't think it is far fetched to imagine that most popular command line tools will have support for JSON output within let's say 5 years. With that, I think nushell's value proposition becomes a whole lot stronger. Granted, there will never be a time when all tools integrate well, but I can see a critical mass evolve.
- fiddlerwoaroof 3y ago
  I think standardized JSON output makes the value of nushell weaker, not stronger: more structure on stdin and stdout (e.g. switching them to unix sockets instead of fifos and passing FDs around) means the shell composing commands can be less sophisticated. The only way a monolith like nushell adds value is if you compare it to the current standard of unstructured bytes everywhere.
- pmarreck 3y ago
  I absolutely think that treating data piped from or to another process as JSON by default, and treating data going back to the terminal as text (by transforming the JSON to tabular data etc.) is the way to go (also because it's backwards-compatible). To the point that I wanted to write some wrappers for the standard commands that automatically did all this via `jq`. I know commands can detect whether they're being piped to another command or to a terminal, since certain commands will automatically skip ANSI coloring when being piped elsewhere...
  - 8n4vidtmkvmk 3y ago
    Those commands that strip color very often get it wrong too. I have to force color so damn often...
  - follower 3y ago
    > I wanted to write some wrappers for the standard commands that automatically did all this via `jq`. If you're not already aware of it, you may wish to check out `jc`[0] which describes itself as a "CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq..." The `jc` documentation[1] & parser[2] for `ls` also demonstrates that reliable & cross-platform parsing of even "basic" commands can be non-trivial. [0] https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc [1] https://kellyjonbrazil.github.io/jc/docs/parsers/ls https://kellyjonbrazil.github.io/jc/docs/parsers/ls [2] https://github.com/kellyjonbrazil/jc/blob/4cd721be8595db52b620cc26cd455d95bf56b85b/jc/parsers/ls.py#L187 https://github.com/kellyjonbrazil/jc/blob/4cd721be8595db52b6...

by imiric 3y ago

nextaccountic 3y ago

You just can't parse the output of ls and every bash guide will tell you that. This means that ls has no place in shell scripts unfortunately

salohcin 3y ago

I think these are both orthogonal arguments. @p-e-w is arguing for tools that neatly and simply compose workflows, while you're arguing for independence of tools set. Let's explore further. > It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work in the exact way you expect it to. Fair statement, but it's never just one example. It's tonnes of examples that can be made. There are groups of tools whose sole function is just to help stitch other commands together (I.e. getting ls and rm into the same workflow). It works, but the point being made is that it could be better. > You could just as well implement a version of `rm` that parses stdin in a specific way by default, but that would likely make it work for that specific use case, and not in the generic and composable way the tools are meant to be used. I would argue this is _exactly_ what's happened - just that it isn't for only rm, but a whole common subset of tools to bootstrap a new way of working. Your exact argument works still for nushell, if someone wants a new tool for rm (or the legacy one) they could still plausibly do that - and your point above would still hold, it would completely defeat the purpose of nushell, as suddenly _where_ doesn't work. That same argument however applies for current shells... If I suddenly swap out rm, my scripts at some point are going to start failing. Unfortunately we're tightly bound to an environment and ecosystem based on legacy decisions. There are plenty of examples of how that's played out with technology in other ways (for good and bad) - the question should be, is this the right evolution? > That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem. I would point out that no good reason has been given for no contract in existing shells? Or maybe we should observe that there is actually a contract, but it's just a lot less structured. To the point that the contract is just an agreed historical structure that popular individual programs are expected to use. Regardless, a contract isn't a bad thing, especially when it's done correctly to balance flexibility and structure. I'm not convinced nushell is the way to go, but I would prefer the debate not to be muddled. Sure current shells have done fine. We expect future shells to do better. Will they imbue the Unix philosophy perfectly? Well like any good philosophy that will be up for debate.

fiddlerwoaroof 3y ago
> There are groups of tools whose sole function is just to help stitch other commands together This is an example of the Unix philosophy working: replace smarts in the tools with smart compositions of tools; i.e. don’t make programs have to determine whether to read file names from arguments or stdin, use arguments by default and `xargs` to convert stdin into arguments.

qazxcvbnm 3y ago

In that sense, perhaps that facet of capabilities of Nushell would be more comparable to a Foreign-data Wrapper for SQL rather than a shell with pipes.

follower 3y ago

> What Nushell has built instead is a closed ecosystem of tools that interoperate well with each other, but not with the outside environment. IMO "closed ecosystem" feels like a mischaracterization of the project--there's multiple features that are designed to support interoperability outside the nushell environment. > how would I use GNU ls within Nushell? To specifically answer this question there's at least 3 possible approaches (including basic parsing of output based on "Parse external ls command and combine columns for datetime" example[0])): # Option 1 (use `^ls`) $ nu --commands '^ls -lb | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB' # Option 2 (use `run-external`) $ nu --commands 'run-external --redirect-stdout "ls" "-lb" | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB' # Option 3 (use `--stdin`) $ ls -lb | nu --stdin --commands 'detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | to nuon' | nu --stdin --commands 'from nuon | where size > 10KiB' ``` You could also use `jc` to parse the ls output to JSON & pipe that into nushell. > I would have to resort to another Nushell-specific helper to integrate the command, which needs to be written in a very generic way to support a wide range of use cases Well, one person would need to write that, once (or just use `jc`). As a matter of interest, how would you do this example task ("display name & size for which of the last 5 files in a directory list are larger than 10KiB", I think) with bash/coreutils command pipeline? (Presumably some combo of `ls`, `cut`, `tail` and others?) [0] https://www.nushell.sh/commands/docs/detect_columns.html#examples https://www.nushell.sh/commands/docs/detect_columns.html#exa...

imiric 3y ago
You're right, I'm being a bit harsh towards Nushell, partly because I don't see the problem these tools are trying to fix as particularly significant, and because the legacy Unix design decisions are what helped make it successful today. My point is that an open ecosystem can't prosper if all tools are part of a single project, and they depend on each other to work. There's a good reason why even GNU tools, and even those part of coreutils, are not built with an assumption that they will be part of the same pipeline. All of them are independent, and "do one thing well", and how they integrate into a pipeline is entirely up to the user. All external tools that follow this same principle can generally be composed in the same way. The benefit of this is that the user doesn't depend on a single project, and each tool is easily replaceable by another. Thanks for those Nushell examples. At first glance, they don't look readable or intuitive to me, but maybe it's because you're parsing the output of `ls`, which is generally a bad idea. I only mentioned `ls` as a contrived example, but imagine you have to integrate one or more external commands, and not just at the start of the pipeline, but somewhere in the middle. Doing those conversion steps from/to Nushell would become very cumbersome. I wouldn't use `jc` or any structured exchange format for this task. `find` can do most of the legwork: find -type f -size +10k -printf '%k\t%P\n' | sort -n -k1 | tail -5 This doesn't show the results in a nice table as Nushell would probably do, but IMO it's much simpler, very clear what each tool does, and the pipeline is easily extensible at any point.

mongol 3y ago

> That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem. I don't think it is far fetched to imagine that most popular command line tools will have support for JSON output within let's say 5 years. With that, I think nushell's value proposition becomes a whole lot stronger. Granted, there will never be a time when all tools integrate well, but I can see a critical mass evolve.

fiddlerwoaroof 3y ago
I think standardized JSON output makes the value of nushell weaker, not stronger: more structure on stdin and stdout (e.g. switching them to unix sockets instead of fifos and passing FDs around) means the shell composing commands can be less sophisticated. The only way a monolith like nushell adds value is if you compare it to the current standard of unstructured bytes everywhere.
pmarreck 3y ago
I absolutely think that treating data piped from or to another process as JSON by default, and treating data going back to the terminal as text (by transforming the JSON to tabular data etc.) is the way to go (also because it's backwards-compatible). To the point that I wanted to write some wrappers for the standard commands that automatically did all this via `jq`. I know commands can detect whether they're being piped to another command or to a terminal, since certain commands will automatically skip ANSI coloring when being piped elsewhere...
- 8n4vidtmkvmk 3y ago
  Those commands that strip color very often get it wrong too. I have to force color so damn often...
- follower 3y ago
  > I wanted to write some wrappers for the standard commands that automatically did all this via `jq`. If you're not already aware of it, you may wish to check out `jc`[0] which describes itself as a "CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq..." The `jc` documentation[1] & parser[2] for `ls` also demonstrates that reliable & cross-platform parsing of even "basic" commands can be non-trivial. [0] https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc [1] https://kellyjonbrazil.github.io/jc/docs/parsers/ls https://kellyjonbrazil.github.io/jc/docs/parsers/ls [2] https://github.com/kellyjonbrazil/jc/blob/4cd721be8595db52b620cc26cd455d95bf56b85b/jc/parsers/ls.py#L187 https://github.com/kellyjonbrazil/jc/blob/4cd721be8595db52b6...