2 ms·

Part of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority

Part of the problem is that the output of commands is both a UI and an API, and because any text UI can be used as an API, the human readable text gets priority. Shell scripting is therefore kind of like building third party browser extensions. You look and you guess, and then you hack some parser up based on your guess, and hope for the best. I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Then stdout can be for humans, and while you can parse it, you know what you’re doing is fragile. Of course, that’s totally backwards incompatible, and as long as we’re unrealistically reinventing CLIs from the foundations to modernize them, I have a long list of changes I’d make.

theblazehen 3y ago
Have you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/ https://www.nushell.sh/, amongst some others
chongli 3y ago
I really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration. And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script. Seems like a big mess to me. Unless there's something I'm missing?
- kbrazil 3y ago
  There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers. That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine. This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.
- mistercow 3y ago
  > So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout I'm not sure how that's a particular burden. If you have `a | b | c` and you want to debug the output of b, you already have to pull that out and debug it or `tee` it, because its stdout is being piped otherwise. This would be the same, except that you'd pipe (or tee) "machine out" to stdout. Or, since we're making wild backwards incompatible changes anyway, add a directive to the shell that makes it dump all "machine out" to stdout, a la "set +x". Now you don't even have to change your code. Just wrap the line in `set -o dump-machine-out` and `set +o dump-machine-out`
Izkata 3y ago
> I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Except for that last jsonl part, various commands already do something like this with stdout by detecting what's on the other end. https://unix.stackexchange.com/questions/515778/how-does-a-program-know-if-stdout-is-connected-to-a-terminal-or-a-pipe https://unix.stackexchange.com/questions/515778/how-does-a-p...
- mistercow 3y ago
  Yeah, but it's kind of a mess. You have multiple options when implementing it without a clear standard, and in practice, it's a lot trickier to make sure that all of your code does this consistently than it is to simply use different file descriptors for different output types. And then if the user wants to debug their script, they need to know to pipe your command to `cat` or whatever to see what's actually getting passed through.

theblazehen 3y ago

Have you seen some of the existing projects currently working on it? Most well known is https://www.nushell.sh/ https://www.nushell.sh/, amongst some others

chongli 3y ago

I really want to agree because it seems to make so much sense in theory. It gets rid of the need to parse by standardizing on syntax. But it doesn't solve everything, so we still don't get to live the dream. Namely, it does not solve the issue of schemas, versioning, and migration. And this is a really big issue that threatens to derail the whole project. If my script runs a pipeline `a | b | c` and utility b gets updated with a breaking change to the schema, it breaks the entire script. Now I've got to go deep into the weeds to figure out what happened, and the breakage might not be visible in the human-readable output. So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout, and then sit there eyeballing the json to figure out how the schema changed and what I need to do to fix my script. Seems like a big mess to me. Unless there's something I'm missing?

kbrazil 3y ago
There could be a major schema change that breaks the contract, but one of the nice things about JSON output is that it allows the creation of new fields without affecting downstream consumers. That is, if I have a CLI program that spits out a list of IP addresses and one day I want to also output the corresponding dns names, I can simply add the "dns" field and existing pipelines will ignore the field and work just fine. This is better than grep/awking/etc. unstructured text to STDOUT because, depending on how the author decides to add the new field, it can easily break existing pipelines that rely on the shape of the data to stay the same.
mistercow 3y ago
> So to debug I'll have to pass the flag to each tool to get it to print all the json to stdout I'm not sure how that's a particular burden. If you have `a | b | c` and you want to debug the output of b, you already have to pull that out and debug it or `tee` it, because its stdout is being piped otherwise. This would be the same, except that you'd pipe (or tee) "machine out" to stdout. Or, since we're making wild backwards incompatible changes anyway, add a directive to the shell that makes it dump all "machine out" to stdout, a la "set +x". Now you don't even have to change your code. Just wrap the line in `set -o dump-machine-out` and `set +o dump-machine-out`

Izkata 3y ago

> I actually wish there was just a third standard output for machine readable content, which your terminal doesn’t print by default. When you pipe, this output is what gets piped (unless you redirect), it’s expected to be jsonl, and the man page is expected to specify a contract. Except for that last jsonl part, various commands already do something like this with stdout by detecting what's on the other end. https://unix.stackexchange.com/questions/515778/how-does-a-program-know-if-stdout-is-connected-to-a-terminal-or-a-pipe https://unix.stackexchange.com/questions/515778/how-does-a-p...

mistercow 3y ago
Yeah, but it's kind of a mess. You have multiple options when implementing it without a clear standard, and in practice, it's a lot trickier to make sure that all of your code does this consistently than it is to simply use different file descriptors for different output types. And then if the user wants to debug their script, they need to know to pipe your command to `cat` or whatever to see what's actually getting passed through.