5 ms·

The problem here is that most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one yo

by EdiX 10y ago

The problem here is that most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. If you want them to only emit data in a standardized format you are implicitly cutting out the user. Since direct use by a human is arguably the main use case, output formatting needs to be reindtroduced somehow. You could make some adapter that you always pipe into that handles most formatting concerns but it's unlikely that it will be truly general. The only general way to do that is for the tool to output, along with the data, also some code that can be used to properly format the data. So there are only three ways to solve this: * Unix solution, with tools doing double duty and each tool having its own incompatible set of flags * Actual programming language solution, tools emit pure data. Great for programming, bad for interactive use. * Powershell solution, programs emit data+code: you are locked into some VM.

skissane 10y ago
Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto). One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C). I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)
- fragmede 10y ago
  Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.) Sadly, it doesn't seem like jq will read an environment setting to always specific -C
TeMPOraL 10y ago
> most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).
- ygra 10y ago
  That's one place where I felt like PowerShell was way more UNIX-y than UNIX commands in that many commands are truly orthogonal.
- cat199 10y ago
  > Which is where they fundamentally break the principle of "doing one thing, and doing it well". unless the 'thing' here is "outputting information in the most generally usable way possible"
marcosdumay 10y ago
You just need to output formating instructions, not code. Read that imagining "formating instructions" as data that will be consumed by a default visualizer that the shell automatically pipes at the end of your command in interactive sessions. I'm sure there's a general architecture for such thing. If no better choice is available, it can be done with plugins.
lobster_johnson 10y ago
There's an easy solution for that, although it requires a richer pipe API. Let each pipe actor, including the terminal/shell, decide what to do. Between each stream, they can rely on a MIME type and additional metadata to make decisions. For example, if you do: generate_data | parse_csv | sort Here, generate_data produces text/csv, parse_csv consumes it and emits application/json or whatever, and sort consumes it. The hypothetical "sort" command wouldn't know what to do with text/csv, so it could default to line-based text. "sort" then emits JSON again, and rather than displaying unfriendly JSON, the terminal/shell combo would see that the final output stream is JSON and invoke a pre-defined presentation handler. For example, there could be a global handler installed that rendered JSON as a table. Or you could insert a different formatter yourself: generate_data | parse_csv | sort | fancy_table_formatter Since fancy_table_formatter emits text (probably some specific terminal MIME type so we can signal that it can include things like ANSI escape codes), the shell doesn't need to do anything except display it.
int_19h 10y ago
> If you want them to only emit data in a standardized format you are implicitly cutting out the user. This is only true if your shell just dumps the raw output as is. But it doesn't have to do that! If we have a standard structured data format for the output, then the shell can be the one to format it. The key point here is that formatting only needs to happen at the very last step of the pipeline, when it is known that the output is about to be displayed to the user. Indeed, this is exactly what PowerShell does, and that bit does not require data + code. It just formats lists of objects into neat tables, using their metadata to generate headers. Note also that this doesn't need to be baked into the shell itself. It can be a separate utility, that sucks in structured data, and outputs formatted user-friendly representation. So in a legacy shell, you could still do: $ ls | fmt and get more or less the same output that you see from ls today (but if you were to do just ls, you'd get JSON). Whereas in a new and fancy shell, you'd get |fmt appended automatically at the end. This, by the way, is also how PowerShell does it, except that fmt is called Out-Default.

by EdiX 10y ago

skissane 10y ago

Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto). One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C). I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)

fragmede 10y ago
Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.) Sadly, it doesn't seem like jq will read an environment setting to always specific -C

TeMPOraL 10y ago

> most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).

ygra 10y ago
That's one place where I felt like PowerShell was way more UNIX-y than UNIX commands in that many commands are truly orthogonal.
cat199 10y ago
> Which is where they fundamentally break the principle of "doing one thing, and doing it well". unless the 'thing' here is "outputting information in the most generally usable way possible"

marcosdumay 10y ago

You just need to output formating instructions, not code. Read that imagining "formating instructions" as data that will be consumed by a default visualizer that the shell automatically pipes at the end of your command in interactive sessions. I'm sure there's a general architecture for such thing. If no better choice is available, it can be done with plugins.

lobster_johnson 10y ago

There's an easy solution for that, although it requires a richer pipe API. Let each pipe actor, including the terminal/shell, decide what to do. Between each stream, they can rely on a MIME type and additional metadata to make decisions. For example, if you do: generate_data | parse_csv | sort Here, generate_data produces text/csv, parse_csv consumes it and emits application/json or whatever, and sort consumes it. The hypothetical "sort" command wouldn't know what to do with text/csv, so it could default to line-based text. "sort" then emits JSON again, and rather than displaying unfriendly JSON, the terminal/shell combo would see that the final output stream is JSON and invoke a pre-defined presentation handler. For example, there could be a global handler installed that rendered JSON as a table. Or you could insert a different formatter yourself: generate_data | parse_csv | sort | fancy_table_formatter Since fancy_table_formatter emits text (probably some specific terminal MIME type so we can signal that it can include things like ANSI escape codes), the shell doesn't need to do anything except display it.

int_19h 10y ago

> If you want them to only emit data in a standardized format you are implicitly cutting out the user. This is only true if your shell just dumps the raw output as is. But it doesn't have to do that! If we have a standard structured data format for the output, then the shell can be the one to format it. The key point here is that formatting only needs to happen at the very last step of the pipeline, when it is known that the output is about to be displayed to the user. Indeed, this is exactly what PowerShell does, and that bit does not require data + code. It just formats lists of objects into neat tables, using their metadata to generate headers. Note also that this doesn't need to be baked into the shell itself. It can be a separate utility, that sucks in structured data, and outputs formatted user-friendly representation. So in a legacy shell, you could still do: $ ls | fmt and get more or less the same output that you see from ls today (but if you were to do just ls, you'd get JSON). Whereas in a new and fancy shell, you'd get |fmt appended automatically at the end. This, by the way, is also how PowerShell does it, except that fmt is called Out-Default.