5 ms·

A +1 from me on something like `--format` - pipe auto-detection feels unnecessary and like an inevitable footgun. As just one example, the Azure CLI defaults t

A +1 from me on something like `--format` - pipe auto-detection feels unnecessary and like an inevitable footgun. As just one example, the Azure CLI defaults to human-readable output, but has an "output" parameter so you can have JSON if you want - I've never once wanted any kind of format auto-detection, and I have to say that I still don't.

hnlmorg 5y ago
The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes. However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes. Take _murex_ for example (disclaimer, I'm the author of that shell): » jobs PID State Background Process Parameters 2104 Executing true exec sleep 9000000 2240 Executing true exec sleep 9000000 It's readable but what if I wanted to pass it as a table? » jobs | cat ["PID","State","Background","Process","Parameters"] [2104,"Executing",true,"exec","sleep 9000000"] [2240,"Executing",true,"exec","sleep 9000000"] ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg » jobs | debug | [[ /Data-Type/Murex ]] jsonl ...but what can we do with a jsonlines table? Well you can select individual columns: » jobs | [ PID State ] [ "PID", "State" ] [ "2104", "Executing" ] [ "2240", "Executing" ] run SQL against it » jobs | select * where PID > 2200 ["PID","State","Background","Process","Parameters"] ["2240","Executing","true","exec","sleep 9000000"] iterate through each row » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } } [2240,"Executing",true,"exec","sleep 9000000"] or even just convert it into another format, like CSV » jobs | format csv PID,State,Background,Process,Parameters 2104,Executing,true,exec,sleep 9000000 2240,Executing,true,exec,sleep 9000000 ...or YAML... » jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000 And it all just works without you having to think or even know what data format is traversing the pipeline. However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.
- preseinger 5y ago
  I don't think typed pipelines in a shell is a virtue. I think dumb bytes are ideal. Text manipulation can be laborious but it's always straightforward.
  - laumars 5y ago
    I used to think that when Powershell was my only exposure typed pipelines. But after using murex I’ve totally changed my opinion. You still have POSIX pipes there for when you want dumb bytes. But you also have typed pipes layered on top to allow one to manage structural data without having to think about it. It’s a bit like having jc integrated into the shell, except that jc supports TOML, YAML, tables of various formats and all sorts - and autodetects the content type too - so you don’t have to learn a dozen different tools for managing a dozen different content types. So it’s the best of both worlds.
  - hnlmorg 5y ago
    It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes. However rather than having to explicitly remember which laborious set of commands work well together, murex will provide you with a suite of builtins that have the same flags and behaviour regardless of the structure of the bytes being read in by them. And if you prefer your pipes to be untyped POSIX byte streams then murex still works here because it will suggest (via tab-completion suggestions) the right commands to use against each previous command.
    - preseinger 5y ago
      > It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes I guess my claim is that the minute you need to parse the dumb bytes as anything other than strings, your problem is complex enough that you shouldn't be using shell to solve it, and you should write a proper program instead. That's not hard and fast, for example I use fish and `math` is very powerful. `jq` is also a bad example, I think, in many ways... it explicitly doesn't follow the Unix philosophy, and instead does it's work by parsing an opaque string according to it's own DSL. Much better is a tool like gron, which allows you to process JSON using familiar tools like grep.
talideon 5y ago
Yup! In fact, there's only one good reason to do automatic pipe detection, and that's if your tool normally outputs ANSI escape codes, which aren't something you want in something going to a pipeline.
- vidarh 5y ago
  Except when you want it. E.g of feeding the output to a pager, or grepping something out and then sending it to the terminal. I hate when tools make this assumption and changes things behind my back.
  - talideon 5y ago
    I said _one_ good reason. If a tool did that, I might be a bit annoyed, but I'd at least understand _why_: I'd be less surprised if the colours and formatting disappeared, but tools like awk, cut, sed, &c., behaved. Ideally, I'd just prefer is libxo was common outside of FreeBSD, and I didn't have to worry about massaging stuff into structured data.
- burntsushi 5y ago
  Do you advocate for ls to change its behavior? Given that it changes its output format based on pipe detection in ways that involve more than just ANSI escape codes. Perhaps this is something that is actually not a "fact," and is something more like an "opinion."
  - talideon 5y ago
    TL;DR: principal of least surprise. Bringing `ls` into this isn't a great idea. Advocate? No. Would I excuse it? Yes. When I write tools, I write two kinds: ones that care intended to be consumed by something and ones that are not. If something is even marginally the former, I assume the former. I can excuse a tool having output changing behaviour if it's the latter, and only in the case of dropping ANSI escape codes, but only because it makes sense to pipe such output into, say `pbcopy` and `tee`, without the escape codes for capture. Automagically removing ANSI escape codes is the _most_ I'd be comfortable with, and only because is _reduces surprise_. Something sent to stdout should be the same as something sent to a pipeline, but I can forgive somebody adding ANSI codes to make things clearer to a human reader, for whom those are invisible in the stream. Would I advocate for `ls` not to send ANSI escape codes to a pipeline? Yes. Do I think it's a good idea to pipe from `ls`? Most certainly not! `ls` is written with assumptions about the consuming terminal that defeat the principal of lease surprise.

hnlmorg 5y ago

The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes. However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes. Take _murex_ for example (disclaimer, I'm the author of that shell): » jobs PID State Background Process Parameters 2104 Executing true exec sleep 9000000 2240 Executing true exec sleep 9000000 It's readable but what if I wanted to pass it as a table? » jobs | cat ["PID","State","Background","Process","Parameters"] [2104,"Executing",true,"exec","sleep 9000000"] [2240,"Executing",true,"exec","sleep 9000000"] ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg » jobs | debug | [[ /Data-Type/Murex ]] jsonl ...but what can we do with a jsonlines table? Well you can select individual columns: » jobs | [ PID State ] [ "PID", "State" ] [ "2104", "Executing" ] [ "2240", "Executing" ] run SQL against it » jobs | select * where PID > 2200 ["PID","State","Background","Process","Parameters"] ["2240","Executing","true","exec","sleep 9000000"] iterate through each row » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } } [2240,"Executing",true,"exec","sleep 9000000"] or even just convert it into another format, like CSV » jobs | format csv PID,State,Background,Process,Parameters 2104,Executing,true,exec,sleep 9000000 2240,Executing,true,exec,sleep 9000000 ...or YAML... » jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000 And it all just works without you having to think or even know what data format is traversing the pipeline. However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.

preseinger 5y ago
I don't think typed pipelines in a shell is a virtue. I think dumb bytes are ideal. Text manipulation can be laborious but it's always straightforward.
- laumars 5y ago
  I used to think that when Powershell was my only exposure typed pipelines. But after using murex I’ve totally changed my opinion. You still have POSIX pipes there for when you want dumb bytes. But you also have typed pipes layered on top to allow one to manage structural data without having to think about it. It’s a bit like having jc integrated into the shell, except that jc supports TOML, YAML, tables of various formats and all sorts - and autodetects the content type too - so you don’t have to learn a dozen different tools for managing a dozen different content types. So it’s the best of both worlds.
- hnlmorg 5y ago
  It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes. However rather than having to explicitly remember which laborious set of commands work well together, murex will provide you with a suite of builtins that have the same flags and behaviour regardless of the structure of the bytes being read in by them. And if you prefer your pipes to be untyped POSIX byte streams then murex still works here because it will suggest (via tab-completion suggestions) the right commands to use against each previous command.
  - preseinger 5y ago
    > It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes I guess my claim is that the minute you need to parse the dumb bytes as anything other than strings, your problem is complex enough that you shouldn't be using shell to solve it, and you should write a proper program instead. That's not hard and fast, for example I use fish and `math` is very powerful. `jq` is also a bad example, I think, in many ways... it explicitly doesn't follow the Unix philosophy, and instead does it's work by parsing an opaque string according to it's own DSL. Much better is a tool like gron, which allows you to process JSON using familiar tools like grep.

talideon 5y ago

Yup! In fact, there's only one good reason to do automatic pipe detection, and that's if your tool normally outputs ANSI escape codes, which aren't something you want in something going to a pipeline.

vidarh 5y ago
Except when you want it. E.g of feeding the output to a pager, or grepping something out and then sending it to the terminal. I hate when tools make this assumption and changes things behind my back.
- talideon 5y ago
  I said _one_ good reason. If a tool did that, I might be a bit annoyed, but I'd at least understand _why_: I'd be less surprised if the colours and formatting disappeared, but tools like awk, cut, sed, &c., behaved. Ideally, I'd just prefer is libxo was common outside of FreeBSD, and I didn't have to worry about massaging stuff into structured data.
burntsushi 5y ago
Do you advocate for ls to change its behavior? Given that it changes its output format based on pipe detection in ways that involve more than just ANSI escape codes. Perhaps this is something that is actually not a "fact," and is something more like an "opinion."
- talideon 5y ago
  TL;DR: principal of least surprise. Bringing `ls` into this isn't a great idea. Advocate? No. Would I excuse it? Yes. When I write tools, I write two kinds: ones that care intended to be consumed by something and ones that are not. If something is even marginally the former, I assume the former. I can excuse a tool having output changing behaviour if it's the latter, and only in the case of dropping ANSI escape codes, but only because it makes sense to pipe such output into, say `pbcopy` and `tee`, without the escape codes for capture. Automagically removing ANSI escape codes is the _most_ I'd be comfortable with, and only because is _reduces surprise_. Something sent to stdout should be the same as something sent to a pipeline, but I can forgive somebody adding ANSI codes to make things clearer to a human reader, for whom those are invisible in the stream. Would I advocate for `ls` not to send ANSI escape codes to a pipeline? Yes. Do I think it's a good idea to pipe from `ls`? Most certainly not! `ls` is written with assumptions about the consuming terminal that defeat the principal of lease surprise.