3 ms·

The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped)

The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes. However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes. Take _murex_ for example (disclaimer, I'm the author of that shell): » jobs PID State Background Process Parameters 2104 Executing true exec sleep 9000000 2240 Executing true exec sleep 9000000 It's readable but what if I wanted to pass it as a table? » jobs | cat ["PID","State","Background","Process","Parameters"] [2104,"Executing",true,"exec","sleep 9000000"] [2240,"Executing",true,"exec","sleep 9000000"] ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg » jobs | debug | [[ /Data-Type/Murex ]] jsonl ...but what can we do with a jsonlines table? Well you can select individual columns: » jobs | [ PID State ] [ "PID", "State" ] [ "2104", "Executing" ] [ "2240", "Executing" ] run SQL against it » jobs | select * where PID > 2200 ["PID","State","Background","Process","Parameters"] ["2240","Executing","true","exec","sleep 9000000"] iterate through each row » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } } [2240,"Executing",true,"exec","sleep 9000000"] or even just convert it into another format, like CSV » jobs | format csv PID,State,Background,Process,Parameters 2104,Executing,true,exec,sleep 9000000 2240,Executing,true,exec,sleep 9000000 ...or YAML... » jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000 And it all just works without you having to think or even know what data format is traversing the pipeline. However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.

preseinger 5y ago
I don't think typed pipelines in a shell is a virtue. I think dumb bytes are ideal. Text manipulation can be laborious but it's always straightforward.
- laumars 5y ago
  I used to think that when Powershell was my only exposure typed pipelines. But after using murex I’ve totally changed my opinion. You still have POSIX pipes there for when you want dumb bytes. But you also have typed pipes layered on top to allow one to manage structural data without having to think about it. It’s a bit like having jc integrated into the shell, except that jc supports TOML, YAML, tables of various formats and all sorts - and autodetects the content type too - so you don’t have to learn a dozen different tools for managing a dozen different content types. So it’s the best of both worlds.
- hnlmorg 5y ago
  It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes. However rather than having to explicitly remember which laborious set of commands work well together, murex will provide you with a suite of builtins that have the same flags and behaviour regardless of the structure of the bytes being read in by them. And if you prefer your pipes to be untyped POSIX byte streams then murex still works here because it will suggest (via tab-completion suggestions) the right commands to use against each previous command.
  - preseinger 5y ago
    > It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes I guess my claim is that the minute you need to parse the dumb bytes as anything other than strings, your problem is complex enough that you shouldn't be using shell to solve it, and you should write a proper program instead. That's not hard and fast, for example I use fish and `math` is very powerful. `jq` is also a bad example, I think, in many ways... it explicitly doesn't follow the Unix philosophy, and instead does it's work by parsing an opaque string according to it's own DSL. Much better is a tool like gron, which allows you to process JSON using familiar tools like grep.

preseinger 5y ago

I don't think typed pipelines in a shell is a virtue. I think dumb bytes are ideal. Text manipulation can be laborious but it's always straightforward.

laumars 5y ago
I used to think that when Powershell was my only exposure typed pipelines. But after using murex I’ve totally changed my opinion. You still have POSIX pipes there for when you want dumb bytes. But you also have typed pipes layered on top to allow one to manage structural data without having to think about it. It’s a bit like having jc integrated into the shell, except that jc supports TOML, YAML, tables of various formats and all sorts - and autodetects the content type too - so you don’t have to learn a dozen different tools for managing a dozen different content types. So it’s the best of both worlds.
hnlmorg 5y ago
It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes. However rather than having to explicitly remember which laborious set of commands work well together, murex will provide you with a suite of builtins that have the same flags and behaviour regardless of the structure of the bytes being read in by them. And if you prefer your pipes to be untyped POSIX byte streams then murex still works here because it will suggest (via tab-completion suggestions) the right commands to use against each previous command.
- preseinger 5y ago
  > It's only straightforward if you're using tools that understand what those dumb bytes are. Often you'll get away with a mixture of grep, sed and awk but the moment you turn to something like jq you've done away with the assumption that the pipeline is just dumb bytes I guess my claim is that the minute you need to parse the dumb bytes as anything other than strings, your problem is complex enough that you shouldn't be using shell to solve it, and you should write a proper program instead. That's not hard and fast, for example I use fish and `math` is very powerful. `jq` is also a bad example, I think, in many ways... it explicitly doesn't follow the Unix philosophy, and instead does it's work by parsing an opaque string according to it's own DSL. Much better is a tool like gron, which allows you to process JSON using familiar tools like grep.