9 ms·

I've been delving into bash scripting a bit more than I'd like as of late and the lack of universally available consistent structured output for the CLI really

I've been delving into bash scripting a bit more than I'd like as of late and the lack of universally available consistent structured output for the CLI really got to me. Most of the script contents end up being these obfuscating and brittle "hacky solutions" that never should have been necessary. When I thought about pursuing fixing this I felt like the task was a bit overwhelming. I'm delighted that these developers are working on this!

kbenson 7y ago
I've generally found you can get most of what you need done with head, tail, grep, cut, paste, sed and tr (and maybe a couple others I'm missing). Oh, and test (both bash's and coreutil's). It can be a little hacky, but once you realize the crazy output you sometimes have to parse from third party programs, you realize that the ability to handle any free form text is pretty essential. That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).
munchbunny 7y ago
You can also just use PowerShell. Its chief crime is its verbosity in naming, but it has the "universally available consistent structured output for the CLI" you're asking for.
- pitaj 7y ago
  Naw its chief crime is not having `&&`
  - samstave 7y ago
    Im speaking from a position of naivete, so dont attack me; Would an ideal situation be “do + while” Whereas one may write While x is true do “blah blah” && if output is still true then end elif do &&? Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))
    - pitaj 7y ago
      There are ways around it but nothing as easy as `&&` in other shells
  - nailer 7y ago
    pwsh 7 is adding it
- travbrack 7y ago
  Is it actually useful outside Windows? I know the latest version is now portable because of .net core, but I haven't tried it on *nix.
  - felixfbecker 7y ago
    Yeah I'm using it as my primary shell on macOS. You can do everything you do with bash, but with objects, autocompletion etc
    - vips7L 7y ago
      The only thing I cant find in PS is fishlike predictability when using it as a shell.
    - travbrack 7y ago
      Fascinating. Is there a command for getting processes for example or do you need to use ps?
- addicted 7y ago
  I spent a lot of time in Powershell on Windows. Almost 3-4 years. I simply could not use it as a shell. I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable. It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else. Bash felt far more comfortable as a daily shell because what you see is what you get. I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
  - munchbunny 7y ago
    PowerShell is more like interactive Python with very smart tab completion than "bash with objects." Basic conveniences like "grep" require some memorized scripting gymnastics because the commands got generalized a little too much. For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid. The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.
    - hyperpallium 7y ago
      (Disclaimer: never used powershell, so unadulterated opinion here) Scripting is programming, so you want data structures. For interaction, you want human-readable syntax. e.g. programming languages mostly have a syntax: C, python, bash, fortran (exception: lisp), but they operate on data structures. We can see powershell, nushell and other structured shells (e.g. some use XML) as an attempt to "get out of the syntax business", which XML began and JSON finished (https://blog.mlab.com/2011/03/why-is-json-so-popular-developers-want-out-of-the-syntax-business/ https://blog.mlab.com/2011/03/why-is-json-so-popular-develop...). This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.
- jen20 7y ago
  The chief crime of PowerShell is actually how long it takes for a prompt to appear after starting it - especially on Windows. I still use it as a login shell on MacOS because it offers useful features, but it's (to put it bluntly) slow as hell to start.
bigtrakzapzap 7y ago
The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines, necessitating all manner of argument escaping and field delimiters, when pipelines should be streams of typed messages that encapsulate data in typed fields. Logs especially (logging to files is a terrible idea, because it creates log rotation headaches and each program requires a log parser because of the loss of structured information) also. Line-oriented pipeline processing is fundamentally too simple. Settling on a common data simple/universal format robust enough for all purposes, including very large data sets, to exchange between programs and is just complicated enough to eliminate escapement and delimiter headaches without throwing away flexibility (by a new/refined set of processing tools/commands) is key.
- whatshisface 7y ago
  It's not so much a flaw as a tradeoff. Because it's so easy to produce lines of text, virtually every program supports it, even programs written by hackers who aren't interested in following OS communication standards.
  - bigtrakzapzap 7y ago
    No, that's your opinion, not mine. It's a flaw because it creates escaping magic, delimiters that are in-band (spaces) and every program has to know how to parse the output of every other program, wasting human time and processing time. Structured communication, say like an open standard for a schema-less, self-describing, flexible de/serialization like protobufs/msgpack, would be far superior, more usable, more efficient and simple but no too simple to process streams of data with structure and programmability already there. Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
    - 7y ago
j88439h84 7y ago
I agree. The idea of Mario is to use Python objects instead of text lines. This means it's much more structured and has a familiar syntax as well. https://github.com/python-mario/mario https://github.com/python-mario/mario
z3t4 7y ago
Thats when you switch to a real programming language. Relying on output from other programs will always be brittle - as error messages etc will change every new dist release.
yread 7y ago
Yeah just iterating through a directory with filenames with spaces..

kbenson 7y ago

I've generally found you can get most of what you need done with head, tail, grep, cut, paste, sed and tr (and maybe a couple others I'm missing). Oh, and test (both bash's and coreutil's). It can be a little hacky, but once you realize the crazy output you sometimes have to parse from third party programs, you realize that the ability to handle any free form text is pretty essential. That said, since I'm a Perl developer, I'll generally just go straight to perl's inlined implicit loop mode (-p or -n) if I need to use more than a couple of those utils, unless I want it to be portable with a minimum of requirements (as even though more binaries are used in the shell version, they're all essentially guaranteed to exist on a Linux/Unix).

munchbunny 7y ago

You can also just use PowerShell. Its chief crime is its verbosity in naming, but it has the "universally available consistent structured output for the CLI" you're asking for.

pitaj 7y ago
Naw its chief crime is not having `&&`
- samstave 7y ago
  Im speaking from a position of naivete, so dont attack me; Would an ideal situation be “do + while” Whereas one may write While x is true do “blah blah” && if output is still true then end elif do &&? Or is that basically how it already works and we just have shitty interactions with the machines to get them too do the above (unless youre a more seniorbash person ((its one of my career regrets that i didnt spend a ton of time bash scripting)))
  - pitaj 7y ago
    There are ways around it but nothing as easy as `&&` in other shells
- nailer 7y ago
  pwsh 7 is adding it
travbrack 7y ago
Is it actually useful outside Windows? I know the latest version is now portable because of .net core, but I haven't tried it on *nix.
- felixfbecker 7y ago
  Yeah I'm using it as my primary shell on macOS. You can do everything you do with bash, but with objects, autocompletion etc
  - vips7L 7y ago
    The only thing I cant find in PS is fishlike predictability when using it as a shell.
  - travbrack 7y ago
    Fascinating. Is there a command for getting processes for example or do you need to use ps?
addicted 7y ago
I spent a lot of time in Powershell on Windows. Almost 3-4 years. I simply could not use it as a shell. I enjoyed converting our many build scripts into Powershell, but using it as a shell was just uncomfortable. It's not clear to me why. If I had to guess, besides the verbosity, it was the additional "indirection". Nothing was what it appeared on the screen. It was something else. Bash felt far more comfortable as a daily shell because what you see is what you get. I want to try Nu because it appears to combine that WYSIWYG simplicity of bash with the Powershell quality of passing data rather than text. Maybe tables are indeed a better data structure to use than objects.
- munchbunny 7y ago
  PowerShell is more like interactive Python with very smart tab completion than "bash with objects." Basic conveniences like "grep" require some memorized scripting gymnastics because the commands got generalized a little too much. For regular shell usage it's a bit painful because the abstractions are a bit too far removed from the file system you're operating on. But for scripting, such as managing and configuring systems, it's remarkably fluid. The "nothing is as it appears" problem never bothered me much because it's still just data. You just specify the output when you're done crunching it. IMO being able to operate with typed data is really nice because you don't have to worry (nearly as much) about serialization between commands.
  - hyperpallium 7y ago
    (Disclaimer: never used powershell, so unadulterated opinion here) Scripting is programming, so you want data structures. For interaction, you want human-readable syntax. e.g. programming languages mostly have a syntax: C, python, bash, fortran (exception: lisp), but they operate on data structures. We can see powershell, nushell and other structured shells (e.g. some use XML) as an attempt to "get out of the syntax business", which XML began and JSON finished (https://blog.mlab.com/2011/03/why-is-json-so-popular-developers-want-out-of-the-syntax-business/ https://blog.mlab.com/2011/03/why-is-json-so-popular-develop...). This has been taken further into human read/write domains... with mixed reception: ant, XSLT, json as a config format. After all, parsing and "the syntax business" has a practical purpose.
jen20 7y ago
The chief crime of PowerShell is actually how long it takes for a prompt to appear after starting it - especially on Windows. I still use it as a login shell on MacOS because it offers useful features, but it's (to put it bluntly) slow as hell to start.

bigtrakzapzap 7y ago

The key flaw of UNIX philosophy is destructuring deserialization and reserialization based on lines, necessitating all manner of argument escaping and field delimiters, when pipelines should be streams of typed messages that encapsulate data in typed fields. Logs especially (logging to files is a terrible idea, because it creates log rotation headaches and each program requires a log parser because of the loss of structured information) also. Line-oriented pipeline processing is fundamentally too simple. Settling on a common data simple/universal format robust enough for all purposes, including very large data sets, to exchange between programs and is just complicated enough to eliminate escapement and delimiter headaches without throwing away flexibility (by a new/refined set of processing tools/commands) is key.

whatshisface 7y ago
It's not so much a flaw as a tradeoff. Because it's so easy to produce lines of text, virtually every program supports it, even programs written by hackers who aren't interested in following OS communication standards.
- bigtrakzapzap 7y ago
  No, that's your opinion, not mine. It's a flaw because it creates escaping magic, delimiters that are in-band (spaces) and every program has to know how to parse the output of every other program, wasting human time and processing time. Structured communication, say like an open standard for a schema-less, self-describing, flexible de/serialization like protobufs/msgpack, would be far superior, more usable, more efficient and simple but no too simple to process streams of data with structure and programmability already there. Being able to dump structured information out of an exception directly into a log, and then from a log into a database, without any loss of information or extraneous log parsing, is a clear win. Or from a command as simple as listing files ("ls") into a database or into any other tool or program. Outputing only line-oriented strings is just throwing away type information, and creates more work for everyone else, even more so than continuing to stay with lines processing tools.
  - 7y ago

j88439h84 7y ago

I agree. The idea of Mario is to use Python objects instead of text lines. This means it's much more structured and has a familiar syntax as well. https://github.com/python-mario/mario https://github.com/python-mario/mario

z3t4 7y ago

Thats when you switch to a real programming language. Relying on output from other programs will always be brittle - as error messages etc will change every new dist release.

yread 7y ago

Yeah just iterating through a directory with filenames with spaces..