7 ms·

In FreeBSD, this problem was solved with libxo[0]: $ ps --libxo=json | jq { "process-information": { "process": [ {

In FreeBSD, this problem was solved with libxo[0]: $ ps --libxo=json | jq { "process-information": { "process": [ { "pid": "41389", "terminal-name": "0 ", "state": "Is", "cpu-time": "0:00.01", "command": "-bash (bash)" }, [...] It's not perfect though. ls had support, but it was removed for reasons[1]. It's not supported by all of the utilities, etc. This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed. It would be nice if utilities coalesced around a common flag to emit structured output. In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits. [0] https://wiki.freebsd.org/LibXo https://wiki.freebsd.org/LibXo [1] https://reviews.freebsd.org/D13959 https://reviews.freebsd.org/D13959

imtringued 3y ago
Now they only need to do the same thing for input and let the operating system or the shell handle the argument parsing so that it is consistent accross the entire operating system.
supriyo-biswas 3y ago
Similarly, in SerentityOS, stuff under /proc return JSON data rather than unstructured text files. A better, structural way in which this could be fixed is to allow data structures to be exported in ELFs and have those data structures serialized into terminal output, which can then be outputted in the preferred format of the user, such as JSON, YAML, or processed accordingly.
- crotchfire 3y ago
  I mean if you have a filesystem you've already got a way to tree-structure your data...
nijave 3y ago
Libxo is neat, in theory, but it seems like applications are left to implement their own logic for a given output format rather than being able to pass a structure to libxo and let it do the formatting. I can't remember the exact utility--I think it was iostat--would use string interpolation to format output lines in JSON and combined with certains flags produced completely mangled output. Not sure if things have improved but I would have expected something like JSON lines when interval is provided. Powershell and kubectl are miles ahead of libxo in useability imo
- simias 3y ago
  Well I suspect that eventually you just run into hard limitations with C's introspection facilities, or lack thereof. I like C a lot but one of the reasons I like Rust more these days is the ability to trivially implement complex serialization schemes without a ton of ad-hoc code and boilerplate.
- gigatexal 3y ago
  far better for applications to be unaware of a such a utility and allow something like jc to grow in support with plugins or something so as to keep the utilities simple and move the logic and burden to the wrapping utility in this case jc.
evnp 3y ago
> In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits. OP has a blog post[0] which describes exactly this. `jc` is described as a tool to fill this role "in the meantime" -- my reading is that it's intended to serve as a stepping stone towards widespread `-j`/`--json` support across unix tools. [0] https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/ https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph...
- im3w1l 3y ago
  If there is to be a push to add structured data to all the unix tools, I wish they would use a format that allows embedding binary data. Yes base64 is an option, but it suffers from the issue that base64(base64(base64(data))) leads to exponential overhead.
  - pkkm 3y ago
    CBOR? It's based on JSON, so it should be pretty easy to add to a tool that already has JSON serialization.
throw0101b 3y ago
> In FreeBSD, this problem was solved with libxo[0]: Libxo happens to be in the base system, but it is generally available: * https://github.com/Juniper/libxo https://github.com/Juniper/libxo * https://libxo.readthedocs.io/en/latest/ https://libxo.readthedocs.io/en/latest/
ekidd 3y ago
> In PowerShell, structured output is the default and it seems to work very well. PowerShell goes a step beyond JSON, by supporting actual mutable objects. So instead of just passing through structured data, you effectively pass around opaque objects that allow you to go back to earlier pipeline stages, and invoke methods, if I understand correctly: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_methods?view=powershell-7.4 https://learn.microsoft.com/en-us/powershell/module/microsof.... I'm rather fond of wrappers like jc and libxo, and experimental shells like https://www.nushell.sh/ https://www.nushell.sh/. These still focus on passing data, not objects with executable methods. On some level, I find this comfortable: Structured data still feels pretty Unix-like, if that makes sense? If I want actual objects, then it's probably time to fire up Python or Ruby. Knowing when to switch from a shell script to a full-fledged programming language is important, even if your shell is basically awesome and has good programming features.
- lukeschlather 3y ago
  Are executable methods really that bad? I mean, they're bad in some abstract sense but that seems more like an objection if we were talking about a "safe" language like Rust than talking about shell scripting. For a shell executable methods seem fine. If you don't make the method executable people are just going to use eval() anyway, might as well do the more predictable thing.
  - ekidd 3y ago
    It might be possible to design a good Unix shell based on objects, with the ability to "call back into" programs. But I haven't seen one yet that I'd prefer over Ruby or Python. I do think objects make plenty of sense in languages like AppleScript, which essentially allowed users to script running GUI applications. And similarly, Powershell's objects might be right for Windows. But nushell shows how far you can push "dumb" structured data. And it still feels "Unix-like", or at least "alternate universe Unix-like." The other reason I'm suspicious of objects in shells is that shell pilelines are technically async coroutines operating over streams! That's already much further into the world of Haskell or async Rust than many people realize. And so allowing "downstream" portions of a pipeline to call back into "upstream" running programs and to randomly change things introduces all kinds of potential async bugs. If you're going to have a async coroutines operating on streams, then having immutable data is often a good choice. Traditional Unix shells do exactly this. Nushell does it, too, but it replaces plain text with structured data.
msla 3y ago
> It's not perfect though. ls had support, but it was removed for reasons In specific: https://svnweb.freebsd.org/base?view=revision&revision=328100 https://svnweb.freebsd.org/base?view=revision&revision=32810... > libxo imposes a large burden on system utilities. In the case of ls, that burden is difficult to justify -- any language that can interact with json output can use readdir(3) and stat(2). Which rather misses the point of being able to use JSON in shell scripts.
- oh_sigh 3y ago
  I'd love to know what the burden was too. I hear comments like that in code reviews and commonly when you push for specifics about the burden, there is very little
rezonant 3y ago
> This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed True, and yet it's extremely common to parse output in bash scripts and other automations, so in a sense it's just centralizing that effort. That being said at least when you do it yourself you can fix problems directly.
nerdponx 3y ago
What I find weird about Powershell is that there's no "fixed-width column" parser, which is a widely used format for Unix-style CLI tools. I don't know if NuShell has it, I haven't tried. In any case, it's much better for tools to output more-parseable data in the first place. Whitespace-delimited columns are fine of course, but not so much when the data can contain whitespace, as in the output from `ps`. I don't see much reason why JSONLines (https://jsonlines.org/ https://jsonlines.org/) / NDJSON (https://ndjson.org/ https://ndjson.org/) can't be a standard output format from most tools, in addition to tables. As for the reason of removal: any language that can interact with json output can use readdir(3) and stat(2). Ugh. Any language of course can do it. But that's basically telling users that they need to reimplement ls(1) themselves if they want to use any of its output and features in scripts. I understand if the maintenance burden is too high to put it in ls(1) itself, but it's a shame that no tool currently does this. The closest we have is a feature request in Eza: https://github.com/eza-community/eza/issues/472 https://github.com/eza-community/eza/issues/472

imtringued 3y ago

Now they only need to do the same thing for input and let the operating system or the shell handle the argument parsing so that it is consistent accross the entire operating system.

supriyo-biswas 3y ago

Similarly, in SerentityOS, stuff under /proc return JSON data rather than unstructured text files. A better, structural way in which this could be fixed is to allow data structures to be exported in ELFs and have those data structures serialized into terminal output, which can then be outputted in the preferred format of the user, such as JSON, YAML, or processed accordingly.

crotchfire 3y ago
I mean if you have a filesystem you've already got a way to tree-structure your data...

nijave 3y ago

Libxo is neat, in theory, but it seems like applications are left to implement their own logic for a given output format rather than being able to pass a structure to libxo and let it do the formatting. I can't remember the exact utility--I think it was iostat--would use string interpolation to format output lines in JSON and combined with certains flags produced completely mangled output. Not sure if things have improved but I would have expected something like JSON lines when interval is provided. Powershell and kubectl are miles ahead of libxo in useability imo

simias 3y ago
Well I suspect that eventually you just run into hard limitations with C's introspection facilities, or lack thereof. I like C a lot but one of the reasons I like Rust more these days is the ability to trivially implement complex serialization schemes without a ton of ad-hoc code and boilerplate.
gigatexal 3y ago
far better for applications to be unaware of a such a utility and allow something like jc to grow in support with plugins or something so as to keep the utilities simple and move the logic and burden to the wrapping utility in this case jc.

evnp 3y ago

> In PowerShell, structured output is the default and it seems to work very well. This is probably too far for Unix/Linux, but a standard "--json" flag would go a long way to getting the same benefits. OP has a blog post[0] which describes exactly this. `jc` is described as a tool to fill this role "in the meantime" -- my reading is that it's intended to serve as a stepping stone towards widespread `-j`/`--json` support across unix tools. [0] https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/ https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph...

im3w1l 3y ago
If there is to be a push to add structured data to all the unix tools, I wish they would use a format that allows embedding binary data. Yes base64 is an option, but it suffers from the issue that base64(base64(base64(data))) leads to exponential overhead.
- pkkm 3y ago
  CBOR? It's based on JSON, so it should be pretty easy to add to a tool that already has JSON serialization.

throw0101b 3y ago

> In FreeBSD, this problem was solved with libxo[0]: Libxo happens to be in the base system, but it is generally available: * https://github.com/Juniper/libxo https://github.com/Juniper/libxo * https://libxo.readthedocs.io/en/latest/ https://libxo.readthedocs.io/en/latest/

ekidd 3y ago

> In PowerShell, structured output is the default and it seems to work very well. PowerShell goes a step beyond JSON, by supporting actual mutable objects. So instead of just passing through structured data, you effectively pass around opaque objects that allow you to go back to earlier pipeline stages, and invoke methods, if I understand correctly: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_methods?view=powershell-7.4 https://learn.microsoft.com/en-us/powershell/module/microsof.... I'm rather fond of wrappers like jc and libxo, and experimental shells like https://www.nushell.sh/ https://www.nushell.sh/. These still focus on passing data, not objects with executable methods. On some level, I find this comfortable: Structured data still feels pretty Unix-like, if that makes sense? If I want actual objects, then it's probably time to fire up Python or Ruby. Knowing when to switch from a shell script to a full-fledged programming language is important, even if your shell is basically awesome and has good programming features.

lukeschlather 3y ago
Are executable methods really that bad? I mean, they're bad in some abstract sense but that seems more like an objection if we were talking about a "safe" language like Rust than talking about shell scripting. For a shell executable methods seem fine. If you don't make the method executable people are just going to use eval() anyway, might as well do the more predictable thing.
- ekidd 3y ago
  It might be possible to design a good Unix shell based on objects, with the ability to "call back into" programs. But I haven't seen one yet that I'd prefer over Ruby or Python. I do think objects make plenty of sense in languages like AppleScript, which essentially allowed users to script running GUI applications. And similarly, Powershell's objects might be right for Windows. But nushell shows how far you can push "dumb" structured data. And it still feels "Unix-like", or at least "alternate universe Unix-like." The other reason I'm suspicious of objects in shells is that shell pilelines are technically async coroutines operating over streams! That's already much further into the world of Haskell or async Rust than many people realize. And so allowing "downstream" portions of a pipeline to call back into "upstream" running programs and to randomly change things introduces all kinds of potential async bugs. If you're going to have a async coroutines operating on streams, then having immutable data is often a good choice. Traditional Unix shells do exactly this. Nushell does it, too, but it replaces plain text with structured data.

msla 3y ago

> It's not perfect though. ls had support, but it was removed for reasons In specific: https://svnweb.freebsd.org/base?view=revision&revision=328100 https://svnweb.freebsd.org/base?view=revision&revision=32810... > libxo imposes a large burden on system utilities. In the case of ls, that burden is difficult to justify -- any language that can interact with json output can use readdir(3) and stat(2). Which rather misses the point of being able to use JSON in shell scripts.

oh_sigh 3y ago
I'd love to know what the burden was too. I hear comments like that in code reviews and commonly when you push for specifics about the burden, there is very little

rezonant 3y ago

> This seems to be a great stop-gap with parsers for a LOT of different commands, but it relies on parsing text output that's not necessarily designed to be parsed True, and yet it's extremely common to parse output in bash scripts and other automations, so in a sense it's just centralizing that effort. That being said at least when you do it yourself you can fix problems directly.

nerdponx 3y ago

What I find weird about Powershell is that there's no "fixed-width column" parser, which is a widely used format for Unix-style CLI tools. I don't know if NuShell has it, I haven't tried. In any case, it's much better for tools to output more-parseable data in the first place. Whitespace-delimited columns are fine of course, but not so much when the data can contain whitespace, as in the output from `ps`. I don't see much reason why JSONLines (https://jsonlines.org/ https://jsonlines.org/) / NDJSON (https://ndjson.org/ https://ndjson.org/) can't be a standard output format from most tools, in addition to tables. As for the reason of removal: any language that can interact with json output can use readdir(3) and stat(2). Ugh. Any language of course can do it. But that's basically telling users that they need to reimplement ls(1) themselves if they want to use any of its output and features in scripts. I understand if the maintenance burden is too high to put it in ls(1) itself, but it's a shame that no tool currently does this. The closest we have is a feature request in Eza: https://github.com/eza-community/eza/issues/472 https://github.com/eza-community/eza/issues/472