4 ms·

But why? Just for people to gron it back into usable form? Unix tools work best with line-based data. Using some "structured" monstrosity like xml or json fo

But why? Just for people to gron it back into usable form? Unix tools work best with line-based data. Using some "structured" monstrosity like xml or json forces all other tools to deal with this particular format, thus breaking the orthogonality between programs.

simonw 3y ago
Newline-delimited JSON is so much more useful to me than weird Unix line-formatted output that I have to parse with pattern matching or regular expressions. It's basically a line-based format that can represent all of the JSON types and includes support for nested data structures where necessary. What's not to like?
- qwertox 3y ago
  It's also supported by simdjson [0] (which has a lot of language bindings [1]): > Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s [0] https://simdjson.org/ https://simdjson.org/ [1] https://github.com/simdjson/simdjson?tab=readme-ov-file#bindings-and-ports-of-simdjson https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...
- bsdetector 3y ago
  JSON is not immediately usable by and is cumbersome to parse correctly in a shell. A simple line-based shell variable name=value format works unreasonably well. For example: # ls --shell-var ./thefile dir="/home/user" file="thefile" size=1234 ... # eval $(ls --shell-var ./thefile); echo $size 1234 If this had been in shells and cmdline tools since the beginning it would have saved so much work, and the security problems could have been dealt with by an eval that only set variables, adding a prefix/scope to variables, and so on. Unfortunately it's too late for this and today you'll be using a pipeline to make the json output shell friendly or use some substring hacks that probably work most of the time.
  - kitd 3y ago
    That's not your original request though, to use line-based data. It seems you're determined not to use jq but if anything, json output | jq is more the unix way than piping everything through shell vars.
    - mlhpdx 3y ago
      And, the author isn’t suggesting only having JSON output, but adding it as an option for those of use that would make use of it. The plain text should remain as well (and has to or many, many things would break). On a separate point, I find the JSON much easier to reason about. The wall of text output doesn’t work for my brain - I just can’t see it all. Structuring/nesting with clear delineations makes it far easier for me to grok.
    - bsdetector 3y ago
      > That's not your original request though, to use line-based data. It wasn't my request and OP (not me) said "line-based data" is best. The comment I replied to said "Newline-delimited JSON ... a line-based format". If the only objection you have is "but that's line-based!" then you're in a completely different conversation. > if anything, json output | jq is more the unix way than piping everything through shell vars. The unix way is line-based. The comment I replied to is talking about line-based output. Line-based output is the only structure for data universal to unix cmdline tools - even tab/space isn't universal; sending structured non-line-delimited data to a program to unpack it is the least unix-like way to do it. Also there's no pipe in the shell-variable output scheme I described, whereas "json | jq" is a shell pipeline.
  - simonw 3y ago
    I use jq - which ChatGPT knows inside out, so I can generally get exactly what I want from it with a single prompt.
  - starttoaster 3y ago
    That's great for key=value data, but more complex data structures don't work so well in that format, JSON does. "Why would you need to represent data as a complex data structure?" Sometimes attributes are owned by a specific entity, and that entity might own multiple attributes. It might even own other sub-entities. JSON represents that. Key=value does not.
deleted 3y ago
[deleted]
eternityforest 3y ago
If you're using UNIX tools to parse it, it sucks, but generally if I'm reading the output of a command, and the command is more than one word, I'm doing the whole thing in Python. That's real programming, and for that I want type checkers and debuggers and modern syntax and all that. And the performance is often faster because you're not spinning up subprocesses for each command.
pluto_modadic 3y ago
./some_command | jq '.memory_use' vs: okay, run it once with head (assuming we have a --dry option...) ah, that's the column. okay cut -d"," -f0, ah whoops it's starting at 1. ah, damn, there's a weird comma in the name/quote. oh weird, that one's null, ah heck. schemas are cool. JSON extends and builds on the UNIX idea of having things pipe and plumb well together.
- alerighi 3y ago
  Plain text is more easy to reason about, because we are used to process text. A good textual output, that is records delimited by spaces, tabs or a delimiter, to me is all it's needed, for most applications. An object structure it's much more complex to use. For example an output that is a set of records can be easily imported in an Excel sheet, in an SQL database, processed line by line, without issues. Processing JSON is not straight forward, not all programs support JSON. Finally JSON can't be processed as a stream, meaning that tools like head, tail, etc. doesn't work on JSON, you have to read it all in memory, or use JSON lines, that is not a standard format, that not all parsers support natively, etc. JSON is good if for integrating the program inside other programs (as a subprocess), so having an option to input/output JSON in a program is useful, but to me it's not as useful for interactive shell usage. I prefer to use UNIX tools such as grep, cut, head, tail, etc.
- mlhpdx 3y ago
  But as the article points out, the advice to use unbuffered JSON lines for commands that are line oriented is well given. Not doing that can really make life sad.
- dylan604 3y ago
  you still need to know the schema of the JSON which could also use a dry run as well. not really sure how just because it's JSON solves that in your mind
  - Jcowell 3y ago
    Orient the idea be that because it’s json there’s a schema somewhere the end user can refer to?
placatedmayhem 3y ago
Line-oriented formats, like most traditional Unix-style tools, are for human consumption. JSON is bad at that, thus gron. On the other hand, structured output formats, like JSON, make it easier to consume with other programs. Standard formats have readily-available and commonly used libraries, whereas line parsing tends to be one-off for every program. Whether JSON is the best format for this is certainly debatable, but it is quite ubiquitous, which is a huge advantage. I doubt many folks would propose XML as a general recommendation. Tools should have both options on their path to maturity -- both human-consumable and computer-consumable output format options.
janderland 3y ago
I’d rather use JMES Path (or jq even tho it’s messier) to restructure my data rather than some mixture of awk, sed, cut, etc. Line output often needs restructuring in my experience, though JSON will always need it.
Spivak 3y ago
Unix tools on newline delineated list like objects work great, but try them on dict like objects and it becomes clunky really fast. Parsing `ip` output is a good example where the data naturally lends itself to iface:attrs. Plucking the value you want by `.[.name | startswith('eth')].addr` is way easier than pulling this out with grep/awk.
- BenjiWiebe 3y ago
  You probably already know this but just in case: You can specify an interface when using ip, e.g. ip addr show dev enp2s0
__MatrixMan__ 3y ago
The caller can just specify the output format that they want, and ever since I switched to nushell I pretty much always want json.
deleted 3y ago
[deleted]
keybored 3y ago
What orthogonality? “Line-based data” that varies randomly from tool to tool? Using json between programs is perfectly orthogonal.
IshKebab 3y ago
Nonsense. Using a structured format means all tools are using the same format, instead of every tool making up their own (usually broken) system. Look at how many flags `ls` has to feed filenames into other programs without breaking. Also, using a proper format like JSON makes it actually robust. Most ad hoc pipelines break if you so much as put a space in a filename.

simonw 3y ago

Newline-delimited JSON is so much more useful to me than weird Unix line-formatted output that I have to parse with pattern matching or regular expressions. It's basically a line-based format that can represent all of the JSON types and includes support for nested data structures where necessary. What's not to like?

qwertox 3y ago
It's also supported by simdjson [0] (which has a lot of language bindings [1]): > Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s [0] https://simdjson.org/ https://simdjson.org/ [1] https://github.com/simdjson/simdjson?tab=readme-ov-file#bindings-and-ports-of-simdjson https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...
bsdetector 3y ago
JSON is not immediately usable by and is cumbersome to parse correctly in a shell. A simple line-based shell variable name=value format works unreasonably well. For example: # ls --shell-var ./thefile dir="/home/user" file="thefile" size=1234 ... # eval $(ls --shell-var ./thefile); echo $size 1234 If this had been in shells and cmdline tools since the beginning it would have saved so much work, and the security problems could have been dealt with by an eval that only set variables, adding a prefix/scope to variables, and so on. Unfortunately it's too late for this and today you'll be using a pipeline to make the json output shell friendly or use some substring hacks that probably work most of the time.
- kitd 3y ago
  That's not your original request though, to use line-based data. It seems you're determined not to use jq but if anything, json output | jq is more the unix way than piping everything through shell vars.
  - mlhpdx 3y ago
    And, the author isn’t suggesting only having JSON output, but adding it as an option for those of use that would make use of it. The plain text should remain as well (and has to or many, many things would break). On a separate point, I find the JSON much easier to reason about. The wall of text output doesn’t work for my brain - I just can’t see it all. Structuring/nesting with clear delineations makes it far easier for me to grok.
  - bsdetector 3y ago
    > That's not your original request though, to use line-based data. It wasn't my request and OP (not me) said "line-based data" is best. The comment I replied to said "Newline-delimited JSON ... a line-based format". If the only objection you have is "but that's line-based!" then you're in a completely different conversation. > if anything, json output | jq is more the unix way than piping everything through shell vars. The unix way is line-based. The comment I replied to is talking about line-based output. Line-based output is the only structure for data universal to unix cmdline tools - even tab/space isn't universal; sending structured non-line-delimited data to a program to unpack it is the least unix-like way to do it. Also there's no pipe in the shell-variable output scheme I described, whereas "json | jq" is a shell pipeline.
- simonw 3y ago
  I use jq - which ChatGPT knows inside out, so I can generally get exactly what I want from it with a single prompt.
- starttoaster 3y ago
  That's great for key=value data, but more complex data structures don't work so well in that format, JSON does. "Why would you need to represent data as a complex data structure?" Sometimes attributes are owned by a specific entity, and that entity might own multiple attributes. It might even own other sub-entities. JSON represents that. Key=value does not.

deleted 3y ago

[deleted]

eternityforest 3y ago

If you're using UNIX tools to parse it, it sucks, but generally if I'm reading the output of a command, and the command is more than one word, I'm doing the whole thing in Python. That's real programming, and for that I want type checkers and debuggers and modern syntax and all that. And the performance is often faster because you're not spinning up subprocesses for each command.

pluto_modadic 3y ago

./some_command | jq '.memory_use' vs: okay, run it once with head (assuming we have a --dry option...) ah, that's the column. okay cut -d"," -f0, ah whoops it's starting at 1. ah, damn, there's a weird comma in the name/quote. oh weird, that one's null, ah heck. schemas are cool. JSON extends and builds on the UNIX idea of having things pipe and plumb well together.

alerighi 3y ago
Plain text is more easy to reason about, because we are used to process text. A good textual output, that is records delimited by spaces, tabs or a delimiter, to me is all it's needed, for most applications. An object structure it's much more complex to use. For example an output that is a set of records can be easily imported in an Excel sheet, in an SQL database, processed line by line, without issues. Processing JSON is not straight forward, not all programs support JSON. Finally JSON can't be processed as a stream, meaning that tools like head, tail, etc. doesn't work on JSON, you have to read it all in memory, or use JSON lines, that is not a standard format, that not all parsers support natively, etc. JSON is good if for integrating the program inside other programs (as a subprocess), so having an option to input/output JSON in a program is useful, but to me it's not as useful for interactive shell usage. I prefer to use UNIX tools such as grep, cut, head, tail, etc.
mlhpdx 3y ago
But as the article points out, the advice to use unbuffered JSON lines for commands that are line oriented is well given. Not doing that can really make life sad.
dylan604 3y ago
you still need to know the schema of the JSON which could also use a dry run as well. not really sure how just because it's JSON solves that in your mind
- Jcowell 3y ago
  Orient the idea be that because it’s json there’s a schema somewhere the end user can refer to?

placatedmayhem 3y ago

Line-oriented formats, like most traditional Unix-style tools, are for human consumption. JSON is bad at that, thus gron. On the other hand, structured output formats, like JSON, make it easier to consume with other programs. Standard formats have readily-available and commonly used libraries, whereas line parsing tends to be one-off for every program. Whether JSON is the best format for this is certainly debatable, but it is quite ubiquitous, which is a huge advantage. I doubt many folks would propose XML as a general recommendation. Tools should have both options on their path to maturity -- both human-consumable and computer-consumable output format options.

janderland 3y ago

I’d rather use JMES Path (or jq even tho it’s messier) to restructure my data rather than some mixture of awk, sed, cut, etc. Line output often needs restructuring in my experience, though JSON will always need it.

Spivak 3y ago

Unix tools on newline delineated list like objects work great, but try them on dict like objects and it becomes clunky really fast. Parsing `ip` output is a good example where the data naturally lends itself to iface:attrs. Plucking the value you want by `.[.name | startswith('eth')].addr` is way easier than pulling this out with grep/awk.

BenjiWiebe 3y ago
You probably already know this but just in case: You can specify an interface when using ip, e.g. ip addr show dev enp2s0

__MatrixMan__ 3y ago

The caller can just specify the output format that they want, and ever since I switched to nushell I pretty much always want json.

deleted 3y ago

[deleted]

keybored 3y ago

What orthogonality? “Line-based data” that varies randomly from tool to tool? Using json between programs is perfectly orthogonal.

IshKebab 3y ago

Nonsense. Using a structured format means all tools are using the same format, instead of every tool making up their own (usually broken) system. Look at how many flags `ls` has to feed filenames into other programs without breaking. Also, using a proper format like JSON makes it actually robust. Most ad hoc pipelines break if you so much as put a space in a filename.