10 ms·

Tips on adding JSON output to your CLI app

183 points by kbrazil 5y ago

b3morales 5y ago
Another DON'T, silently switching between JSON and human-readable depending on whether the output destination is a pipe. Just an extra hassle when I'm writing my downstream command. Or could be phrased as a DO: give the user a switch to pick the output format, if you have both.
- GauntletWizard 5y ago
  I started to try to make an argument for pipe autodetection, but I just can't. It seems like a useful feature but is actually a trap. Shell scripts that are going to rely on json output should always explicitly specify that they want json output, and giving them the ability to shortcut that by autodetecting a pipe will only make it easy to ignore that - And then break when some other format comes into vogue. Human readable output should generally be the default, unless the programs are explicitly designed as only part of a pipeline. Having multiple outputs is a great feature, though. I'm especially fond of tooling in Kubernetes that allows you to nicely pipe things in and out in multiple formats.
- strictfp 5y ago
  Pretty much every linux tool pulls these shenanigans. I hate it when there's no flag to control output.
- lytedev 5y ago
  What about things like ANSI color codes?
  - MereInterest 5y ago
    My preference would be to leave them in if they're the default, but to have an option to switch entirely to a more machine-readable format. It's a single line of sed to strip them out[0], and it's a bigger pain to figure out why a program has different behavior when debugging. [0] https://superuser.com/a/380778 https://superuser.com/a/380778
    - kbrazil 5y ago
      I typically run the command output through `cat` to see if it does different things when outputting to the terminal vs piping. The biggest changes are typically ANSI color codes and column length. Some programs do strange things, like changing how they escape characters (ls). JC turns off ANSI color codes when it’s output is not a terminal, but that’s it.
      - laumars 5y ago
        Changes in column length isn’t something that can’t be helped. The issue is that when a command outputs to a TTY, you can use that file descriptor to get the width of the TTY (ie your terminal emulator). But if you are piping a command then it’s STDOUT is no longer a TTY and thus you have no idea how wide the terminal is.
  - foodstances 5y ago
    Hope it supports NO_COLOR (http://no-color.org/ http://no-color.org/) env NO_COLOR=1 ...
    - kbrazil 5y ago
      I hadn’t seen this before - thanks for sharing!
  - b3morales 5y ago
    Good question; those are more reasonable to me since they aren't visible as characters and don't change the structure. So if I'm looking at the colorized output I can still use that as the basis for a sed or awk script operating on the non-colorized version.
    - b3morales 5y ago
      Oh, and I still want `--color=always` or equivalent for when I'm piping into a pager.
lunfard000 5y ago
come to the powershell-side, we have ConvertFrom-Json
umvi 5y ago
In addition to an option for writing output as JSON, consider also adding an option for streaming output to stdout. Those two features were added to GCC9 gcov and are what enabled me to write a tool that parallelizes coverage report generation. In practice this enabled generating coverage reports orders of magnitude faster than traditional gcov wrappers like lcov
enriquto 5y ago
If you ever need to fight against this annoying json trend (e.g., when your tool only emits certain information in stupid ungrepable json), consider filtering the output through a gron so that it becomes saner.
- dylan604 5y ago
  why is JSON ungrepable? grep key file.json | awk -F: '{print $2}' if you're already searching for a key, seems like you're just wanting the value. granted, i hardly ever (have i actually ever??) interact with JSON this way, so i'm not exactly familiar with pitfalls.
  - b3morales 5y ago
    When it's not formatted, just emitted as a single line. (Granted grep still works, but...not nicely.)
    - dylan604 5y ago
      gotcha. i'm imagining a pretty gnarly regex to return the data after the first colon after the match but only up until the next comma or square bracket or curly bracket. yeah, that's unpleasant. thankfully, there are tools like jq to help maintain one's sanity
      - akyoan 5y ago
        To be honest I find it funny that anyone would attempt to “parse” such a simple format with the wrong tool when there are plenty that can extract data in dot notation.
  - fragmede 5y ago
    The pitfall is that JSON has zero guarantees for how often line breaks do and don't occur, and is often used to represent hierarchical data. Grepping for 'key: foo', and some liberal use of -A and -B may find you what you're looking for, but grep is simply the wrong tool for that job. (And how do you handle a key with newlines in it?) jq [0] is the right tool, but jq's syntax is it's own, and is harder to use (unless you use it regularly). [0] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
wruza 5y ago
CLI and JSON would be amazing if terminals made a step forward too. Because both raw json and triangled in-browser console json logging just suck for daily reading. A new terminal could either detect patterns or use explicit cues in json to format structures, and show raw data on demand. E.g. this json5: [{_repr:"ls:file", name:"foo.txt", type:"text/plain", size:512, access:"664", …}] could be presented as usual ls does, but processed further as json. A whole lot of representations (and editors) – including ui-based – could be added to the system (e.g. /usr/local/share/repr/ls:file (+x)) to format any sort of data, instead of formatting it in-program with getopt and printf. And when there is no repr file, well, you still have triangles mode. We’re too stuck with text=text=text idiom. Structure=text=ui would be so much better. (I’m aware of powershell and am ignoring it consciously)
- cwalv 5y ago
  This kindof reminds me of the ipython display protocol https://carreau.github.io/posts/29-JupyterCon-DisplayProtocol.ipynb/ https://carreau.github.io/posts/29-JupyterCon-DisplayProtoco...
- laumars 5y ago
  Such a shell already exists https://github.com/lmorg/murex https://github.com/lmorg/murex You’d have to learn a new shell syntax but at least it’s compatible with existing CLI tools (which Powershell isn’t)
  - wruza 5y ago
    I was talking about terminal [emulators], not a shell. A shell has nothing to do with how the output stream is displayed. Murex is more like powershell in this regard, which is on/around the level of implementation that I personally find inappropriate.
    - laumars 5y ago
      I know you said terminal but short of rewriting the entire stack, TTYs and all, you have to work with what you’ve got. With murex, it does actually have a lot of intelligence built in that adapts how the output stream is displayed (eg rendering images in the terminal, “prettifying” JSON if STDOUT is a TTY, colourising STDERR red, etc. Plus I disagree that your GP points about how JSON is handled should be a shell thing. You’re talking about data being re-encoded in different formats depending on the output. That absolutely should be a shell thing (where the logic of the pipelines are handled) with the terminal being a dumb rendering client. The last thing I want is output to be modified by the terminal leaving me scratching my head as to whether a command is running correctly or whether the terminal is displaying it weirdly.
throw0101a 5y ago
See also libxo: > The libxo library allows an application to generate text, XML, JSON, and HTML output using a common set of function calls. The application decides at run time which output style should be produced. The application calls a function "xo_emit" to product output that is described in a format string. A "field descriptor" tells libxo what the field is and what it means. * https://github.com/Juniper/libxo https://github.com/Juniper/libxo Then add an "--output-format" option.
- GordonS 5y ago
  A +1 from me on something like `--format` - pipe auto-detection feels unnecessary and like an inevitable footgun. As just one example, the Azure CLI defaults to human-readable output, but has an "output" parameter so you can have JSON if you want - I've never once wanted any kind of format auto-detection, and I have to say that I still don't.
  - hnlmorg 5y ago
    The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes. However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes. Take _murex_ for example (disclaimer, I'm the author of that shell): » jobs PID State Background Process Parameters 2104 Executing true exec sleep 9000000 2240 Executing true exec sleep 9000000 It's readable but what if I wanted to pass it as a table? » jobs | cat ["PID","State","Background","Process","Parameters"] [2104,"Executing",true,"exec","sleep 9000000"] [2240,"Executing",true,"exec","sleep 9000000"] ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg » jobs | debug | [[ /Data-Type/Murex ]] jsonl ...but what can we do with a jsonlines table? Well you can select individual columns: » jobs | [ PID State ] [ "PID", "State" ] [ "2104", "Executing" ] [ "2240", "Executing" ] run SQL against it » jobs | select * where PID > 2200 ["PID","State","Background","Process","Parameters"] ["2240","Executing","true","exec","sleep 9000000"] iterate through each row » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } } [2240,"Executing",true,"exec","sleep 9000000"] or even just convert it into another format, like CSV » jobs | format csv PID,State,Background,Process,Parameters 2104,Executing,true,exec,sleep 9000000 2240,Executing,true,exec,sleep 9000000 ...or YAML... » jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000 And it all just works without you having to think or even know what data format is traversing the pipeline. However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.
simonw 5y ago
I hadn't seen jc before (by the author of this piece: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc ) - what a great idea! It has parsers for around 80 different classic Unix utilities such that it can convert their output to JSON. ~ % dig example.com | jc --dig | jq [ { "id": 61315, "opcode": "QUERY", "status": "NOERROR", "flags": [ "qr", "rd", "ra" ], "query_num": 1, "answer_num": 1, "authority_num": 0, "additional_num": 1, "opt_pseudosection": { "edns": { "version": 0, "flags": [], "udp": 512 } }, "question": { "name": "example.com.", "class": "IN", "type": "A" }, "answer": [ { "name": "example.com.", "class": "IN", "type": "A", "ttl": 85586, "data": "93.184.216.34" } ], "query_time": 29, "server": "10.0.0.1#53(10.0.0.1)", "when": "Sun Dec 05 15:12:08 PST 2021", "rcvd": 56, "when_epoch": 1638745928, "when_epoch_utc": null } ]
- EdwardDiego 5y ago
  Oh Christ yes, it supports lsof! The output from lsof has always been hard to script up.
- aasasd 5y ago
  Turns out I can cross out a todo item off my list, because I wanted to make exactly the same thing. Now, to find a query tool with a saner language than Jq...
  - ZeroGravitas 5y ago
    What kind of thing are you trying to do? jq can get pretty deep but for most things in this area I'm not sure how it could improve upon, but would be interested in hearing alternatives. https://github.com/fiatjaf/jiq https://github.com/fiatjaf/jiq Is a realtime feedback wrapper which I find useful when crafting one-off command line uses for jq and it starts getting crazy.
1vuio0pswjnm7 5y ago
Honestly I do not really understand this idea because, AFAIK, JSON was designed for Javascript in a web browser. By and large, Javascript-enabled web browser expect access to generous amounts of memory. This is not the case for the common UNIX userland programs. These programs do not expect large amounts of memory and many are written with the intent that they may be used to process text line-by-line. This JSON idea reminds me of Windows "PowerShell". Microsoft actually has to limit how much memory can be used by the shell. Why is that. https://devblogs.microsoft.com/scripting/learn-how-to-configure-powershell-memory/ https://devblogs.microsoft.com/scripting/learn-how-to-config... One of the things I like most about the UNIX userland is that I can use small programs to edit vary large files, without needing lots of memory. I want programs that are designed to accomodate the possibility of line-by-line processing. If the intent is to make output network friendly, maybe something like netstrings is useful. Easy to parse. Low memory footprint. Seems to me this JSON idea is not designed to improve performance, agility or resource efficiency but to ignore the UNIX example in favour of a different, slower, approach that is perceived as easier for some people to use. Namely those who do not want to spend the time to learn how to use an existing, faster solution with lower resource requirements.
- wruza 5y ago
  JSONL is the answer for arrays. “[…50mb…]” is too big to be processed in a streaming mode, but particular items usually cannot be split anyway, and “{…}\n50mb more” is what you need.
- tester756 5y ago
  for me it is: readable, handy since probably every language has libs that work with it fine, there's a lot of tools that work with jsons e.g generating code classes from json, it's insanely popular, really easy to learn
- kbrazil 5y ago
  In actuality most CLI output is quite small and can easily be represented as a single JSON document. There are a few commands that can produce huge amounts of output and those are good candidates for using JSON Lines as noted in the article. JC has streaming parsers that lazily output JSON lines for these types of commands. (ls, ping, vmstat, iostat, etc.)
nyuszika7h 5y ago
I don't really understand the point about flattening. > This way I can easily filter the data in jq or other tools without having to traverse levels. How is doing `jq '.cpu.speed'` any harder than doing `jq '.cpu_speed'`? IMO as long as you aren't going insane with nesting levels, it's actually better to have a proper structure than dumping everything into an ugly flat object.
- kbrazil 5y ago
  The article is not advocating flattening willy nilly. The point is to have bias for flatter structures to make it easier for the user, but of course not all structures can or should be flat. On the other hand, don’t over-engineer your data structure so it makes finding things difficult. Grabbing an attribute is not necessarily any harder in a deeply nested structure, but filtering based on multiple deeply nested attributes in different branches can make a query quite complex.
  - nyuszika7h 5y ago
    > The article is not advocating flattening willy nilly. It certainly seems like it does when the first example for flattening is oversimplifying an already simple structure that doesn't really need flattening. Maybe that was not meant to be a serious example but rather just for ease of understanding, but then the article should have probably said so. > filtering based on multiple deeply nested attributes in different branches can make a query quite complex Can you elaborate on this please? Maybe I'm just too tired to think clearly at 1 am, but I don't see how filtering is any harder. You would just do something like `jq '.foo | select(.bar.baz >= 42 and .qux.moo.asd == "abc")`.
    - kbrazil 5y ago
      I can’t think of one off the top of my head, so maybe I’m misremembering why I originally decided I preferred flatter structures. It’s possible it was due to some limitation on my part as I was first learning to work with JSON. Looking back at some other JSON output, like `ip addr`, I’m not seeing anything egregious. There are just a couple useless top-level keys in the `iostat` output that make the queries longer and don’t seem to add value. Using the flat structure with a type field lends itself well to lazily outputting JSON lines, so maybe that’s why I tend to represent cli output that way. It’s sort of how you would format the data when sending to ELk or Splunk. Since I initially thought of JC as a cli tool, I was trying to reduce line length. Even the name JC was selected so it wouldn’t significantly increase the line length. So not nesting data inside meta objects is something I would encourage.
IgorPartola 5y ago
"kb_read_s": 0.12 This worries me. JSON doesn’t have support for fixed point math, does it? When will some random POSIX tool spit out scientific notation at me. Also, if you just output a flat schema, is there much of a point in this vs just: cpu: 0.2 kw_read_s: 0.12 The difference is that you can use a JSON parser vs splitting on new lines and colons? I do like the idea of JSON output as an option but before every bug and mistake gets canonized as POSIX or some other standard can we at least talk about the output format for a bit?
- zeroimpl 5y ago
  You'll prefer JSON the minute it becomes: cpu: 0.2 kb_read_s: 0.12 mac_addr: 10:AA:FF:00:55:66
  - EuAndreh 5y ago
    cut -d: -f2-
  - IgorPartola 5y ago
    dict((key.strip(), val.strip()) for key, _, val in line.strip().partition(‘:’) for line in text.split(‘\n’) if line and line.strip()) Also who says we can’t have a universal parser for this format just like we have for JSON? Not everyone needs to write the one liner like above, just use the libtextformat.parse(text) or whatever we would call it.
    - wruza 5y ago
      Parsing outputs is a point where unix philosophy always breaks down for me. It looks like this: {program -> human-text -> parser}+ When it could look like this: program -> {struct-text -> program}+ [-> formatter] -> human-text .
      - IgorPartola 5y ago
        I don’t think your approach is a problem. I do think that JSON might be not the ideal format for struct text. Especially if you really just use it as a flat dict of key value pairs.
giaour 5y ago
This is one area where I find Powershell preferable. It allows commands to return structured data in a standardized way, which really helps interop between programs from different publishers
lukeholder 5y ago
MacOS Monterey added a CLI tool for speedtesting, and I noticed they have a -c: Produce computer-readable output So I tried it out: ~ networkquality -c | jq '{dl: (.dl_throughput / 1000000), ul: (.ul_throughput / 1000000)}' awesome! { "dl": 176.861488, "ul": 6.742952 } (Starlink in Perth, Western Australia)
nickysielicki 5y ago
I don't know if it's just me but I've found jq and/or json output of little relevance to my day-to-day command line usage. I will almost always reach for `python -c` before I reach for jq -- for better or worse.
- kbrazil 5y ago
  That’s why I created jello[0]. You get the power of python without the boilerplate so the experience is closer to jq with python syntax. [0] https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello
  - nickysielicki 5y ago
    Starred and bookmarked, will definitely reach for this the next time I need to work with json in a pipeline. Thanks for the link! Cool project.
gorgoiler 5y ago
I took a leaf out of zfs’s book and make all my apps’ output look like zfs/zpool. Optional header, selectable columns, one line per record, machine readable (raw) vs human readable numbers. I’ve nothing against JSON output but I just don’t need it when you can print out two columns, select on the first, and print the second. $ users -H -o name,hair | > awk ‘$1 == “gorgoiler” {print $2}’ gray Admittedly, that awk invocation is so commonly used it could probably be a lot more terse. Also, this whole house of cards collapses when you have data containing spaces.
gumby 5y ago
If you’re going to make a schema, which is a good idea, then make a command line option that emits it.
- kbrazil 5y ago
  Yep, in JC you can see the schema for any parser like so: $ jc -h --dig
setheron 5y ago
When I was at Amazon long time ago, there was a suite of tools that would use structured text. I think it was called "recs"
- kseistrup 5y ago
  There's also GNU recutils: https://www.gnu.org/software/recutils/ https://www.gnu.org/software/recutils/
agjmills 5y ago
Imagine if http APIs has similar output to lsof or df -h - nobody would write a script to use them! JSON makes a lot of sense, but a human-parseable format is also needed.
- hnlmorg 5y ago
  You're conflating two different concerns: 1. human review 2. scripting / automation In case one, human readable formats are obviously preferable. But the moment you need to script a command, you want it in a machine readable format. A perfect example of the differences between the two are how badly spaces in file names are handled. Granted POSIX deserve a lot of the blame here too.
billpg 5y ago
This would be a great option for find/xargs. -print0/-0 do the job for each other but then there's a disconnect with a user being able to view the list of files.
anyfactor 5y ago
My opinion about JSON is that- 1. You don't need to categorize every piece of data 2. You don't need to include everything in a single JSON file. Deep nesting JSON is very annoying. The key-value pair structure of JSON is simply being abused at this point. Also, I really don't appreciate using numerical values as keys. Please use a list.

183 points by kbrazil 5y ago

Another DON'T, silently switching between JSON and human-readable depending on whether the output destination is a pipe. Just an extra hassle when I'm writing my downstream command. Or could be phrased as a DO: give the user a switch to pick the output format, if you have both.

GauntletWizard 5y ago
I started to try to make an argument for pipe autodetection, but I just can't. It seems like a useful feature but is actually a trap. Shell scripts that are going to rely on json output should always explicitly specify that they want json output, and giving them the ability to shortcut that by autodetecting a pipe will only make it easy to ignore that - And then break when some other format comes into vogue. Human readable output should generally be the default, unless the programs are explicitly designed as only part of a pipeline. Having multiple outputs is a great feature, though. I'm especially fond of tooling in Kubernetes that allows you to nicely pipe things in and out in multiple formats.
strictfp 5y ago
Pretty much every linux tool pulls these shenanigans. I hate it when there's no flag to control output.
lytedev 5y ago
What about things like ANSI color codes?
- MereInterest 5y ago
  My preference would be to leave them in if they're the default, but to have an option to switch entirely to a more machine-readable format. It's a single line of sed to strip them out[0], and it's a bigger pain to figure out why a program has different behavior when debugging. [0] https://superuser.com/a/380778 https://superuser.com/a/380778
  - kbrazil 5y ago
    I typically run the command output through `cat` to see if it does different things when outputting to the terminal vs piping. The biggest changes are typically ANSI color codes and column length. Some programs do strange things, like changing how they escape characters (ls). JC turns off ANSI color codes when it’s output is not a terminal, but that’s it.
    - laumars 5y ago
      Changes in column length isn’t something that can’t be helped. The issue is that when a command outputs to a TTY, you can use that file descriptor to get the width of the TTY (ie your terminal emulator). But if you are piping a command then it’s STDOUT is no longer a TTY and thus you have no idea how wide the terminal is.
- foodstances 5y ago
  Hope it supports NO_COLOR (http://no-color.org/ http://no-color.org/) env NO_COLOR=1 ...
  - kbrazil 5y ago
    I hadn’t seen this before - thanks for sharing!
- b3morales 5y ago
  Good question; those are more reasonable to me since they aren't visible as characters and don't change the structure. So if I'm looking at the colorized output I can still use that as the basis for a sed or awk script operating on the non-colorized version.
  - b3morales 5y ago
    Oh, and I still want `--color=always` or equivalent for when I'm piping into a pager.

lunfard000 5y ago

come to the powershell-side, we have ConvertFrom-Json

umvi 5y ago

In addition to an option for writing output as JSON, consider also adding an option for streaming output to stdout. Those two features were added to GCC9 gcov and are what enabled me to write a tool that parallelizes coverage report generation. In practice this enabled generating coverage reports orders of magnitude faster than traditional gcov wrappers like lcov

enriquto 5y ago

If you ever need to fight against this annoying json trend (e.g., when your tool only emits certain information in stupid ungrepable json), consider filtering the output through a gron so that it becomes saner.

dylan604 5y ago
why is JSON ungrepable? grep key file.json | awk -F: '{print $2}' if you're already searching for a key, seems like you're just wanting the value. granted, i hardly ever (have i actually ever??) interact with JSON this way, so i'm not exactly familiar with pitfalls.
- b3morales 5y ago
  When it's not formatted, just emitted as a single line. (Granted grep still works, but...not nicely.)
  - dylan604 5y ago
    gotcha. i'm imagining a pretty gnarly regex to return the data after the first colon after the match but only up until the next comma or square bracket or curly bracket. yeah, that's unpleasant. thankfully, there are tools like jq to help maintain one's sanity
    - akyoan 5y ago
      To be honest I find it funny that anyone would attempt to “parse” such a simple format with the wrong tool when there are plenty that can extract data in dot notation.
- fragmede 5y ago
  The pitfall is that JSON has zero guarantees for how often line breaks do and don't occur, and is often used to represent hierarchical data. Grepping for 'key: foo', and some liberal use of -A and -B may find you what you're looking for, but grep is simply the wrong tool for that job. (And how do you handle a key with newlines in it?) jq [0] is the right tool, but jq's syntax is it's own, and is harder to use (unless you use it regularly). [0] https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

wruza 5y ago

CLI and JSON would be amazing if terminals made a step forward too. Because both raw json and triangled in-browser console json logging just suck for daily reading. A new terminal could either detect patterns or use explicit cues in json to format structures, and show raw data on demand. E.g. this json5: [{_repr:"ls:file", name:"foo.txt", type:"text/plain", size:512, access:"664", …}] could be presented as usual ls does, but processed further as json. A whole lot of representations (and editors) – including ui-based – could be added to the system (e.g. /usr/local/share/repr/ls:file (+x)) to format any sort of data, instead of formatting it in-program with getopt and printf. And when there is no repr file, well, you still have triangles mode. We’re too stuck with text=text=text idiom. Structure=text=ui would be so much better. (I’m aware of powershell and am ignoring it consciously)

cwalv 5y ago
This kindof reminds me of the ipython display protocol https://carreau.github.io/posts/29-JupyterCon-DisplayProtocol.ipynb/ https://carreau.github.io/posts/29-JupyterCon-DisplayProtoco...
laumars 5y ago
Such a shell already exists https://github.com/lmorg/murex https://github.com/lmorg/murex You’d have to learn a new shell syntax but at least it’s compatible with existing CLI tools (which Powershell isn’t)
- wruza 5y ago
  I was talking about terminal [emulators], not a shell. A shell has nothing to do with how the output stream is displayed. Murex is more like powershell in this regard, which is on/around the level of implementation that I personally find inappropriate.
  - laumars 5y ago
    I know you said terminal but short of rewriting the entire stack, TTYs and all, you have to work with what you’ve got. With murex, it does actually have a lot of intelligence built in that adapts how the output stream is displayed (eg rendering images in the terminal, “prettifying” JSON if STDOUT is a TTY, colourising STDERR red, etc. Plus I disagree that your GP points about how JSON is handled should be a shell thing. You’re talking about data being re-encoded in different formats depending on the output. That absolutely should be a shell thing (where the logic of the pipelines are handled) with the terminal being a dumb rendering client. The last thing I want is output to be modified by the terminal leaving me scratching my head as to whether a command is running correctly or whether the terminal is displaying it weirdly.

throw0101a 5y ago

See also libxo: > The libxo library allows an application to generate text, XML, JSON, and HTML output using a common set of function calls. The application decides at run time which output style should be produced. The application calls a function "xo_emit" to product output that is described in a format string. A "field descriptor" tells libxo what the field is and what it means. * https://github.com/Juniper/libxo https://github.com/Juniper/libxo Then add an "--output-format" option.

GordonS 5y ago
A +1 from me on something like `--format` - pipe auto-detection feels unnecessary and like an inevitable footgun. As just one example, the Azure CLI defaults to human-readable output, but has an "output" parameter so you can have JSON if you want - I've never once wanted any kind of format auto-detection, and I have to say that I still don't.
- hnlmorg 5y ago
  The problem with auto-detection is that in POSIX-like shells it only works for the most simplest of problems (eg `ls` becoming a single column list when piped) because ultimately everything is treated by the shell as white space delimited list and treated by the OS as an untyped stream of bytes. However more modern shells fix this problem with having typed pipelines and builtins written to understand more than just a flat file of bytes. Take _murex_ for example (disclaimer, I'm the author of that shell): » jobs PID State Background Process Parameters 2104 Executing true exec sleep 9000000 2240 Executing true exec sleep 9000000 It's readable but what if I wanted to pass it as a table? » jobs | cat ["PID","State","Background","Process","Parameters"] [2104,"Executing",true,"exec","sleep 9000000"] [2240,"Executing",true,"exec","sleep 9000000"] ok, so it auto-detects it is running as a pipe and outputs it as a jsonlines table. That would be annoying in Bash. But with a type aware shell, that shell knows it's a jsonlines table, eg » jobs | debug | [[ /Data-Type/Murex ]] jsonl ...but what can we do with a jsonlines table? Well you can select individual columns: » jobs | [ PID State ] [ "PID", "State" ] [ "2104", "Executing" ] [ "2240", "Executing" ] run SQL against it » jobs | select * where PID > 2200 ["PID","State","Background","Process","Parameters"] ["2240","Executing","true","exec","sleep 9000000"] iterate through each row » jobs | foreach proc { if { =$proc[0]>2200 } then { echo $proc } } [2240,"Executing",true,"exec","sleep 9000000"] or even just convert it into another format, like CSV » jobs | format csv PID,State,Background,Process,Parameters 2104,Executing,true,exec,sleep 9000000 2240,Executing,true,exec,sleep 9000000 ...or YAML... » jobs | format csv - - PID - State - Background - Process - Parameters - - "2104" - Executing - "true" - exec - sleep 9000000 - - "2240" - Executing - "true" - exec - sleep 9000000 And it all just works without you having to think or even know what data format is traversing the pipeline. However unfortunately none of this is possible with Bash. And thus the majority of tools are forced to be dumb to compensate.

simonw 5y ago

I hadn't seen jc before (by the author of this piece: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc ) - what a great idea! It has parsers for around 80 different classic Unix utilities such that it can convert their output to JSON. ~ % dig example.com | jc --dig | jq [ { "id": 61315, "opcode": "QUERY", "status": "NOERROR", "flags": [ "qr", "rd", "ra" ], "query_num": 1, "answer_num": 1, "authority_num": 0, "additional_num": 1, "opt_pseudosection": { "edns": { "version": 0, "flags": [], "udp": 512 } }, "question": { "name": "example.com.", "class": "IN", "type": "A" }, "answer": [ { "name": "example.com.", "class": "IN", "type": "A", "ttl": 85586, "data": "93.184.216.34" } ], "query_time": 29, "server": "10.0.0.1#53(10.0.0.1)", "when": "Sun Dec 05 15:12:08 PST 2021", "rcvd": 56, "when_epoch": 1638745928, "when_epoch_utc": null } ]

EdwardDiego 5y ago
Oh Christ yes, it supports lsof! The output from lsof has always been hard to script up.
aasasd 5y ago
Turns out I can cross out a todo item off my list, because I wanted to make exactly the same thing. Now, to find a query tool with a saner language than Jq...
- ZeroGravitas 5y ago
  What kind of thing are you trying to do? jq can get pretty deep but for most things in this area I'm not sure how it could improve upon, but would be interested in hearing alternatives. https://github.com/fiatjaf/jiq https://github.com/fiatjaf/jiq Is a realtime feedback wrapper which I find useful when crafting one-off command line uses for jq and it starts getting crazy.

1vuio0pswjnm7 5y ago

Honestly I do not really understand this idea because, AFAIK, JSON was designed for Javascript in a web browser. By and large, Javascript-enabled web browser expect access to generous amounts of memory. This is not the case for the common UNIX userland programs. These programs do not expect large amounts of memory and many are written with the intent that they may be used to process text line-by-line. This JSON idea reminds me of Windows "PowerShell". Microsoft actually has to limit how much memory can be used by the shell. Why is that. https://devblogs.microsoft.com/scripting/learn-how-to-configure-powershell-memory/ https://devblogs.microsoft.com/scripting/learn-how-to-config... One of the things I like most about the UNIX userland is that I can use small programs to edit vary large files, without needing lots of memory. I want programs that are designed to accomodate the possibility of line-by-line processing. If the intent is to make output network friendly, maybe something like netstrings is useful. Easy to parse. Low memory footprint. Seems to me this JSON idea is not designed to improve performance, agility or resource efficiency but to ignore the UNIX example in favour of a different, slower, approach that is perceived as easier for some people to use. Namely those who do not want to spend the time to learn how to use an existing, faster solution with lower resource requirements.

wruza 5y ago
JSONL is the answer for arrays. “[…50mb…]” is too big to be processed in a streaming mode, but particular items usually cannot be split anyway, and “{…}\n50mb more” is what you need.
tester756 5y ago
for me it is: readable, handy since probably every language has libs that work with it fine, there's a lot of tools that work with jsons e.g generating code classes from json, it's insanely popular, really easy to learn
kbrazil 5y ago
In actuality most CLI output is quite small and can easily be represented as a single JSON document. There are a few commands that can produce huge amounts of output and those are good candidates for using JSON Lines as noted in the article. JC has streaming parsers that lazily output JSON lines for these types of commands. (ls, ping, vmstat, iostat, etc.)

nyuszika7h 5y ago

I don't really understand the point about flattening. > This way I can easily filter the data in jq or other tools without having to traverse levels. How is doing `jq '.cpu.speed'` any harder than doing `jq '.cpu_speed'`? IMO as long as you aren't going insane with nesting levels, it's actually better to have a proper structure than dumping everything into an ugly flat object.

kbrazil 5y ago
The article is not advocating flattening willy nilly. The point is to have bias for flatter structures to make it easier for the user, but of course not all structures can or should be flat. On the other hand, don’t over-engineer your data structure so it makes finding things difficult. Grabbing an attribute is not necessarily any harder in a deeply nested structure, but filtering based on multiple deeply nested attributes in different branches can make a query quite complex.
- nyuszika7h 5y ago
  > The article is not advocating flattening willy nilly. It certainly seems like it does when the first example for flattening is oversimplifying an already simple structure that doesn't really need flattening. Maybe that was not meant to be a serious example but rather just for ease of understanding, but then the article should have probably said so. > filtering based on multiple deeply nested attributes in different branches can make a query quite complex Can you elaborate on this please? Maybe I'm just too tired to think clearly at 1 am, but I don't see how filtering is any harder. You would just do something like `jq '.foo | select(.bar.baz >= 42 and .qux.moo.asd == "abc")`.
  - kbrazil 5y ago
    I can’t think of one off the top of my head, so maybe I’m misremembering why I originally decided I preferred flatter structures. It’s possible it was due to some limitation on my part as I was first learning to work with JSON. Looking back at some other JSON output, like `ip addr`, I’m not seeing anything egregious. There are just a couple useless top-level keys in the `iostat` output that make the queries longer and don’t seem to add value. Using the flat structure with a type field lends itself well to lazily outputting JSON lines, so maybe that’s why I tend to represent cli output that way. It’s sort of how you would format the data when sending to ELk or Splunk. Since I initially thought of JC as a cli tool, I was trying to reduce line length. Even the name JC was selected so it wouldn’t significantly increase the line length. So not nesting data inside meta objects is something I would encourage.

IgorPartola 5y ago

"kb_read_s": 0.12 This worries me. JSON doesn’t have support for fixed point math, does it? When will some random POSIX tool spit out scientific notation at me. Also, if you just output a flat schema, is there much of a point in this vs just: cpu: 0.2 kw_read_s: 0.12 The difference is that you can use a JSON parser vs splitting on new lines and colons? I do like the idea of JSON output as an option but before every bug and mistake gets canonized as POSIX or some other standard can we at least talk about the output format for a bit?

zeroimpl 5y ago
You'll prefer JSON the minute it becomes: cpu: 0.2 kb_read_s: 0.12 mac_addr: 10:AA:FF:00:55:66
- EuAndreh 5y ago
  cut -d: -f2-
- IgorPartola 5y ago
  dict((key.strip(), val.strip()) for key, _, val in line.strip().partition(‘:’) for line in text.split(‘\n’) if line and line.strip()) Also who says we can’t have a universal parser for this format just like we have for JSON? Not everyone needs to write the one liner like above, just use the libtextformat.parse(text) or whatever we would call it.
  - wruza 5y ago
    Parsing outputs is a point where unix philosophy always breaks down for me. It looks like this: {program -> human-text -> parser}+ When it could look like this: program -> {struct-text -> program}+ [-> formatter] -> human-text .
    - IgorPartola 5y ago
      I don’t think your approach is a problem. I do think that JSON might be not the ideal format for struct text. Especially if you really just use it as a flat dict of key value pairs.

giaour 5y ago

This is one area where I find Powershell preferable. It allows commands to return structured data in a standardized way, which really helps interop between programs from different publishers

lukeholder 5y ago

MacOS Monterey added a CLI tool for speedtesting, and I noticed they have a -c: Produce computer-readable output So I tried it out: ~ networkquality -c | jq '{dl: (.dl_throughput / 1000000), ul: (.ul_throughput / 1000000)}' awesome! { "dl": 176.861488, "ul": 6.742952 } (Starlink in Perth, Western Australia)

nickysielicki 5y ago

I don't know if it's just me but I've found jq and/or json output of little relevance to my day-to-day command line usage. I will almost always reach for `python -c` before I reach for jq -- for better or worse.

kbrazil 5y ago
That’s why I created jello[0]. You get the power of python without the boilerplate so the experience is closer to jq with python syntax. [0] https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello
- nickysielicki 5y ago
  Starred and bookmarked, will definitely reach for this the next time I need to work with json in a pipeline. Thanks for the link! Cool project.

gorgoiler 5y ago

I took a leaf out of zfs’s book and make all my apps’ output look like zfs/zpool. Optional header, selectable columns, one line per record, machine readable (raw) vs human readable numbers. I’ve nothing against JSON output but I just don’t need it when you can print out two columns, select on the first, and print the second. $ users -H -o name,hair | > awk ‘$1 == “gorgoiler” {print $2}’ gray Admittedly, that awk invocation is so commonly used it could probably be a lot more terse. Also, this whole house of cards collapses when you have data containing spaces.

gumby 5y ago

If you’re going to make a schema, which is a good idea, then make a command line option that emits it.

kbrazil 5y ago
Yep, in JC you can see the schema for any parser like so: $ jc -h --dig

setheron 5y ago

When I was at Amazon long time ago, there was a suite of tools that would use structured text. I think it was called "recs"

kseistrup 5y ago
There's also GNU recutils: https://www.gnu.org/software/recutils/ https://www.gnu.org/software/recutils/

agjmills 5y ago

Imagine if http APIs has similar output to lsof or df -h - nobody would write a script to use them! JSON makes a lot of sense, but a human-parseable format is also needed.

hnlmorg 5y ago
You're conflating two different concerns: 1. human review 2. scripting / automation In case one, human readable formats are obviously preferable. But the moment you need to script a command, you want it in a machine readable format. A perfect example of the differences between the two are how badly spaces in file names are handled. Granted POSIX deserve a lot of the blame here too.

billpg 5y ago

This would be a great option for find/xargs. -print0/-0 do the job for each other but then there's a disconnect with a user being able to view the list of files.

anyfactor 5y ago

My opinion about JSON is that- 1. You don't need to categorize every piece of data 2. You don't need to include everything in a single JSON file. Deep nesting JSON is very annoying. The key-value pair structure of JSON is simply being abused at this point. Also, I really don't appreciate using numerical values as keys. Please use a list.