5 ms·
Tips on adding JSON output to your command line utility. (2021)
- cynicalsecurity 3y agoThis basically turns command line tolls into APIs. Crazy, in a good way.
- inetknght 3y agoNot really that crazy at all. And yes it's definitely good.
- enriquto 3y agoBut why? Just for people to gron it back into usable form? Unix tools work best with line-based data. Using some "structured" monstrosity like xml or json forces all other tools to deal with this particular format, thus breaking the orthogonality between programs.
- simonw 3y agoNewline-delimited JSON is so much more useful to me than weird Unix line-formatted output that I have to parse with pattern matching or regular expressions. It's basically a line-based format that can represent all of the JSON types and includes support for nested data structures where necessary. What's not to like?
- qwertox 3y agoIt's also supported by simdjson [0] (which has a lot of language bindings [1]): > Multithreaded processing of gigantic Newline-Delimited JSON (ndjson) and related formats at 3.5 GB/s [0] https://simdjson.org/ https://simdjson.org/ [1] https://github.com/simdjson/simdjson?tab=readme-ov-file#bindings-and-ports-of-simdjson https://github.com/simdjson/simdjson?tab=readme-ov-file#bind...
- bsdetector 3y agoJSON is not immediately usable by and is cumbersome to parse correctly in a shell. A simple line-based shell variable name=value format works unreasonably well. For example: # ls --shell-var ./thefile dir="/home/user" file="thefile" size=1234 ... # eval $(ls --shell-var ./thefile); echo $size 1234 If this had been in shells and cmdline tools since the beginning it would have saved so much work, and the security problems could have been dealt with by an eval that only set variables, adding a prefix/scope to variables, and so on. Unfortunately it's too late for this and today you'll be using a pipeline to make the json output shell friendly or use some substring hacks that probably work most of the time.
- kitd 3y agoThat's not your original request though, to use line-based data. It seems you're determined not to use jq but if anything, json output | jq is more the unix way than piping everything through shell vars.
- mlhpdx 3y agoAnd, the author isn’t suggesting only having JSON output, but adding it as an option for those of use that would make use of it. The plain text should remain as well (and has to or many, many things would break). On a separate point, I find the JSON much easier to reason about. The wall of text output doesn’t work for my brain - I just can’t see it all. Structuring/nesting with clear delineations makes it far easier for me to grok.
- bsdetector 3y ago> That's not your original request though, to use line-based data. It wasn't my request and OP (not me) said "line-based data" is best. The comment I replied to said "Newline-delimited JSON ... a line-based format". If the only objection you have is "but that's line-based!" then you're in a completely different conversation. > if anything, json output | jq is more the unix way than piping everything through shell vars. The unix way is line-based. The comment I replied to is talking about line-based output. Line-based output is the only structure for data universal to unix cmdline tools - even tab/space isn't universal; sending structured non-line-delimited data to a program to unpack it is the least unix-like way to do it. Also there's no pipe in the shell-variable output scheme I described, whereas "json | jq" is a shell pipeline.
- deleted 3y ago[deleted]
- eternityforest 3y agoIf you're using UNIX tools to parse it, it sucks, but generally if I'm reading the output of a command, and the command is more than one word, I'm doing the whole thing in Python. That's real programming, and for that I want type checkers and debuggers and modern syntax and all that. And the performance is often faster because you're not spinning up subprocesses for each command.
- pluto_modadic 3y ago./some_command | jq '.memory_use' vs: okay, run it once with head (assuming we have a --dry option...) ah, that's the column. okay cut -d"," -f0, ah whoops it's starting at 1. ah, damn, there's a weird comma in the name/quote. oh weird, that one's null, ah heck. schemas are cool. JSON extends and builds on the UNIX idea of having things pipe and plumb well together.
- alerighi 3y agoPlain text is more easy to reason about, because we are used to process text. A good textual output, that is records delimited by spaces, tabs or a delimiter, to me is all it's needed, for most applications. An object structure it's much more complex to use. For example an output that is a set of records can be easily imported in an Excel sheet, in an SQL database, processed line by line, without issues. Processing JSON is not straight forward, not all programs support JSON. Finally JSON can't be processed as a stream, meaning that tools like head, tail, etc. doesn't work on JSON, you have to read it all in memory, or use JSON lines, that is not a standard format, that not all parsers support natively, etc. JSON is good if for integrating the program inside other programs (as a subprocess), so having an option to input/output JSON in a program is useful, but to me it's not as useful for interactive shell usage. I prefer to use UNIX tools such as grep, cut, head, tail, etc.
- mlhpdx 3y agoBut as the article points out, the advice to use unbuffered JSON lines for commands that are line oriented is well given. Not doing that can really make life sad.
- dylan604 3y agoyou still need to know the schema of the JSON which could also use a dry run as well. not really sure how just because it's JSON solves that in your mind
- Jcowell 3y agoOrient the idea be that because it’s json there’s a schema somewhere the end user can refer to?
- placatedmayhem 3y agoLine-oriented formats, like most traditional Unix-style tools, are for human consumption. JSON is bad at that, thus gron. On the other hand, structured output formats, like JSON, make it easier to consume with other programs. Standard formats have readily-available and commonly used libraries, whereas line parsing tends to be one-off for every program. Whether JSON is the best format for this is certainly debatable, but it is quite ubiquitous, which is a huge advantage. I doubt many folks would propose XML as a general recommendation. Tools should have both options on their path to maturity -- both human-consumable and computer-consumable output format options.
- janderland 3y agoI’d rather use JMES Path (or jq even tho it’s messier) to restructure my data rather than some mixture of awk, sed, cut, etc. Line output often needs restructuring in my experience, though JSON will always need it.
- Spivak 3y agoUnix tools on newline delineated list like objects work great, but try them on dict like objects and it becomes clunky really fast. Parsing `ip` output is a good example where the data naturally lends itself to iface:attrs. Plucking the value you want by `.[.name | startswith('eth')].addr` is way easier than pulling this out with grep/awk.
- BenjiWiebe 3y agoYou probably already know this but just in case: You can specify an interface when using ip, e.g. ip addr show dev enp2s0
- __MatrixMan__ 3y agoThe caller can just specify the output format that they want, and ever since I switched to nushell I pretty much always want json.
- deleted 3y ago[deleted]
- keybored 3y agoWhat orthogonality? “Line-based data” that varies randomly from tool to tool? Using json between programs is perfectly orthogonal.
- IshKebab 3y agoNonsense. Using a structured format means all tools are using the same format, instead of every tool making up their own (usually broken) system. Look at how many flags `ls` has to feed filenames into other programs without breaking. Also, using a proper format like JSON makes it actually robust. Most ad hoc pipelines break if you so much as put a space in a filename.
- pluto_modadic 3y agoJSON is so much more usable sometimes. Especially when things don't neatly parse into cut -d splits!
- JohnMakin 3y agoYes - and this is mostly because the jq parser tool is so useful. yq is similarly great.
- sureglymop 3y agoSince yaml is a superset of json, sometimes when I have to use yaml for some tool/config I just use json (and optionally with yml comments).
- rwmj 3y agoThis is really the only reason, because JSON as a format is otherwise terrible. No good way to represent 64 bit numbers, very limited types and no comments.
- moregrist 3y agoI’m a big fan of tools producing json output, but I couldn’t help but chuckle a bit at: > This allows easier parsing in scripts by using JSON parsing tools like jq, jello, jp, etc. without arcane awk, sed, cut, tr, reverse, etc. incantations. While I love jq, I find its syntax to be utterly baroque. Far more than awk and sed (let alone simple tools like cut and tr). To the point where I end up looking at the jq man page to remind myself how to do even seemingly simple things at times. And often jq is just the beginning of a pipeline to get hierarchical data into columnar format so I can hit it with awk and sed. But maybe this is all about familiarity; I have years of experience cobbling together shell tools and reach for jq comparatively less. I suspect the author may be the opposite.
- Mordisquitos 3y agoI would argue it's not just about familiarity. I had the same reaction to that quote and feel the same as you do, and yet I have been using jq before I first learned and started using awk. For a long time I had been familiar with jq, but every time I had to use it for any non-trivial task after not touching it for a while I had to go to the docs and really scratch my head. Meanwhile, I had no idea about awk, and for ages I would just see those awk one-liners on Stack Overflow as weird arcane alternatives to "simply" piping sed, grep and tr over bash. However, one day I finally had the time and the reason to have a look at AWK Programming for a specific use that I needed, and it immediately clicked. Awk makes sense and now I can easily fall back to it every time, even if I haven't used it for ages. At most, all I need to look up is the order of parameters in the gawk regex functions. And yet, time and time again I am baffled by jq, however much I have used it in the past. Whenever I need to parse a JSON I'm happy enough if I can use jq to get it to a good enough halfway point so that I can pipe it to awk to do the actual work.
- DangitBobby 3y agoChatGPT flawlessly constructs jq commands for whatever banal task you have. Game changer.
- simonw 3y agoCompletely agree. I use jq multiple times a day now, purely because GPT-4 or Claude Opus mean I never have to think about how to express anything in it.
- ramses0 3y agoI'll my $0.02: when possible, assume/emit "arrays of records". eg: `ls ...` should be an an array of homogenous entities (objects) with usually simple key=>value relationships. Slight variations such as "field contains an array of strings" (eg: tags, groups, file path hierarchy, temperature per each CPU) is also totally fine. Seeing the wildly differing record types and deeply nested "stuff" in some of the examples weirds me out. { "type_cpu": [ ... ], "type_ram": [ ... ], "type_network": [ ... ], ... } ...obviously, in some cases a graph/hierarchy is very useful to be able to traverse, but emitting records and "coalescing" the path into a record-field makes simple operations simple. Example: `hwstats | jq '.type_cpu | .[] | select( .temp > 40 ) | .hierarchy_field` Don't make me go like `.cpu.bank[0].chip[3].temp` or whatever... just give me "all the CPU's" and then give me a good URI(!!) to describe it or search for it later. Rationale: coalescing and traversing weird object paths is tough to do dynamically via current shell DSLs, and the URI concept as an ID excellent application.
- jiehong 3y agoAn array of records, or just 1 record per line, also works nicely with jq. For that matter, it can even be imported in like duckdb and handled like a table :)
- dec0dedab0de 3y agoI prefer jsonl from command line tools, just cause it’s nicer to log, and parse using standard tools. Plus it’s like a two second function if you’re doing it in a real programming language. I think it’s the best of both worlds. They say to use JSON lines for streaming, but even if you’re not it’s still very nice to be able to do my_command | grep xyz and still have it be parseable
- isatty 3y agoAbsolutely not a fan of JSON. It’s not typed, and I definitely don’t want to know about each tools schema (and it’ll be worse without the definition). There is nothing wrong with Linux tools, they do one thing and do it well.
- throw10920 3y agoThis is self-contradictory. JSON is more typed and more predictable than what comes out of UNIX tools.
- xomodo 3y agoI don't mind json outputs, but pls keep cli options as low letters, eg.: 'kubectl get pods -o json'.
- echoangle 3y agoWhy is that important?
- al_borland 3y agoMixing case in a case sensitive environment is unnecessarily confusing and annoying.
- Karellen 3y ago> The file systems of Unix machines all have the same general structure. [...] Note the obsessive use of abbreviations and avoidance of capital letters; this is a system invented by people to whom repetitive stress disorder is what black lung is to miners. Long names get worn down to three-letter nubbins, like stones smoothed by a river. -- Neal Stephenson, In the Beginning Was the Command Line (Ch. 14), 1999 https://steve-parker.org/articles/others/stephenson/oral.shtml https://steve-parker.org/articles/others/stephenson/oral.sht...
- fragmede 3y ago. --json it's one character shorter than -o json
- matheusmoreira 3y agoParsing is hard and this definitely beats having to write ad-hoc parsers for stuff. I worry we're essentially reinventing a less verbose form of SQL though.
- kunley 3y agoNo, parsing is easy... I recommend the "Crafting Interpreters" book.
- matheusmoreira 3y agoI love that book. Parsing Techniques is even more detailed. It's still pretty hard. Easy would be a parse(grammar, input) function in the standard library. Earley algorithm is perfect for that. I wonder why no one's thought of it.
- chriswarbo 3y ago> I worry we're essentially reinventing a less verbose form of SQL though. We already have that with paste, join, etc. from Coreutils https://learnbyexample.github.io/cli_text_processing_coreutils/join.html https://learnbyexample.github.io/cli_text_processing_coreuti...
- jiehong 3y agoOn linux systems, I was thinking that it would be great if there was a dbus wrapper that outputs all data as json by default. As many things are exposed on dbus (like systemd units, timers, services, etc), and busctl [0] has a --json output. We could have a dbus-services, or dbus-timers, etc to retrieve that on dbus and emit json. If most things were on dbus, one could have access to almost everything on the system as json directly, while talking to the underlying implementation of most cli by default. I suppose something similar could be done on MacOS with launchctl and others (I don't know much about MacOS, though). [0]: https://www.man7.org/linux/man-pages/man1/busctl.1.html https://www.man7.org/linux/man-pages/man1/busctl.1.html
- 1vuio0pswjnm7 3y agoGiven that I prefer using ubiquitous, relatively small and simple utilities such as sed and tr, I use a simple custom utility I wrote myself using flex to make JSON output line-delimited so it's (a) human readable and (b) can be processed with traditional UNIX utilities such as sed and tr. Although it's quite popular jq, also originally written using flex (and yacc), is (c) not as ubiquitous as UNIX utilties such as sed and tr and (d) too large and complicated for me.
- fragmede 3y agohave you tried gron?
- deleted 3y ago[deleted]
- 1vuio0pswjnm7 3y agoI have tried many alternatives, even including large Go binaries such as gron. None of them appealed to me. I use a comparatively simple C program that is 45.9 KiB when statically-compiled. Of the alternatives I have tried, this one was the most interesting: https://news.ycombinator.com/item?id=31166956 https://news.ycombinator.com/item?id=31166956
- zie 3y agoJust use LibXo? https://libxo.readthedocs.io/en/latest/ https://libxo.readthedocs.io/en/latest/