6 ms·
JSON on the Command Line with Jq
- NinoScript 7y agoJq is a great tool, I wish more people knew about it!
- smitty1e 7y agoIt's a bit arcane to use, but has proven itself over and over when doing little tasks with AWS CLI returns.
- GhettoMaestro 7y agoExactly right. I use jq to mangle get/list requests from AWS (and others) and then recursively hit each element for whatever reason. Without jq none of my workflows in this area would be remotely easy. TLDR: Learn JQ! It is like sed and grep, but for JSON.
- wsgeek 7y ago100%
- dastx 7y agoIf you're using AWS you're better using the built in option available for the cli. The --query option (and often combined with the --output option) does this, and uses JMESPath. Especially when you're scripting, it eliminates the need for having yet another tool.
- smitty1e 7y agoThat's a great point. Leverage --query for good justice. Unless you need some additional juice, where jq supports actually transforming the return. Quite arguably a heavier tool, e.g. a boto3 script becomes appropriate where --query falls short. But options are a great thing to have.
- dragonwriter 7y ago> If you're using AWS you're better using the built in option available for the cli. The --query option (and often combined with the --output option) does this, and uses JMESPath. Jq is so much more capable than jmespath that I find myself using jq on the json output of AWS CLI commands more than using the built-in query option. Sure, it requires another tool, but it’a often the best tool for the job.
- commandlinefan 7y agoI’ve been using it as a pretty-printer for a while now, it never occurred to me that you could use it to filter items out. In fact, I’ve been piping jq to grep to find things!
- dang 7y agoThread from 2016: https://news.ycombinator.com/item?id=13090604 https://news.ycombinator.com/item?id=13090604 2015: https://news.ycombinator.com/item?id=9446980 https://news.ycombinator.com/item?id=9446980 2014: https://news.ycombinator.com/item?id=7895076 https://news.ycombinator.com/item?id=7895076 2013: https://news.ycombinator.com/item?id=5734683 https://news.ycombinator.com/item?id=5734683 2012: https://news.ycombinator.com/item?id=4985250 https://news.ycombinator.com/item?id=4985250 https://news.ycombinator.com/item?id=4679933 https://news.ycombinator.com/item?id=4679933 (for the curious)
- Gys 7y agoJq is nice and I regularly use it. Does anyone know a more interactive json tool? With, for example, auto-completion? Showing possible fields to select next and maybe even possible values if fields are choosen.
- frou_dh 7y ago> Does anyone know a more interactive json tool? With, for example, auto-completion? https://github.com/antonmedv/fx https://github.com/antonmedv/fx It just uses JS for the query language and can be hooked in to bash completion. Personally I don't get on with jq, because I don't feel that plinking with JSON is something that warrants memorising a bespoke language.
- aagat 7y agoI have been very happy with https://github.com/fiatjaf/jiq https://github.com/fiatjaf/jiq
- glutton 7y agoI like https://github.com/raine/ramda-cli https://github.com/raine/ramda-cli It has interactive mode which gives you instant feedback. I also don't need to learn a completely new API, as I'm already familiar with ramda.
- DC-3 7y agoFor those interested in JSON as a command line protocol, I think it's interesting to see how Andreas Kling has integrated it into his Serenity OS project. While the project is aesthetically retro, its architecture is forward-facing, and the usage of JSON for the representation of such data as the process tree is, I think, an interesting take on old Unix ideas.
- akling 7y agoIt definitely makes writing the userspace code a lot nicer! The main issue I've run into so far is that my JSON parser can generate quite heavy in-memory representations of parsed data. It's far from optimized though, and a lot can be done before needing to worry about the limitations of the format itself. :)
- tejtm 7y agoA line of `jq` to turn any .json into a set of xpaths. https://github.com/TomConlin/json2xpath https://github.com/TomConlin/json2xpath
- GhettoMaestro 7y agoThis is useful! Thank you for sharing!!!!
- jayd16 7y agoBeing reminded of JQ has me thinking... I actually don't hate powershell or at least I like it's attempt at typed pipes. Has anyone gone down the road of posix like Json pipes? Or maybe just Json bridging the gap between fully typed pipes and the string based Unix cli world? Even just having a Json output format for the core posix apps would be good. If it had OS support, maybe you wouldn't even have to serialize to strings if both sides of the pipe happened to be compatible.
- inopinatus 7y agojq is the AWK of today. I'm in favour of jq in *nix base. I use it almost as much as grep | cut | sort. JSON is ubiquitous now. So much so that even my <conservative enterprise customers>, who 20 years preferred data in CSV, 10 years ago preferred XML, now want to exchange data in JSON.
- dmurray 7y agoCSV still seems to be the standard in finance.
- solatic 7y agoPrecisely, because JSON is ubiquitous (unlike XML) and relatively safe to parse (unlike CSV). It's not perfect - it doesn't have higher-order features like types/schemas or functions built in. But JSON + jq is a workhorse. I agree that distros do people a disservice by not bundling jq in base (and attempting to provide JSON output for distro-specific tooling).
- lenkite 7y agoSomething better than jq please. It has the most arcane and inconsistent syntax that I have ever seen in a CLI tool. There are newer tools like jp with simpler language that can be understood by mere mortals with limited brain power.
- kmelve 7y agoYou should check out GROQ (https://groq.dev https://groq.dev), it's a query language for JSON.
- aasasd 7y agoEvery time I try to do something involved with jq (i.e. more than a couple commands deep) and inevitably end up wandering through the manual, I get a persistent feeling that jq could be cleaned up quite a bit. That it either has structure throughout that I fail to grasp, or that it's really rather disorganized and convoluted in places. This is exacerbated by me never being able to find anything in the manual without skimming half of it each single time. E.g.: Filtering and transforming objects by reaching a few levels deeper in them seems to be way easier to slap together in Python's list comprehensions or some functional notation. Stuff like [{ * * x, qwe: x.qwe*2} for x in a.b if x.y.z == 1]. Add nested lists to the above, and I can spend another fifteen minutes writing a one-liner (i.e. `a` is a list of objects with lists in some field). I once had to deal with a conditional structure where a field might be one object or a list of them. Hoo boy. Every time I want to check the number of entries my filters print out, I need to wrap them in an array. Repeat this a couple dozen times during writing a complex query. Weirdly, this is where strict structure gets in the way―and while I understand the reasoning, I feel that something could be done for such basic need. (Like, aggregate filters should be able to aggregate instead of acting on individual items? Maybe it's already in the language, but I'm not in the mood to go over the manual again.) Variables seem to be bolted on as an afterthought, or at least the syntax doesn't exactly accommodate them. Meanwhile, they're necessary to implement some filters omitted in the language. Compare that to Lisp's simple `(let)`. IIRC the manual also says that jq has some semblance of functions, but I'm afraid to think about them with this syntax. I like the idea a lot, but execution not so much. Frankly I'll probably end up throwing together a script in Lumo or something, that will accept Lisp expressions and feed JSON structure to them. (I'd use Fennel, but JSON has actual null while Lua... doesn't.) Btw, I have pretty much the same sentiment about Git. Git structures, great. Git tools, oy vey. Maybe I need Lisp for Git, or at least Python for Git.
- cube2222 7y agoHey, just wanted to say that this comment made me create jql, an alternative to jq with a lispy syntax: https://github.com/cube2222/jql https://github.com/cube2222/jql Thanks for the inspiration!
- tomlockwood 7y ago
- dan-robertson 7y agoJq can do lots of things that are similar to what’s normally done with awk, cut, sed, and grep. The languages pipeline model is slightly opaque (eg as soon as you use a function like select) and I feel tricky to get the hang of but overall I think it works well for the kind of job jq does. That said, I usually get on well with a halfway house of piping jq -c ... into eg grep or sort -u for easy processing (-c writes one json object per line, aka compact form), or selecting a few elements, putting them in an array, jq-piping into @csv, and passing -r to get raw output which I can then process with awk or sort or other Unix commands. One subtle caveat is that jq normalises all the json it processes, so for example escaped characters in strings may change and numbers which are not IEEE doubles get converted (eg if someone writes a 64-bit int or a bignum as a number not a string in the json)
- llimllib 7y agoI like jq, but the documentation is super thin. Every time I go to do anything involved, I inevitably end up at stack overflow hoping somebody else has had to do it before. I really wish there were a much greater selection of usage examples in the manual.
- dxxvi 7y agoI'd like to have a command line tool that allows me to modify the json with javascript. Does jq provide any feature to process json that it's very difficult/cumbersome to do in js?
- dastx 7y agoJq allows you to fully transform a json structure. Batteries included. Similar to awk/sed etc you can even write your jq code in a separate file, and point jq to it.