8 ms·
Jq: A lightweight and flexible command-line JSON processor
- JoshTheGeek 12y agounderscore-cli: a similar tool, in JS (node) https://news.ycombinator.com/item?id=7896054 https://news.ycombinator.com/item?id=7896054
- edgarbob 12y agoDoes anyone know of a similar tool for XML? In particular, it would be cool to have a useful way to run xpath, and maybe xquery, from the command line. The tools I've found have unfortunate interfaces, e.g. they won't read XML from stdin.
- gav 12y agoHave you tried XMLStarlet[1]? It's my go-to for command line XML processing. [1] http://xmlstar.sourceforge.net/ http://xmlstar.sourceforge.net/
- unhammer 12y agoEven with libxml's xmllint you can do some xpath stuff: http://wiki.apertium.org/wiki/Xml_grep http://wiki.apertium.org/wiki/Xml_grep
- cgrubb 12y agoYou might check out these: http://www.w3.org/Tools/HTML-XML-utils/ http://www.w3.org/Tools/HTML-XML-utils/ However, I just use a Ruby script which loads the Nokogiri library: https://github.com/clarkgrubb/data-tools/blob/master/src/dom-awk.rb https://github.com/clarkgrubb/data-tools/blob/master/src/dom... https://github.com/clarkgrubb/data-tools/blob/master/doc/dom-awk.1.md https://github.com/clarkgrubb/data-tools/blob/master/doc/dom...
- jeroenjanssens 12y agoI'd just use xml2json and then apply jq. Usually gets the job done. The benefit of that is that you only need to learn jq. See this blog post I wrote a while ago for an example: http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html http://jeroenjanssens.com/2013/09/19/seven-command-line-tool...
- cryptonector 12y agoMost of the JSON conversions from XML get it all wrong though. XML is not really an object (dict, hash) based meta-schema. It's an array-based meta-schema. XML->JSON conversions should look something like this: [ {"tag":"something", "attributes": { ... }, "nodes": [ ...] }, ... ] with nodes being objects with one key to indicate if the node is a text node or an element, and if a text node then a value, and so on.
- bartc 12y agoxmllint - http://xmlsoft.org/xmllint.html http://xmlsoft.org/xmllint.html
- A_COMPUTER 12y agoI am not a huge fan of PHP, but you can do some useful stuff to XML using the PHP command line tool, provided you have the right libraries installed. I have run into situations where the package manager on a machine I'm using doesn't have an up to date xmlstartlet or xmllint, and have tidily gotten around this problem using PHP.
- facorreia 12y agoThis can help Docker automation by processing the output of `docker inspect`.
- Zolomon 12y agoThey use this tool extensively at Spotify. I saw a presentation by https://twitter.com/jooon https://twitter.com/jooon, when I was over there in September last year. It is a really cool tool. Especially since they use JSON internally for almost all of their data representations. Jq + Percol[1] is kind of cool. [1] https://github.com/mooz/percol https://github.com/mooz/percol - was on HN a few days ago.
- X4 12y agoIf percol is based on aut-fu, then it's probably not that good. The screenshot shows auto-fu, I know that zsh-plugin, but it's really messy and does very aggressive auto-completion (erasing your previous input). I'd rather use fish-shell instead.
- cryptonector 12y agoI was also thinking of Xiki (http://xiki.org/ http://xiki.org/) + jq. A Unix-like shell with JSON for object serialization could approximate the PowerShell, no? Think of ksh93's awful compound variables... done right.
- cryptonector 12y agoIs this the link to that presentation http://www.slideshare.net/jooon http://www.slideshare.net/jooon ?
- otikik 12y agoI just use it just as a pretty-printer for the console (jq '.') , and it does the job very well.
- simonw 12y agoThis is a really neat tool. If you just need pretty printing and don't have jq installed, you can use the python command line with the built in JSON module: wget http://reddit.com/.json http://reddit.com/.json | python -mjson.tool
- jamiesonbecker 12y ago# (fixed) output to stdout: wget -O- http://reddit.com/.json http://reddit.com/.json | python -mjson.tool
- rev087 12y agoI've been using this for a while, and it's wonderful to parse the output from API calls requested via curl, as well as quickly checking a specific package.json field from the npm output. The --json flag is handy for the latter.
- mathieuruellan 12y agoI'm using this tool with curl in scripts calling AWS api. Looking for an id of an object (security group, autoscaling group etc.) with its name and referencing it afterward in another command.
- pavel_lishin 12y agoHey, me too! It's pretty awesome, although the documentation/examples aren't as great as they could be - it took me some time to figure out how to properly parse arrays, and I'm still not sure how to actually do what I want to do (I want to convert an array of two instance objects into an array with just two instance IDs.
- mh- 12y ago% aws ec2 describe-instances --instance-ids '["i-3026a249","i-28739551"]' \ | jq '[.Reservations[].Instances[].InstanceId]' [ "i-28739551", "i-3026a249" ]
- pavel_lishin 12y agoThanks!
- gatehouse 12y agoI've been using it for the exact same thing. One difficulty that I had was that you need to use the raw option, "jq -r" for string output without quotes, but I was looking at the query language documentation and not finding it. EDIT: here is one I used. Given VOLUME_ID, gets the most recent snapshot-id. SNAPSHOT_ID=$(\ aws ec2 describe-snapshots --owner-ids xxxxxxxxxxxx --output json | jq -r " .Snapshots | sort_by(.StartTime) | reverse[] | select(.VolumeId==\"$VOLUME_ID\" and .State==\"completed\") | .SnapshotId ") EDIT2: now that I'm looking at it, I think that I expected it to work with multiple snapshots but I'm not sure that I tested it.
- mentat 12y agoI've been using this to transform the output of ElasticBeanstalk environment queries into things I can feed into environment updates since moving between versions is a PITA. Should post that code sometime...
- skywhopper 12y agoI've been using this tool to build a bash library for a JSON/REST API and I can attest to its usefulness. Filter huge data sets down to just the fields you want, mutate, select, search, filter. Most everything I've wanted to do this can handle. Plus it's fast and dead simple to install. Kudos to the devs for this fantastic tool that I hope will soon be part of the default CL toolset across the board.
- Tyr42 12y agoHere's a cute little jq script I used to munge some json around. https://gist.github.com/tbelaire/d00cb77fb0db7b38bed2 https://gist.github.com/tbelaire/d00cb77fb0db7b38bed2
- reinier_s4g 12y agoAwesome stuff, thanks! Will give it a try really soon.
- slapresta 12y agoI can't stress how useful this is. I use it to process AWS SQS messages for debugging. Incredibly powerful tool. I have also used it to validate sudokus, because I can. https://github.com/slapresta/jq-sudoku https://github.com/slapresta/jq-sudoku
- kolev 12y agoI've been using it for over a year - especially with AWS CLI, but, I'm a little displeased that it only has the query feature (duh) and does not have a little more like merging JSONs (I think it only has appending). Another issue is that I wasn't able to find release notes for v1.4 - what's new since v1.3? There other similar tools, but they require node, Python packages, etc. jq really is installed with one command and no dependencies.
- lugg 12y agoYou should be able to do some pipe fu and carry out some form of merging via bash. I mostly use it for making our output more readable or producing CSV style reports with the filtered data. Its filters are probably the biggest feature, at least in my mind.
- kolev 12y agoSorry for not being clear. I meant a merge in the form of {"a": 1, "b": 2, "c": 3} merged with {"b": 20} returning {"a": 1, "b": 20, "c": 3}. Like Python's update() or PHP's array_replace_recursive() although PHP's a little richer.
- cryptonector 12y agojq 'reduce .[] as $obj ({}; .+$obj)' or jq -s 'reduce .[] as $obj ({}; .+$obj)' should do it. `+` adds objects by merging them.
- kolev 12y agoThanks! I'm eager to try this and hopefully clean up my code!
- slapresta 12y agoIf you can code it using jq itself, you can submit a pull request and have it added to the builtins.
- 12y ago
- glesica 12y agojq is written in portable C, and it has zero runtime dependencies. Thank you! When I clicked on the link I was fully expecting it to be written in Node and I was going to cry.
- Xavi 12y agoWhy is having node as a dependency such a bad thing? No snark. Genuinely curious.
- aendruk 12y agoThe subsequent sentence suggests that the benefit is convenience of impromptu deployment: > You can download a single binary, scp it to a far away machine, and expect it to work.
- ajtaylor 12y agoI don't think that node is bad, but it's simply another dependency you have to worry about. Most devs likely won't have node pre-installed. Having a no-dependency binary is a great thing.
- cryptonector 12y agoNode is great, and a port of jq to node would be awesome (someone once hinted at such a port, but I've lost track of it). But jq also has libjq, a C library. And jq uses a copy-on-write, reference counted representation of JSON values, which is, for example, inherently thread-safe (though jq isn't using atomic operations for refcount management yet). The library is easy to use and powerful. Ultimately the main thing I love about jq (and why I contribute to it) is the jq language itself. It reminds me of my one-time favorite, Icon. But in a world where C is still the champion of systems programming (until Rust takes over?), it's real handy to have a C JSON library _and_ a functional DSL that's easy to invoke from C.
- slapresta 12y agoOne of jq's maintainers (nicowilliams) often mentions Icon as a source of inspiration.
- chrismonsanto 12y agoThis is a great tool. I've longed for an equivalent for YAML.
- lugg 12y agoBeen using jq for a while now. Probably since the last hn post. My biggest gripe with jq is the name and asking for help on stack overflow - the jq tag is an alias to the jquery tag. Someone with tag edit level rep should really fix that.
- yincrash 12y agoYou can probably make a post on meta stack about it
- cryptonector 12y agohttp://meta.stackoverflow.com/search?q=jq http://meta.stackoverflow.com/search?q=jq
- yincrash 12y agothe system works!
- davidbanham 12y agoThis script drops you into a coffescript REPL with the json loaded into memory under the variable j. I find it nice to be able to loop the data, define new subsets, filter it with functions, etc. It was a 10 minute slap-job to get it together, so it does barf on particularly large datasets. The obvious reason is that it serialises the JSON into an argument for a child process. Shouldn't be too hard to resolve if anyone feels like it. https://github.com/davidbanham/dotfiles/commit/0ea950373604e74f71c44fe99c8758572dc656db https://github.com/davidbanham/dotfiles/commit/0ea950373604e...
- minaguib 12y agoIf you get and build latest from master, you get the --sort-keys commandline option And just like that, you have an amazing base for diffing json documents of arbitrary complexity: vimdiff <( cat doc1 | jq --sort-keys . ) <( cat doc2 | jq --sort-keys . )
- cryptonector 12y agoIt's in 1.4!
- samuell 12y agoCool, just grabbed it with sudo apt-get install jq