8 ms·
Jq – Work with JSON on the command line
- preek 10y agoThis is actually awesome for demos in lectures! Great to have better tooling for such a regular task on the command line.
- haswell 10y agoIt's always surprising when I learn that this isn't more widely known - it's such a core part of my daily workflow! The links to the project aren't immediately obvious, so here they are: https://stedolan.github.io/jq/ https://stedolan.github.io/jq/ https://jqplay.org/ https://jqplay.org/
- dozzie 10y agoIt's always surprising when I learn that people think jq is a rich tool. https://metacpan.org/pod/App::RecordStream https://metacpan.org/pod/App::RecordStream
- deleted 10y ago[deleted]
- kbenson 10y agoI found, as a programmer, going a little less specific is more versatile. ojo[1], which I've compared to jq before[2], works well for this. 1: https://metacpan.org/pod/ojo https://metacpan.org/pod/ojo 2: https://news.ycombinator.com/item?id=6419479 https://news.ycombinator.com/item?id=6419479
- jonhohle 10y ago+1 for RecordStream
- setheron 10y agoI remember using recs at Amazon. Is that where it originated ?
- dozzie 10y agoI don't know, but supposedly Amazon is somewhat hostile to publishing open source, so I doubt it.
- wellsjohnston 10y agoI made a gui for it, if you're interested https://github.com/wellsjo/JSON-Splora https://github.com/wellsjo/JSON-Splora
- gizmo385 10y agoJust tried this! Really cool! Is there any way that you could allow for the minify and format commands to be accessible from the in-GUI command-line?
- wellsjohnston 10y agoThanks :) https://github.com/wellsjo/JSON-Splora/issues/38 https://github.com/wellsjo/JSON-Splora/issues/38
- asadm 10y agoAmazing tool. Thanks!
- seanp2k2 10y agoLove jq and https://paw.cloud https://paw.cloud is another of my favorite tools for working with APIs, when curl | jq . Isn't enough.
- chewmieser 10y agoThis basically becomes a necessity when working with the AWS cli. Great tool!
- empath75 10y agoThe aws cli tool basically has this built in with --query
- koolba 10y agojq is awesome. While it can do a lot of really cool processing to filter out nested details, the majority of my usage of it is: $PROGRAM_THAT_OUTPUTS_JSON | jq . Which simply pretty prints the JSON input with two spaces per indentation. Also, $PROGRAM_THAT_OUTPUTS_JSON is usually a script that simply outputs the clipboard (alias for "xclip -o").
- grayrest 10y agoMy usage is generally pretty simple as well but if people are looking for a more complex example, I wrote one yesterday for putting together some test data. Turns DynamoDB responses into newline separated flat arrays: pbpaste | jq -c '.actionResponses[].data.items[] | [.attributes[] | .stringValue // (.numberValue | tonumber)]'
- zyang 10y agoRecently discovered jq and it has been indispensable part of my toolbox. A lot of json tasks I used to write scripts for, such as filtering and multi file concat, are now possible for a short command.
- dekhn 10y agoi want to love this tool but it's almost impossible to do anything complicated. the docs have a lot of examples but they sort of stop sort
- ajmurmann 10y agoI've been using jq quite a bit lately and agree that the docs would benefit from more complex examples. However, I've been able to get some fairly complex solutions done with jq. Do you have a concrete example of something you are trying to do?
- teej 10y agoI have server logs in jsonlines format in a file. My goal is to display some basic information (URL, referrer, time stamp, IP address, and user agent) for each log line with a particular header set to a specific value. Headers are in $.request.headers and are an array of key-value pair arrays. Figuring out this use case from the jq docs or jqplay has been a major struggle for me. I feel that this use case reflects 90% of what I would want to use jq for. I feel that if I can't get over this hump just from docs, there's no way I could justify bringing this tool on board with my team.
- svs 10y agoThe JQ stackoverflow community is VERY helpful.
- ekimekim 10y agoSounds like you're after the select() filter, which evaluates a boolean expression and passes the input to the output only if the expression is true. So your example would be something like: select(.request.headers|map(.[0] == "X-My-Header" and .[1] == "my target value")|any)|<pick out some keys to display> I realise though that your point wasn't the specific example, it's that the docs for these kind of cases are poor, and you aren't wrong. I guess my main advice is to not think of jq as something like cut or sort - instead, it's more like awk or sed. You can do lots of crazy things with sed, but it's not immediately obvious how just from reading the man page. These kinds of tools require a little more time investment but are very powerful. (edit for code typo)
- edmundhuber 10y agoHi, I wrote a similar tool: https://github.com/edmund-huber/jsonq https://github.com/edmund-huber/jsonq
- jedisct1 10y agoAn alternative that completely replaced jq for me: rq https://github.com/dflemstr/rq https://github.com/dflemstr/rq Not limited to JSON, and super fast.
- azdle 10y agorq is super great, it's been indespsible for me while working with CBOR. It's also way more useful than jq if you want to filter down your data. Some examples: ~> curl -s http://httpbin.org/ip | rq { "origin": "184.100.112.11" } ~> curl -s http://httpbin.org/ip | rq 'at origin' "184.100.112.11" ~> curl -s http://httpbin.org/ip | rq 'at origin | split "." | spread | drop 3' "11" ~> curl -s http://httpbin.org/ip | rq -C ?foriginn184.100.112.11⏎ (These are a bit contrived, but that was the only public API that returns JSON I could think of.) The only thing I haven't figured out is how to unquote string elements once filtered down to just that string.
- dflemstr 10y agoI'm the author of rq, and it's great to hear about your use-cases! Feel free to submit a feature request about the raw string use-case, I've been thinking about it for a while but some people have wanted me to add line support (maybe -l and -L?) and some people have wanted raw text support (maybe -r and -R?) while other people still have been talking about TSV/CSV support (not sure how that would look tbh...). I'm not sure what to do exactly, so feel free to tell me!
- danso 10y agoAw dang, I wish I knew about this earlier. May have made teaching command-line data processing to beginners quite a bit easier. Example lesson using Spotify API (which is still public and free): http://www.compciv.org/recipes/data/touring-the-spotify-api/ http://www.compciv.org/recipes/data/touring-the-spotify-api/ (just noticed that you used the Spotify API too for the demo :) https://github.com/dflemstr/rq/blob/master/doc/demo.md https://github.com/dflemstr/rq/blob/master/doc/demo.md) I personally like using jq but not sure how easy-to-grok it is for people who are new to both the command-line and to serialized data structures in general. Will have give rq a spin (waiting for it to compile on my machine) but it looks quite nice, at least for my own uses, which heavily involve YAML. In terms of TSV/CSV support, you may want to checkout csvjson, which is part of the csvkit command-line suite: http://csvkit.readthedocs.io/en/0.9.1/scripts/csvjson.html http://csvkit.readthedocs.io/en/0.9.1/scripts/csvjson.html csvkit is my go-to tool for data, so much that I rarely ever use SQL for anything other than when I actually need a database. I suppose if rq were to be my go-to data-processing tool, I'd probably do this: csvjson mydata.csv | rq ... Unless I'm missing it in the documents, there doesn't appear to be a way to convert to CSV? That would be helpful. FWIW, csvkit has a tool named in2csv, which will read from line-delimited JSON: curl -s 'https://api.spotify.com/v1/search?q=rihanna&type=track&limit=50' \ | rq 'at "tracks.items" | spread | flatMap artists' \ | in2csv -f ndjson http://csvkit.readthedocs.io/en/stable/scripts/in2csv.html http://csvkit.readthedocs.io/en/stable/scripts/in2csv.html
- agumonkey 10y agojq embeds fp bits, you can define functions too, crazy.
- nikolay 10y agoIf you use AWS CLI [0], you can embed JMESPath [1] queries via the --query switch, i.e. you don't ned jq. I was a loyal supporter of it, but switched to JMESPath [1], and I love its query languages more. [0]: https://aws.amazon.com/cli/ https://aws.amazon.com/cli/ [1]: http://jmespath.org/ http://jmespath.org/
- svs 10y agoJMESPath is nowhere near us powerful. No unique(), for example, which is very important when munging JSON
- nikolay 10y agoOkay, but in 99.9% of the cases it's powerful enough.
- Slippery_John 10y agoAzure also uses this in their new CLI [0]. The biggest advantage of JMESPath over jq is that JMESPath is a spec, and thus can be implemented in several languages [1]. [0]: https://github.com/Azure/azure-cli https://github.com/Azure/azure-cli [1]: http://jmespath.org/libraries.html http://jmespath.org/libraries.html
- dasil003 10y agojq is awesome, but what stands out to me in the article is the use of curl. You can shrink those ~200 bytes of curl by 2/3rds using HTTPie(https://httpie.org/ https://httpie.org/): http put your.api.endpoint email=your@email.address password=swaggerrocks HTTPie and jq go together like peanut butter and chocolate, except without the caloric bloat.
- paulddraper 10y agoAnd swift death to those with allergies.
- abraae 10y agoI wish jq was available from Java. We built a microservice recently that was just an aggregator of a bunch of other api calls. Jq would have been perfect.
- wryun 10y agojmespath is: http://jmespath.org/libraries.html http://jmespath.org/libraries.html (also, a nicer language IMO, even if it has slightly fewer features)
- kbenson 10y agoSome prior submissions: 2 years ago: https://news.ycombinator.com/item?id=9446980 https://news.ycombinator.com/item?id=9446980 (91 comments) https://news.ycombinator.com/item?id=7895076 https://news.ycombinator.com/item?id=7895076 (55 comments) 4 years ago: https://news.ycombinator.com/item?id=4679933 https://news.ycombinator.com/item?id=4679933 (36 comments) https://news.ycombinator.com/item?id=4985250 https://news.ycombinator.com/item?id=4985250 (10 comments) https://news.ycombinator.com/item?id=5734683 https://news.ycombinator.com/item?id=5734683 (29 comments)
- a3n 10y agoLooking at the jq github repo led me to also find ag, an ack-grep replacement. Both are in my distro's repositories. https://github.com/stedolan/jq https://github.com/stedolan/jq https://github.com/ggreer/the_silver_searcher https://github.com/ggreer/the_silver_searcher "The command name is 33% shorter than ack!"
- baq 10y agoSkip ag and go straight to rg (ripgrep).
- idsout 10y agoNow check out ripgrep (command `rg`) https://github.com/BurntSushi/ripgrep https://github.com/BurntSushi/ripgrep, the ag/ack killer ;).
- danso 10y agoI like ag, but unless I'm missing something, ack has this killer flag which ag lacks: --output=expr Output the evaluation of expr for each line (turns off text highlighting) If PATTERN matches more than once then a line is output for each non-overlapping match. For more information please see the section "Examples of --output". Basically, it allows you to use captured groups in the output: cat file | ag '(\w+) (\d+)' --output '$1 is $2 years old' On the other hand, ag allows multiline matching. So I end up using ag and ack together frequently.
- kbenson 10y agoRelated HN discussion, but more encompassing of things you might need to do on the command line with data: Command-line tools for data science (2013) [1] 1: https://news.ycombinator.com/item?id=6412190 https://news.ycombinator.com/item?id=6412190
- foota 10y agoSeems like you could totally write a graphql command line utility to filter through json output.
- vemv 10y agoMy jq killer: function ruby_json { ruby -e "require 'json'; require 'active_support/all'; puts JSON.parse(STDIN.read)$1" } Performs/prints arbitrary ruby over a readily available representation of the JSON object coming from stdin. Example: curl the_endpoint | ruby_json "['funnels'].map{|a| puts a.to_i * 2 }"
- svs 10y agoCannot accept user inputted script for json transform because Security nightmare! That's JQ's and JMESPath's secret sauce.
- DrAwesome 10y agoI actually just discovered this tool a few days ago when I wanted to do something with the Gitlab API from a bash script. It's really cool!
- deleted 10y ago[deleted]
- ericHosick 10y agoFriend wrote a feedforward neural network using jq. https://github.com/kevin-albert/jq-neural-network https://github.com/kevin-albert/jq-neural-network
- smetj 10y agoThe Wishbone event stream processing framework (http://wishbone.readthedocs.io http://wishbone.readthedocs.io) has the wishbone.flow.jq module (https://github.com/smetj/wishbone-flow-jq https://github.com/smetj/wishbone-flow-jq) to do pattern matching on JSON streams. The article http://smetj.net/processing_webhooks_using_wishbone_part_2.html http://smetj.net/processing_webhooks_using_wishbone_part_2.h... shows how it's used.
- iopuy 10y agoHey guys, I work with JSON on a daily basis on machines that don't have access to jq. My advice to you in similar situations is as follows: To pretty print JSON: cat myjson.json | python -m json.tool > prettyPrinted.json To validate json: cat myjson.json | python -m json.tool echo $? If the result is 0 it is well formed and 1 otherwise. You can also pretty print JSON by opening the scratchpad in firefox and clicking the "Pretty print" button.
- dajohnson89 10y agoWhy do you have access to Python, but not jq?
- rhubarbquid 10y agoI would guess: a policy against installing unapproved software on production servers, systems that don't have compilers installed (for security, jq appears to be C code), or debugging on client/customer machines
- scott_karana 10y agoHe might not administer (or have permission to modify) the machine, and Python ships with most distros.
- 13of40 10y agoI'm sure someone's going to throw a shoe at me for this, but...this sort of thing is why you want an object-oriented shell. PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]' PS C:\> $x.banana.peel yellow
- awssailin 10y agoI don't want an object oriented shell but I have on occasion found jq to be very useful.
- fabianfabian 10y ago$ echo '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]' | jq '.[].banana.peel' "yellow"
- 13of40 10y agoCome on now, you're missing the point. If the lingua franca of your shell is objects instead of text, you can import from whatever format you want and work with it in high fidelity: PS C:\> $x = [XML]"<banana><stem>brown</stem><peel>yellow</peel><flesh>white</flesh></banana>" PS C:\> $x.banana.peel yellow PS C:\> $x = ConvertFrom-Json -InputObject '[{"banana" : {"stem" : "brown", "peel" : "yellow", "flesh" : "white"}}]' PS C:\> $x.banana.peel yellow PS C:\> $x = New-Object PSObject -Property @{banana = New-Object PSObject -Property @{stem = "brown"; peel = "yellow"; flesh = "white"}} PS C:\> $x.banana.peel yellow PS C:\> $x = @{banana = @{stem = "brown"; peel = "yellow"; flesh = "white"}} PS C:\> $x.banana.peel yellow
- fabianfabian 10y agoGood point, thanks for clarifying.
- taeric 10y agoI won't throw shoes. However, I do prefer anything that forces a commands output into text. Which JSON at least gets that right. Specifically, if I have something that output objects, it is possible (and likely) that I can not persist that output to less and explore what is going on. Great if I already know exactly what I want. Terrible if I just want a visual exploration. Worse, if I can't dump it to file, I may have to constantly do some expensive operations just to find out what will give me the answer I want.
- purple-dragon 10y agoConceptually similar is Mike Bostock's wonderful ndjson-cli[0] [0]: https://github.com/mbostock/ndjson-cli https://github.com/mbostock/ndjson-cli
- s_m 10y agoA few months ago I wrote a blog post about how to use jq in conjuction with pup to debug HTML-in-JSON endpoints: https://the1x.engineer/2016/08/20/efficient-endpoint-testing-curl-jq-pup/ https://the1x.engineer/2016/08/20/efficient-endpoint-testing... jq is one of my favourite tools - can't recommend it enough.
- asamarin 10y agoI _love_ jq. It's been an incredibly useful tool for me since I discovered it ~6 months ago. However, article mentions "jq -n"; I personally find jq syntax less appealing when it comes to generating JSON instead of parsing it. For that particular task, I prefer using "jo": https://github.com/jpmens/jo https://github.com/jpmens/jo jq & jo are the Calvin & Hobbes of the shell: lots of fun together! A few examples: - Generate a simple JSON object: $ jo foo=12 bar=true baz=null {"foo":12,"bar":true,"baz":"null"} - Want a nested object? $ jo foo=12 bar=true baz=null thing=$(jo number=57 test=string) {"foo":12,"bar":true,"baz":"null","thing":{"number":57,"test":"string"}} - What about arrays? $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5) {"thing":{"number":57,"test":"string"},"anarray":[1,2,3,4,5]} - Now let's add some jq magic to sum all the values in the array together: $ jo thing=$(jo number=57 test=string) anarray=$(jo -a 1 2 3 4 5) | jq '.anarray | add' 15 Great stuff :)
- taormina 10y agoFor all of you on Macs, it's as easy as `brew install jq`. One of the first things I grab on a new box.
- bootload 10y agoQ? How does Jq handle crap (malformed) JSON? ... quickly scanned the docs, tutorial and readme. I see no FAQ.
- spinningarrow 10y agojq is really nice and I always install it as part of my tool chain. However I don't use it very frequently because of which I can never remember the syntax which imho is not very intuitive :(
- Zombieball 10y agoAnyone here used jsawk that could compare and contrast to jq?
- zimbatm 10y agoFor shell scripting, jshon is also very useful ( http://kmkeen.com/jshon http://kmkeen.com/jshon ).
- telmich 10y agoVery nice!