20 ms·
Jq – A Command-line JSON processor
- deleted 11y ago[deleted]
- xahrepap 11y agoI've been using this for a couple months now. With httpie[1], it's become as essential to my work day as Grep. [1] https://github.com/jakubroztocil/httpie https://github.com/jakubroztocil/httpie
- discardorama 11y agoWow, thanks for that link. I hadn't heard of httpie, but it looks pretty slick.
- yunong 11y agoIsn't that the same as curl(1)?
- mugsie 11y agoExcept much nicer - especially if you work with json APIs
- jkbr 11y agoNope. HTTPie is much, much more focused than cURL (the CLI tool). It is exclusively an HTTP client designed primarily for interactive use. cURL, on the other hand, is meant to be used mostly non-interactively and is more of a general client for "URLs" (it also supports DICT, FILE, FTP, FTPS, GOPHER, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP). HTTPie's goal is user-friendliness, so it comes with features like built-in JSON support, output formatting, syntax highlighting, extensions, etc. Comparison of the two in one screenshot can be seen in the README: https://github.com/jakubroztocil/httpie#readme https://github.com/jakubroztocil/httpie#readme (Disclaimer: I created HTTPie)
- giovannibajo1 11y agoI use and love httpie but I start to regret that it's written in Python as the Python platform is a very very late adopter of modern TLS standards. Even today, in 2015, it's still a hurdle to use httpie with a SNI endpoint, and forget about it making it talk to a ECDSA-based endpoint like those exposed by the free tier of Cloudflare (so called "modern TLS"). I wonder how long it will take to HTTP2 to land. I tried a random rewrite of httpie in Go and it worked perfectly with those endpoints, as expected, and Go is getting full HTTP2 with 1.5 in June. Too bad the rewrite was still feature incomplete, when I tried it.
- jkbr 11y agoThe situation is not ideal but more recent versions of Python (2.7.9 and 3.4) have a much improved TLS support and things like SNI work out of the box. For older Python versions there are some additional dependencies to be installed manually (will happen automatically in next version of HTTPie): https://github.com/jakubroztocil/httpie#sni-server-name-indication https://github.com/jakubroztocil/httpie#sni-server-name-indi... As for HTTP/2, there is an experimental extension for now: https://github.com/jakubroztocil/httpie-http2 https://github.com/jakubroztocil/httpie-http2
- xahrepap 11y agoServes the same purpose, for sure. However, I think the interface is more intuitive/clean for json rest apis. I usually end up using HTTPie for manual work / ad-hoc checks. And then when I script it I tend to end up using curl, because curl is available everywhere and I never need to worry if it's installed everywhere I'll want to the script to run.
- _lce0 11y agoI had the same feelings but gave it a try .. it works great, and you can construct http(s) request more naturaly, like.. http POST example.org name=jhon pass=12345 which results in .. POST / HTTP/1.1 Accept: application/json Accept-Encoding: gzip, deflate, compress Content-Length: 33 Content-Type: application/json; charset=utf-8 Host: example.com User-Agent: HTTPie/0.8.0 { "name": "jhon", "pass": "12345" } ..everything beautifully printed in nice colors thanks for sharing!
- cgag 11y agoI found httpie last week when I started to think I should write a replacement for curl with a much better UI.
- xcombelle 11y agowhat is your work is about ?
- xahrepap 11y agoI do DevOps for a medium sized SaaS company (~500 employees). So often working with scripts, and JQ is a lifesaver any time a bash script needs to grab data from a REST api. Before it was grep/sed/etc. and the scripts would turn ugly quick. Now you get an at-a-glance understanding what the script is trying to grab from the endpoint.
- themartorana 11y agoLots of previous discussions about this on HN (for instance [0]) but still a sweet tool. [0] https://news.ycombinator.com/item?id=5734683 https://news.ycombinator.com/item?id=5734683
- AdrianRossouw 11y agoi love it so much. i've used it for things like migrating between json schemas, and even converting normal json into elasticsearch bulk imports on the fly.
- andrewd18 11y agoLove JQ, been using it for the past year. I'm very much looking forward to the regex support in 1.5.
- useerup 11y agojq is awesome, a real achievement. At the same time it demonstrates everything that is wrong with traditional shells and what PowerShell gets so right. jq is not a "Unixy" tool in the sense that it should do one thing and do it right. jq implements its own expression language, command processor complete with internal "pipelines". Why would a tool need to do that? find is another utility that does many, many other things than to "find" items: It executes commands, deletes objects etc. Consider this challenge that included parsing json, filtering, projecting and csv-formatting output: https://news.ycombinator.com/item?id=9438109 https://news.ycombinator.com/item?id=9438109 Several solutions uses jq - to good effect. But the PowerShell solution uses PowerShell expressions to filter, sort and project items. The problem is - at the core - that the traditional command tools are severely restricted by only being able to rely on a text pipeline convention. You cannot parse json and send the "objects" along to another tool. Well, you can, if the json tree is extremely basic - like 2 levels. PowerShell also has a json parser tool: https://technet.microsoft.com/en-us/library/hh849898.aspx https://technet.microsoft.com/en-us/library/hh849898.aspx. It is called ConvertFrom-Json (follows the verb-noun convention) and is distributed as part of PowerShell (built-in if you want - but that's a misnomer for PowerShell modules). It is extremely simple - it doesn't even take any parameters - it just converts a string (parses it as json) and outputs the object/objects which can be arbitrarily complex. It truly does only one thing. If you want to select specific properties, sort, group etc you do it with other PS tools like select (alias for Select-Object), sort (alias for Sort-Object), group (alias for Group-Object) etc. If you want to convert back to json you use ConvertTo-Json.
- kbenson 11y agoWhile jarring when I first used PowerShell, once I realized what it was doing, I immediately understood it as a powerful concept. It allows shell pipeline processing to work in much the same way you can filter lists of hashes/dicts/arrays in more traditional scripting languages. I haven't used it much, and not at all in the last few years as I'm not often on Windows boxes, but I've always remained impressed.
- fragmede 11y agoDepends how you look at it. jq is a "Unixy" tool in the sense that it should do one thing and do it right - and that 'one thing' is parse json data to output it in the desired format. Awk, arguably one of the oldest unix tools, also has its own language. Would you consider it non-Unixy or is it grandfathered in due to age?
- misiti3780 11y agoone of the most useful command lines tools in my toolbox these days - thank you for all that have contributed to this masterpiece.
- piva00 11y agoUsing it for parsing AWS CLI responses, it's been a sweet tool to parse those JSON responses from AWS, most of the time I'd fire some python script to parse the data and now with jq I can keep most of the one-time scripts limited to bash only, if I need something more maintainable or longer living then I'll drop a script in python or ruby.
- bramgg 11y agoThat's exactly what I did with Pouch (https://github.com/bramgg/pouch https://github.com/bramgg/pouch).
- Terretta 11y agoJmesPath is built into the AWS CLI. You don't need separate JQ. See the --query option: http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html http://docs.aws.amazon.com/cli/latest/userguide/controlling-... Check out http://jmespath.org http://jmespath.org for details.
- piva00 11y agoOnly saw your comment today, thanks so much for this, I really overlooked this part from the docs.
- shmerl 11y agoI'm using it in bash scripts - it's a very good tool.
- yunong 11y agoFor a similar tool written in javascript, check out json (https://github.com/trentm/json https://github.com/trentm/json). I use both.
- devy 11y agoJq is truly amazing swiss army knife for JSON viewing/processing in command line. Btw, I've seen people posting this github link at least 10 times in the past. I thought HN filters dupes, what gives? https://hn.algolia.com/?query=http:%2F%2Fstedolan.github.io%2Fjq%2F&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=http:%2F%2Fstedolan.github.io%...
- kbenson 11y agoI think it filters dupes within a certain time frame. You don't want important front-page news at a site being considered a dupe because something was big on the front page a year ago.
- devy 11y agoGotcha. In this particular case though, it does feel like every few months someone will remind us there is this great tool some of us have been already using for 2 years now:)
- Zecc 11y agoFWIW, I hadn't heard of it before. (actually that's a lie, I saw it mentioned in the CSV challenge thread. But that was just a few days ago, so the point still stands)
- feld 11y agoThis appears to be in FreeBSD ports already, so just pkg install jq and you can have it :)
- klinquist 11y agoI love JQ. I use it along with Ansible's AWS dynamic inventory script to manage my instances.
- XorNot 11y agoHow weird - I just started using this for some simple scripting against a rest interface. Its a great tool.
- platz 11y agoI recently had to consume an xml service; it would've been great to have something like Jq for xml. For some reason I wasn't really interested in xml -> json conversion as a first step. I also found a bunch of xml cli tools, but they are older, had to learn things like XQuery (which actually few tools today seem to support).
- xaduha 11y agoWell there's XProc. But as with everything XML - you shouldn't really edit it by hand without proper editor support at the very least (preferably DTD/XML Schema aware).
- platz 11y agoI understand XML has namespaces, but I'm not sure what "edit by hand" means in this context. The tools I found previously were xqilla and xmlstarlet, but they seem pretty old and just not as convienient as Jq. I still think there is space for an XML cli tool that is more convenient for the majority of common tasks working with XML, at simply treating the data as a hierarchical structure, without requiring an interface that is full fully-qualified/needs XSLT/etc..
- kstrauser 11y agoFor me, that "some reason" is normally the overlap between children and parameters: <foo bar="123" baz="456"><qux>789</qux></foo> There's not a great round-trip mapping of that to JSON, like: {"foo": {"bar": 123, "baz": "456", "children": {"qux": 789}}} (which also illustrates why I dislike XML being untyped unless you have a schema file to interpret it with).
- jeroenjanssens 11y agoI have found `xmlstarlet` to be really useful for working with XML. You can achieve a lot with some basic XPath expressions. No need to learn XSLT. (I should actually update my book Data Science at the Command Line to include this tool.)
- drdoom 11y agoLooks interesting. Given this json, [{"name":"john", "grades":[1,2,3]}, {"name":"jane", "grades":[4,5,6]}], can it print it out as this: john,1 john,2 john,3 jane,4 jane,5 jane,6
- aquassaut 11y agoYes, kind of like so: echo '[{"name":"john", "grades":[1,2,3]}, {"name":"jane", "grades":[4,5,6]}]'| jq '.[]| "\(.name), \(.grades|.[])"'
- beamatronic 11y agoI appreciate jq and use it almost every day. At the same time, I recently had a task that was easier to do in Node.js. I had a file containing a JSON object with a structure like so: [ { }, { }, ... ] and I wanted to turn each object into a separate file.
- TheDong 11y agoThat task is trivially accomplishable in jq. I have no clue why you say it's "easier" in nodejs. i=1 for line in `echo '[{"a":"b"},{"c":"d"},{"e":"f"}]' | jq '.[]' -c`; do echo $line | jq '.' > jq_out_$i.json i=$((i+1)) done The above prints each element of the array to a file named by its index, including pretty-printing it. If you use it every day, the above should have come to you easily enough.
- danso 11y agoLove, love jq...after awhile, I realized I was twisting my brain too much to get bash to do what it wasn't meant to do, but I got a lot of mileage of just playing with APIs because of how jq made it so easy. I came up with some fun lessons on how to use jq and other CLI-tools to tour the Spotify API [1] and to mash up images from Instagram [2] [1] http://www.compciv.org/recipes/data/touring-the-spotify-api/ http://www.compciv.org/recipes/data/touring-the-spotify-api/ [2] http://www.compciv.org/recipes/data/api-exploration-with-gmaps-instagram/ http://www.compciv.org/recipes/data/api-exploration-with-gma... (note: while I say that I like the command-line, I'm not die-hard enough to have stopped to learn awk, so take that fwiw)
- doki_pen 11y agoNice. I made jipe[0] a while back to handle common troubleshooting tasks with streams of json objects. I still use it quite a bit. I didn't see anything about streaming in the jq docs. [0] https://github.com/dokipen/jipe https://github.com/dokipen/jipe
- slapresta 11y agoIt does streaming. You can pass a stream of JSON objects through standard input and jq will process them on stream. The master branch also has experimental support for JSON sequences.
- aggregator-ios 11y agoFor those on OS X, check out JSON Query: https://itunes.apple.com/us/app/json-query/id953006734?mt=12 https://itunes.apple.com/us/app/json-query/id953006734?mt=12 It's not nearly as powerful as jq, but for simple querying and downloading JSON (with POST body and HTTP headers), I find it quite useful. Disclaimer: I built it.
- lectrick 11y ago"brew install jq"
- blacksqr 11y agoCompare: rl_json -- Extends Tcl with a json value type and a command to manipulate json values directly. The commands accept a path specification that names some subset of the supplied json entity. The paths used by [json] allow indexing into JSON arrays by the integer key. https://github.com/RubyLane/rl_json https://github.com/RubyLane/rl_json
- rmgraham 11y agoI know it is right on the linked page, but it is useful enough I think it deserves a direct link in any conversation: https://jqplay.org/ https://jqplay.org/
- _RPM 11y agoI love that it is written in C and not Go, or Rust, Java
- deleted 11y ago[deleted]
- jimmy0x52 11y agoNo mention of jsawk in here? We use it quite a bit for shell scripts, works great.
- kalehrishi 11y agoWe are using jq to parse JSON output from various API in shell scripts. It served us well even for large JSON strings. If you want to write small scripts and parse some JSON in it then jq is best fit.