17 ms·
jq 1.7
- paradox460 3y agojq and miller[1] are essential parts of my toolbelt, right up there with awk and vim. [1]: https://github.com/johnkerl/miller https://github.com/johnkerl/miller
- porridgeraisin 3y agoThanks for pointing me to miller. Too good.
- reegnz 3y agoLet me drop my jq-zsh-plugin here: https://github.com/reegnz/jq-zsh-plugin https://github.com/reegnz/jq-zsh-plugin It's a line-editor, and allows me to query json documents ultra-fast. It optimizes for a really fast feedback.
- cryptonector 3y agoWoohoo!! Finally! It's really awesome how the community pulled together and helped us recruit new maintainers to revive the project. Special thanks to, well, all involved, but especially @stedolan, @itchyny, and @owenthereal (all GitHub usernames).
- saziba 3y agoI can't stress enough how I love jq. It saved my life countless times as data engineer. Good job guys!
- mtmail 3y agoRelated last month "First release of jq in 5 years" https://news.ycombinator.com/item?id=36951830 https://news.ycombinator.com/item?id=36951830 Huge fan, I use it all the time.
- nikolay 3y agoI love jq, but I also use JMESPath (especially with AWS CLI), yq (bundled with tomlq and xq as well), and dasel [2]. I also wish hclq [3] wasn't so dead! [0]: https://jmespath.org/ https://jmespath.org/ [1]: https://kislyuk.github.io/yq/ https://kislyuk.github.io/yq/ [2]: https://github.com/TomWright/dasel https://github.com/TomWright/dasel [3]: https://hclq.sh/ https://hclq.sh/
- rmwaite 3y agoI really like the JMESPath interactive tutorial page (https://jmespath.org/tutorial.html https://jmespath.org/tutorial.html). It helped me when I was first learning the syntax and I still go to it if I run into a particularly weird syntax that throws me off.
- js2 3y agoMake JSON greppable! https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron I've been using `jq` for years and I'm always able to cobble together what I need, but I have yet to find it intuitive and I'm rarely able to arrive at a solution of any complexity without spending a lot of time reading its documentation. I wish I found it easier to use. :-(
- lathiat 3y agoI also love gron, if nothing else to find the paths I need to use with jq later. But ChatGPT has genuinely solved my suffering writing jq, it does a pretty good job. It even almost replaces gron, if you feed it an exmaple json and ask for jq, it gives you something. It usually needs a little adjusting but it gets me 90% of the way there and saves me a bit of time. I rarely use it for much else but its a jq winner :)
- cryptonector 3y agojq can be used trivially to approach gron-like output: jq -r 'paths(scalars) as $p | getpath($p) | "\($p|join(".")) = \(.)"' See elsewhere in this subthread for a full gron implementation in jq.
- gobdovan 3y agoThought this is an array programming language considering the name's made out of J and Q. Nice util
- fiddlerwoaroof 3y agoThe way it works is sort of reminiscent of array programming.
- jfoutz 3y agojq is great. I don't know how many times I've had to explain to engineers the invalid numeric literal error means your json is bad. no really, don't trust me, copy it into the ide. it's not jq. your message is malformed. Strangely, I also have ECMA-404 and RFC8259 open in other tabs. mostly annoyance with the occasional flashes of anger over number formats and duplicate keys.
- d_t_w 3y agoThis is great, JQ is brilliant. I love JQ so much we implemented a subset of JQ in Clojure so that our users could use it to munge/filter data in our product (JVM and browser based Kafka tooling). One of the most fun coding pieces I've done, though I am a bit odd and I love writing grammars (big shoutout to Instaparse![1]). I learned through my implementation that JQ is a LISP-2[2] which surprised me as it didn't feel obvious from the grammar. [1] https://github.com/Engelberg/instaparse https://github.com/Engelberg/instaparse [2] https://github.com/jqlang/jq/wiki/jq-Language-Description#:~:text=The%20jq%20language%20is%20a,their%20use%20after%20being%20deallocated https://github.com/jqlang/jq/wiki/jq-Language-Description#:~....
- chatmasta 3y agoI can't stand jq. I realize this is an unpopular opinion, and our codebase at work has plenty of jq in the bash scripts, some of it even code that I wrote. I begrudgingly use it when it's the best option for me. But something about it rubs me the wrong way - I think it's the unintuitive query syntax and the need to search for every minute step of what I'm trying to do, and the frequency with which that leads to cryptic answers that I can only decipher if I am some sort of jq expert. But I have this instinctive reaction to all DSL languages that embed themselves into strings, like htmx and tailwind (both embedded in attribute string values). I realize some people like it, and it's a well-made piece of software, and I will even admit that sometimes there is no better choice. But I guess I just hate that it's necessary? I guess I could also admit it's the least-bad option, in the sense that it's a vast improvement over various sed/awk/cut monstrosities when it comes to parsing JSON in bash. Certainly once you find the right incantation, it's perfect - it transforms some raw stdin into parsed JSON that you can manipulate into exactly what you need. But for me, it ranks right next to regex in terms of "things I (don't) want to see in my code." I hate that the jq command is always some indecipherable string in the middle of the script. The only real alternative I've ever used is piping to a Python program that I define inline in a heredoc, but that ends up being at least as nasty as the JQ script.
- michaelcampbell 3y ago`jq` and `GNU Parallel` share a world in my brain where I know they're wonderful tools, but I spend more time grokking the syntax of each one as rarely as I need either, than just writing a bash/sed/awk/perl, ruby, or python script to do what I need.
- tambourine_man 3y agoIf you’re like me and use jq infrequently enough that you have to consult the documentation every time, try gron. It’s greppable json https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- Galanwe 3y agoDamn thats incredibly well thought! You can even `gron | grep | sed | gron -u`. Awesome tool thanks for sharing.
- zimpenfish 3y agoI do this for archiving my YouTube watchlist into git - all the video/thumbnail/subtitle URLs are dynamic with expiry which makes them pointless to keep around. A quick `gron | sed | gron -u` replaces them all with "URL" and my diffs are much smaller and happier now.
- xiphias2 3y agoIf you like gron, try fastgron :) It supports everything gron supports, but 50x faster.
- preommr 3y agoWow, that's a pretty cool - so simple but so useful. I'll also share that I've been using `curl cheat.sh/jq` (cheat.sh in general is a great resource) for years. Although now I'd probably use something like chatgpt.
- jkrubin 3y agoChad is super helpful with jq!
- dewey 3y agoI was always struggling with the jq syntax...but I'm consistently impressed by ChatGPT coming up with the right commands from an example JSON.
- pyrolistical 3y agoLove jq. Solved many problems But at one point I started write long jq modules and while it was pretty straightforward, there are less people familiar with jq. So I declared jq bankruptcy and rewrote it as a nodejs script. The rest of the team was relived
- cryptonector 3y ago> I started write long jq modules If you're writing long jq modules, you probably do want a different (faster, better) language.
- nikolay 3y agoIn addition to my previous comment about jq-like tools, I want to share a couple other interesting tools, which I use alongside jq are jo [0] and jc [1]. [0]: https://github.com/jpmens/jo https://github.com/jpmens/jo [1]: https://github.com/kellyjonbrazil/jc https://github.com/kellyjonbrazil/jc
- goku12 3y agoAnd jless [1] and gron [2]. This is the first I'm hearing of gron, but adding here for completeness sake. Meanwhile, JSON seems to be becoming a standard for CLI tools. Ideal scenario would be if every CLI tool has a --json flag or something similar, so that jc is not needed anymore. [1] https://jless.io/ https://jless.io/ [2] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- dang 3y agoRecent and related: First release of jq in 5 years - https://news.ycombinator.com/item?id=36951830 https://news.ycombinator.com/item?id=36951830 - Aug 2023 (27 comments)
- wanderingmind 3y agoA slight tangential question. I use ripgrep-all along with fzf to interactively search through files from cli. Is it possible to integrate jq (or some equivalent) into this search ecosystem to search through json files?
- lordgrenville 3y agoYes https://gist.github.com/reegnz/b9e40993d410b75c2d866441add2cb55 https://gist.github.com/reegnz/b9e40993d410b75c2d866441add2c...
- black_puppydog 3y ago> Adds new builtin pick(stream) to emit a projection of the input object or array. > jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | pick(.a, .b.c, .x)' This is a godsend! Thanks to the contributors! <3
- nivekastoreth 3y agoThis is a fantastic new feature. I would also love a version of ‘pick’ that works on streaming data, since that doesn’t seem to be possible currently without reassembling the stream first
- nnutter 3y agoI just installed from Git the other day to use this feature. It’s very useful!
- wwader 3y agoIf you don't need to digg deep it's also possible to do: $ jq -n '{"a": 1, "b": {"c": 2, "d": 3}, "e": 4} | {a, e}' { "a": 1, "e": 4 }
- arwineap 3y agoThe next killer feature for postman alternatives is integrated jq support for your json output. Postman handles this especially poorly...
- cryptonector 3y agoWhat is postman?
- Omatic810 3y agoIt's an API tool: https://www.postman.com/ https://www.postman.com/
- ParetoOptimal 3y agohttps://github.com/pashky/restclient.el#usage https://github.com/pashky/restclient.el#usage
- kenrose 3y agoIt's great to see jq get some releases again. It's a big part of our check evaluation infra at OpsLevel.
- lwhsiao 3y agoI'll give a plug for jaq [0], a clone focused on correctness, speed, and simplicity. It only implements a subset of jq, but I've been enjoying it so far. [0]: https://github.com/01mf02/jaq https://github.com/01mf02/jaq
- faizshah 3y agoI wish there was a faster version of jq, I was only able to get a few mb/s throughput out of jq vs few hundred mb/s throughput out of ripgrep. I often use ripgrep to setup quick bash pipelines for rapid data analysis, would love to be able to use jq for that purpose. These days I am setting up scripts with simdjson but the cost of writing a script vs quickly setting up jq or ripgrep in a bash pipeline are orders of magnitude different.
- cryptonector 3y agoDid you try gojq or jaq?
- thiht 3y agoThese are great release notes. Looks like the project is in good hands
- ithkuil 3y agoOne of the reasons I like/tolerate jq is that it's stable, i.e scripts written for it a few years ago still work the same today. I have some code around for yq instead that keep breaking because yq keeps improving in non backward compatible ways (I didn't investigate how often yq introduced backwards incompatible changes, but the issue affected me several times in unrelated places, CI scripts or whatnot, that by their nature end up running with different versions of base tooling and update them at various pace) I was always thus grateful to the great wisdom of the jq maintainers for their understanding of the importance of backwards compatibility. I hope this announcement doesn't mean that this stability was just an accidental side product of stagnation and that once stagnation is "fixed" it will be done at the expense of stability.
- El_RIDO 3y agoReg. yq/jq incompatibility - unsurprisingly there are many projects that provide yq functionality as wrappers around jq. To list but a few: - https://kislyuk.github.io/yq/ https://kislyuk.github.io/yq/ - https://github.com/kyle-long/yq#yq https://github.com/kyle-long/yq#yq - https://github.com/up1/yq-1#yq-command-line-yamlxml-processor---jq-wrapper-for-yaml-and-xml-documents https://github.com/up1/yq-1#yq-command-line-yamlxml-processo... - https://github.com/onixspot/yq-2#yq-command-line-yamlxmltoml-processor---jq-wrapper-for-yaml-xml-toml-documents https://github.com/onixspot/yq-2#yq-command-line-yamlxmltoml... - https://github.com/simonrupf/convert2json#convert2json https://github.com/simonrupf/convert2json#convert2json
- ainar-g 3y agoWhat are you using yq for? Unless you use YAML-only features (e.g. integers, arrays, and objects as object keys), it seems like it would be easier to just pipe-convert your YAML to JSON and process it with jq.
- dividedbyzero 3y agoNot them but I use it in a few places to make automated edits to yaml files.
- donatj 3y agojq is a tool that is so powerful and useful it really should be made part of the POSIX standard. It should come with everything preinstalled. It’s an amazing tool like that of an earlier more refined age.
- kouru225 3y agoReading “JQ” in this context makes me realize how terminally online I’ve been all these years
- ta8903 3y agoMore programmers need to be aware of the jq.
- anonymoushn 3y agoA new jq release is exciting because it means that jq can finally stop mangling numbers that it does not do any math on.
- Dowwie 3y ago'jq' is a lot easier to use with ChatGPT assistance. Simply provide sample JSON and tell the LLM how you want to use JQ to parse and transform.
- VoodooJuJu 3y agoWorks well enough, but it really tripped me up at first with the syntax. Had to constantly look things up while using it. jq -r \ '.apps.http.servers.srv1.routes[0] | .match[0].header.[env.AUTH_USER_HEADER][0] = "$username"' That's an error because you can't select an env var key '[env.AUTH_USER_HEADER]' in the middle of a chain like that, only immediately following a pipe: | .match[0].header | .[env.AUTH_USER_HEADER][0] = "$username"' But then I need to preserve the parent object after the assignment and that pipe throws it out. Thankfully, parentheses fix that: | (.match[0].header | .[env.AUTH_USER_HEADER][0]) = "$username"' After working out the gotchas, it's quite powerful, like regex, but a little clunky, though not nearly as much as regex.
- rewmie 3y ago> Almost feels like I'm learning regex again. But isn't that what jq is and always has been? I mean,what led you to believe that an entirely unrelated pattern matching language would work?
- cryptonector 3y ago> jq -r \ > '.apps.http.servers.srv1.routes[0] > | .match[0].header.[env.AUTH_USER_HEADER][0] = "$username"' > That's an error because you can't select an env var key '[env.AUTH_USER_HEADER]' in the middle of a chain like that, only immediately following a pipe: > | .match[0].header | .[env.AUTH_USER_HEADER][0] = "$username"' You can use `env.AUTH_USER_HEADER` as a key the way you wanted. The issue is that you had to write `... | .match[0].header[env.AUTH_USER_HEADER] ...` -- no `.` between "header" and the index operator! This complaint is a fairly frequent one, so in fact we did "fix" this in 1.7! You can now write `.a.[0]` and it works.
- deleted 3y ago[deleted]
- crest 3y agoThe addition of raw 0-byte separated output will make reliable scripting (e.g. piping into xargs -0) a lot easier.
- ccakes 3y agoThere seems to be a few comments from people that dislike jq. If you're that way inclined, gron[0] might be more your thing. [0] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- goku12 3y agoSeeing this news today, I decided to give jq another try and ended up discovering jq-mode [1] for emacs. It doesn't just support jq filter file editing, it supports jq in org-mode and something else called 'jq-interactively'. This interactive mode allows you to apply jq interactively on a JSON or YAML (with yq) buffer. The buffer contents become the filtered value when you finish editing the jq filter with a return. This is especially impressive to see in yaml files. [1] https://github.com/ljos/jq-mode https://github.com/ljos/jq-mode
- G3rn0ti 3y agoDidn’t know this. Thanks for the tip! Personally, when I test REST APIs, I use „restclient.el“ all the time which also comes with a great JQ integration („jq-set-var“ for example for deriving request variables from responses). For traversing larger responses I use „counsel-jq“ in a customized JSON mode: https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq But I’ll give the major mode a try, too.
- niam 3y agoIs this a proper summary of the use cases over Deno/Python scripts? * Run directly from the command line * Small interpreter ~1MB * Compact language (for better or worse) * Stable
- cryptonector 3y agoThis is a common refrain. So much so that when we changed the release artifact names in 1.7 we broke scripts and docker recipes that would download jq from the github releases 'latest' URIs! (This is fixed now.)
- qalmakka 3y agojq is amazing, and I'd like it even more if it supported CBOR too. The forks are just too finicky and are quickly abandoned, and the format is basically binary JSON.
- cryptonector 3y agoI'm working on binary support, then we can have CBOR, yeah.
- pantsforbirds 3y agoChatGPT has massively improved my ability to work with jq. I use it just infrequently enough that I constantly have to read the docs for anything non trivial, but ChatGPT lets me pump out scripts quickly. It's been super nice for my work flow.
- cjvirtucio 3y agomy favorite tool; makes writing a bash fun for me.
- pygar 3y agoIt would be nice to have this in something like coreutils so that I can be reasonably sure it's included in most systems.