9 ms·

Extracting Objects Recursively with Jq

268 points by edward 5y ago

jcims 5y ago
jq seems incredibly powerful. I only find myself using it a few times a year though, and have never been able to conceptualize the syntax enough to use it without prodigious googling.
- xcambar 5y ago
  Yes, jq is very powerful yet incredibly inefficient to work with when you don't master its arcanes.
- minxomat 5y ago
  Think of it just like bash. jq is the ultimate functional language and/or ETL tool. If I look back at larger jq transforms I've written a while ago (e.g. https://git.io/JBSfB https://git.io/JBSfB) they still make perfect sense to me.
  - deleted 5y ago
    [deleted]
uluyol 5y ago
jq is a fantastic tool for exploring data and doing simple transformations. I often wish it could consume/write data in other formats.
- dangerbird2 5y ago
  In python-land, I like to use glom[1], which is basically jq but operating on arbitrary python data structures. I believe there are bindings for jq in python in other languages, which would allow operating on data structures, but I imagine they are just spawning jq as a subprocess, since it doesn't seem like jq has a public C api. [1] https://glom.readthedocs.io/en/latest/index.html https://glom.readthedocs.io/en/latest/index.html
- matthewtovbin 5y ago
  I moved to JSONata - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html
psanford 5y ago
JQ is often frustrating when you want to do something non-trivial but you can't figure out how and the documentation is of little help. I think JQ could really benefit from having a classic programming language style "book", like "The AWK Programming Language". JQ is fundamentally a functional programming language with semantics that are not obvious reading its current docs.
- doytch 5y ago
  I've also found the documentation tough to use when I need to jump back in and answer something like "how do I _____?" It feels like one of those sets of docs that's better approached by just reading the whole damn thing, working through some examples, etc. Studying the docs as opposed to referring to them.
  - psanford 5y ago
    Right, I agree. But I think the way the docs are currently written are not really meant to be read through like that. Thats why I mentioned "The AWK Programming Language" book, which is excellently written, and easy to read from start to end. I don't mean to denigrate the current docs, writing documentation is hard!
- matthewtovbin 5y ago
  Try JSONata instead - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html
  - loa_in_ 5y ago
    It doesn't seem to provide a command-line tool, which is the point of jq
    - matthewtovbin 5y ago
      Look no more - https://github.com/blgm/jfq https://github.com/blgm/jfq
- 5y ago
keymone 5y ago
jq is nice, but the moment i need anything more complex than "pull this attribute out of bunch of objects" i vastly prefer spinning up an actual language runtime. or use a tool built around a language (e.g. https://github.com/borkdude/jet https://github.com/borkdude/jet) rather than a language built around a tool.
vlmutolo 5y ago
I love posts that are this information-dense. I know for a fact that this will be useful to me some day.
deleted 5y ago
[deleted]
doytch 5y ago
I've really loved having jq at my disposal ever since learning about it, but I feel like it took the combination of it and gron [1] to really transform my debugging and JSON workflows. 1: https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron
- danso 5y ago
  I can't believe the number of times I've wanted to grep a JSON file, and yet never thought to look and see if there was any kind of tool for it. Thanks!
- terrywang 5y ago
  `gron` is really lifesaver for people like me who can't survive JSON (by simply looking / reading it) but have to deal with JSON on a daily basis (REST API, K8s related). Often times a few gron / ungron runs save the day ;-)
pcr910303 5y ago
A bit offtopic, but I don't see much people knowing/using the Algolia API[0]. It's much better to use than the HN official API[1], since it returns the whole tree data in one request. Unfortunately (I guess this is a big reason why people don't use it), it doesn't sort the comments – if you need the orders, you'll have to parse HN HTML (or just use the official API). Still just two requests (the HN site, the Algolia API) is much better than recursively requesting a hundred requests, so I use this approach in my client[2]. [0]: https://hn.algolia.com/api https://hn.algolia.com/api [1]: https://github.com/HackerNews/API https://github.com/HackerNews/API [2]: https://github.com/goranmoomin/HackerNews https://github.com/goranmoomin/HackerNews
xcambar 5y ago
I love jq for many reasons: * it is very potent * it has extensive documentation, written more like parabolas and new-age sorcery than actually helpful content * it improves your search skills greatly, for many internet results try to give sense to its format * it gives great satisfaction after you've died-and-retried 300x a whole afternoon for a command you'll only need once. * it looks cool to use jq instead of [any language you're already familiar with]. Yes, I'm being sarcastic, yet honest. Note that I still use it and recommend it for simpler use cases though. _[shrug]_
- damagednoob 5y ago
  I too find the jq syntax arcane and I've found https://jqplay.org/ https://jqplay.org/ to be an invaluable help.
- cinntaile 5y ago
  Jq makes me feel stupid, it's comforting to know that it's probably not just me.
hoshsadiq 5y ago
I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It includes functions to change the preview to output raw, compact (or not), and some other things. It is essentially similar to jqplay but local. I didn't use jid/jiq because jid uses go-simplejson, which is nowhere near as powerful as jq, and jiq seemed very buggy when I used it and it felt like it was hacked together. Plus there was no where to change jq's arguments while running it. I'm sure this function can be improved on, but this has been good enough for me so far. Also, I run gojq[2] instead of jq. It is a drop-in replacement for jq but is written in Go, and has some improvements over jq such as bug fixes, support for yaml input, and it also provides more helpful error messages. [0] https://github.com/hoshsadiq/dot_files/blob/master/zshrc.d/misc/fuzzy.zsh#L27-L159 https://github.com/hoshsadiq/dot_files/blob/master/zshrc.d/m... [1] https://github.com/junegunn/fzf https://github.com/junegunn/fzf [2] https://github.com/itchyny/gojq/ https://github.com/itchyny/gojq/
- rattray 5y ago
  This looks cool - would you be willing to share it in a brew-installable format?
  - hoshsadiq 5y ago
    Not particularly. I don't use a Mac, but I'd be happy to separate out the function into it's own script so it can be downloaded and put in your $PATH. I personally use zinit to manage individual files from random repos.
- fisxoj 5y ago
  If anyone is an emacs user and this sounds compelling, I recommend counsel-jq[0] for the sort of feedback loop described here. [0]: https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq
matthewtovbin 5y ago
Forget about jq. JSONata is much more powerful - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html
- sidpatil 5y ago
  jq is a single binary which I can easily just drop into a remote server or throw into my ~/.local/bin, where as JSONata requires NPM.
- turbocon 5y ago
  Yea what's going on here, you have a couple of other comments pushing Jsonata? You an author or something?
  - matthewtovbin 5y ago
    Not at all. After using jq in the past and seeing both it's potential and flaws I was looking for something more powerful and easy to use, so I stumbled upon JSONata and became a fan.
- deleted 5y ago
  [deleted]
- maleldil 5y ago
  It doesn't appear to have a CLI, which is the whole point of jq
  - matthewtovbin 5y ago
    There is one - https://github.com/blgm/jfq https://github.com/blgm/jfq
yewenjie 5y ago
Can someone comment on how does `jq` compare with `fx`? - https://github.com/antonmedv/fx https://github.com/antonmedv/fx
baldeagle 5y ago
The most powerful thing about jq, IMO is that is can alter some of the json without having to parse and convert to an object all of the json. It is like using data transformation lasers.
jeffbee 5y ago
I often use jq for random hacks but for something like this I would turn to XPath. I know that sounds impossibly retro, but XPath 3.1 for JSON is awesome and its language makes so much more sense than jq’s. There are several good implementations, and all of them are faster than jq, too.
tejtm 5y ago
To better appreciate the structure of the document the author is dealing with (and to cast a bit of light on which words are variables in the document and which are `jq` syntax. I offer a shameless plug to a one liner well, I would but the result is "too long for a HN comment" so here a bunch is sniped out of the middle (unedited the result would currently be 140 lines) curl -s https://hn.algolia.com/api/v1/items/27941108 | ~/bin/json2jqpath.jq . .author .children .children|.[] .children|.[]|.author .children|.[]|.children .children|.[]|.children|.[] .children|.[]|.children|.[]|.author .children|.[]|.children|.[]|.children .children|.[]|.children|.[]|.children|.[] .children|.[]|.children|.[]|.children|.[]|.author .children|.[]|.children|.[]|.children|.[]|.children .children|.[]|.children|.[]|.children|.[]|.children|.[] <snip> ... </snip> .children|.[]|.children|.[]|.created_at .children|.[]|.children|.[]|.created_at_i .children|.[]|.children|.[]|.id .children|.[]|.children|.[]|.options .children|.[]|.children|.[]|.parent_id .children|.[]|.children|.[]|.points .children|.[]|.children|.[]|.story_id .children|.[]|.children|.[]|.text .children|.[]|.children|.[]|.title .children|.[]|.children|.[]|.type .children|.[]|.children|.[]|.url .children|.[]|.created_at .children|.[]|.created_at_i .children|.[]|.id .children|.[]|.options .children|.[]|.parent_id .children|.[]|.points .children|.[]|.story_id .children|.[]|.text .children|.[]|.title .children|.[]|.type .children|.[]|.url .created_at .created_at_i .id .options .parent_id .points .story_id .text .title .type .url [0] https://github.com/TomConlin/json_to_paths https://github.com/TomConlin/json_to_paths
- woodruffw 5y ago
 Thanks for linking this! I've wanted exactly this script, many times.
- naniwaduni 5y ago
 Looks a bit like `jq -rc '[path(..)|map(strings//"[]")]|unique[]|"."+join("|.")'`, but might I suggest `jq -rc '[paths|map(("."+strings)//"[]")|join("")]|unique[]'`?
 - tejtm 5y ago
 There is no need to guess, you can just look in the repo. I like the explicit separators when I'm reading them instead of writing them. Helps the individual steps stand out for me. Is there a reason other than a more compact line?
thangalin 5y ago
An introductory script showing the jq, REST, and JSON trinity: https://github.com/DaveJarvis/github-email/blob/master/github-email.sh https://github.com/DaveJarvis/github-email/blob/master/githu...
deleted 5y ago
[deleted]
triska 5y ago
JSON data is also a valid Prolog term, and the declarative programming language Prolog is ideally suited for handling tree-shaped data. Using for example Scryer Prolog, we can conveniently relate the data to a flat list of items with Prolog's built-in grammar mechanism, definite clause grammars (DCGs): flat_json(JSON) --> { JSON = {A,B,C,D,E,F,_:Cs} }, [{A,B,C,D,E,F}], flat_items(Cs). flat_items([]) --> []. flat_items([I|Is]) --> { I = {(A,B,C,D,E,_:Cs)} }, [{A,B,C,D,E}], flat_items(Cs), flat_items(Is). Sample query, using the example JSON data from the article: ?- JSON = { "id": 27941108, "created_at": "2021-07-24T14:15:05.000Z", "type": "story", "author": "edward", "title": "Fun with Unix domain sockets", "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/", "children": [ { "id": 27942287, "created_at": "2021-07-24T16:31:18.000Z", "type": "comment", "author": "DesiLurker", "text": "one lesser known...", "children": [] }, { "id": 27944615, "created_at": "2021-07-24T21:26:33.000Z", "type": "comment", "author": "galaxyLogic", "text": "I read this from Wikipedia...", "children": [ { "id": 27944746, "created_at": "2021-07-24T21:49:07.000Z", "type": "comment", "author": "hughrr", "text": "Yes although I ...", "children": [] } ] } ] }, phrase(flat_json(JSON), Cs), maplist(portray_clause, Cs). yielding the flat list of entries, as desired: [{("id":27941108,"created_at":"2021-07-24T14:15:05.000Z","type":"story","author":"edward",...)}, {("id":27942287,"created_at":"2021-07-24T16:31:18.000Z","type":"comment",...)}, {("id":27944615,"created_at":"2021-07-24T21:26:33.000Z","type":"comment",...)}, {("id":27944746,"created_at":"2021-07-24T21:49:07.000Z","type":"comment",...)}]
- ec109685 5y ago
 Anyway to avoid the need to enumerate the json fields that come before children with those placeholders? Otherwise, it will be brittle to modifications. Prolog was the one language I couldn’t get my head around in the programming languages class I took at school.
mftb 5y ago
I'm happy to see that I'm not alone in my struggles with jq. I wanted to love it right out of the box. It appears to be very well engineered, but over and over again I have struggled with it's syntax. What I think I want is a syntax closer to css selectors. What I think I'm going to have to do is really stop and learn jq. It looks like some of the links in here may help.
ur-whale 5y ago
There's a real need for a tool like jq, but jq unfortunately isn't it. What I mean is this: the functionality offered by jq (parsing json on the command line and extracting what you need from it) is really needed in many modern data processing tasks, but jq's DSL is one of the most horrible thing I've had to learn in recent years. The only way a casual user of that thing can hope to succeed in actually crafting a working jq query is pray that there is a stack overflow topic answering his exact need. Here's the way I use jq 99% of the time: cat somefile.json | jq . | <long pipeline of traditional, well though-out unix text processing tools such as sed, grep, awk, cut, etc...> And when this doesn't cut it, I write a python script.
- seppel 5y ago
 You might prefer gron for this use case: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
quickthrower2 5y ago
Jq is a Command-line JSON processor, if like me you didn’t know and thought it might be Jquery abbreviated
smitty1e 5y ago
Jq cries out for a good O'Reilly cookbook.
arecurrence 5y ago
Jq has a playground that’s great for learning how to use it https://jqplay.org/ https://jqplay.org/ I often build my Jq steps in the playground before committing.
ittan 5y ago
Could the title be changed to specify `Extracting JSON objects recursively with jq` please?
TeeMassive 5y ago
In a job I did years ago, we had to deal with errors caused by small changes made in huge object trees in C# in a domain that requires handling thousands of parameters that all influence each other. Fortunately there was a Visual Studio extension that could export objects into JSON/XML/C# representation; then all I had to do was to diff the object dump. Was often called a "Jedi Wizard" for using this trick lol
sillysaurusx 5y ago
I wish jq could be used in Python. It would be so nice. Imagine writing a simple jq string and running that, rather than writing complicated loops for examining json data. It does exist as a library for Python, but unfortunately it links to jq’s C library. That means if I use it, then whoever uses my program has to compile C code whenever they pip install my program. I’m unwilling to do that, because history shows that as t approaches infinity, the chance of C code failing to compile during a pip install approaches 100%. If anyone’s looking for a challenge, please implement all of jq’s logic in pure Python. It would be so wonderful…
0xbadcafebee 5y ago
As a systems engineer working with containers, while I find jq really annoying to use, I much prefer it as a dependency to Python or Perl. Not only is it smaller, it's easier to install in weird environments. Say you need to jump into a running container that isn't running as root, doesn't have sudo/su, and doesn't even have packaging tools. The only practical way to quickly install a debugging tool is either 1) volume-mount a file, 2) copy in a file, or 3) curl/wget. So a small(ish) static binary is the best thing you can hope for. I've actually built a dozen static versions of common debugging tools just to make this easier. If I had more free time I would port jq's semantics to BusyBox and gain that much more utility out of that lovely little environment.
otikik 5y ago
Jq's query language feels like regular expressions for JSON. With the same amount of benefits and drawbacks.
xg15 5y ago
Instead of using recurse(), you can also define a function in jq and call that one recursively. E.g., the example from the OP could also be written as: def my_func: del(.children), (.children[] | my_func) ; [my_func] (The function first emits its input object without the "children" field, then emits each element of the "children" array separately, after passing each through a recursive call of the function. The ; char ends the function definition. Finally, the function is called once on the top-level input and all emitted results are collected into an array.) This is more verbose than recurse() but may sometimes be easier to understand if you want to know what is going on.

268 points by edward 5y ago

jq seems incredibly powerful. I only find myself using it a few times a year though, and have never been able to conceptualize the syntax enough to use it without prodigious googling.

xcambar 5y ago
Yes, jq is very powerful yet incredibly inefficient to work with when you don't master its arcanes.
minxomat 5y ago
Think of it just like bash. jq is the ultimate functional language and/or ETL tool. If I look back at larger jq transforms I've written a while ago (e.g. https://git.io/JBSfB https://git.io/JBSfB) they still make perfect sense to me.
- deleted 5y ago
  [deleted]

uluyol 5y ago

jq is a fantastic tool for exploring data and doing simple transformations. I often wish it could consume/write data in other formats.

dangerbird2 5y ago
In python-land, I like to use glom[1], which is basically jq but operating on arbitrary python data structures. I believe there are bindings for jq in python in other languages, which would allow operating on data structures, but I imagine they are just spawning jq as a subprocess, since it doesn't seem like jq has a public C api. [1] https://glom.readthedocs.io/en/latest/index.html https://glom.readthedocs.io/en/latest/index.html
matthewtovbin 5y ago
I moved to JSONata - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html

psanford 5y ago

JQ is often frustrating when you want to do something non-trivial but you can't figure out how and the documentation is of little help. I think JQ could really benefit from having a classic programming language style "book", like "The AWK Programming Language". JQ is fundamentally a functional programming language with semantics that are not obvious reading its current docs.

doytch 5y ago
I've also found the documentation tough to use when I need to jump back in and answer something like "how do I _____?" It feels like one of those sets of docs that's better approached by just reading the whole damn thing, working through some examples, etc. Studying the docs as opposed to referring to them.
- psanford 5y ago
  Right, I agree. But I think the way the docs are currently written are not really meant to be read through like that. Thats why I mentioned "The AWK Programming Language" book, which is excellently written, and easy to read from start to end. I don't mean to denigrate the current docs, writing documentation is hard!
matthewtovbin 5y ago
Try JSONata instead - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html
- loa_in_ 5y ago
  It doesn't seem to provide a command-line tool, which is the point of jq
  - matthewtovbin 5y ago
    Look no more - https://github.com/blgm/jfq https://github.com/blgm/jfq
5y ago

keymone 5y ago

jq is nice, but the moment i need anything more complex than "pull this attribute out of bunch of objects" i vastly prefer spinning up an actual language runtime. or use a tool built around a language (e.g. https://github.com/borkdude/jet https://github.com/borkdude/jet) rather than a language built around a tool.

vlmutolo 5y ago

I love posts that are this information-dense. I know for a fact that this will be useful to me some day.

deleted 5y ago

[deleted]

doytch 5y ago

I've really loved having jq at my disposal ever since learning about it, but I feel like it took the combination of it and gron [1] to really transform my debugging and JSON workflows. 1: https://github.com/TomNomNom/gron https://github.com/TomNomNom/gron

danso 5y ago
I can't believe the number of times I've wanted to grep a JSON file, and yet never thought to look and see if there was any kind of tool for it. Thanks!
terrywang 5y ago
`gron` is really lifesaver for people like me who can't survive JSON (by simply looking / reading it) but have to deal with JSON on a daily basis (REST API, K8s related). Often times a few gron / ungron runs save the day ;-)

pcr910303 5y ago

A bit offtopic, but I don't see much people knowing/using the Algolia API[0]. It's much better to use than the HN official API[1], since it returns the whole tree data in one request. Unfortunately (I guess this is a big reason why people don't use it), it doesn't sort the comments – if you need the orders, you'll have to parse HN HTML (or just use the official API). Still just two requests (the HN site, the Algolia API) is much better than recursively requesting a hundred requests, so I use this approach in my client[2]. [0]: https://hn.algolia.com/api https://hn.algolia.com/api [1]: https://github.com/HackerNews/API https://github.com/HackerNews/API [2]: https://github.com/goranmoomin/HackerNews https://github.com/goranmoomin/HackerNews

xcambar 5y ago

I love jq for many reasons: * it is very potent * it has extensive documentation, written more like parabolas and new-age sorcery than actually helpful content * it improves your search skills greatly, for many internet results try to give sense to its format * it gives great satisfaction after you've died-and-retried 300x a whole afternoon for a command you'll only need once. * it looks cool to use jq instead of [any language you're already familiar with]. Yes, I'm being sarcastic, yet honest. Note that I still use it and recommend it for simpler use cases though. _[shrug]_

damagednoob 5y ago
I too find the jq syntax arcane and I've found https://jqplay.org/ https://jqplay.org/ to be an invaluable help.
cinntaile 5y ago
Jq makes me feel stupid, it's comforting to know that it's probably not just me.

hoshsadiq 5y ago

I tend to use jq a lot. As others have said, sometimes jq can be hard to grasp. Often it requires multiple attempts to get the correct answer. To make it a little easier for me, I've written a helper function[0] that combines it with fzf[1] to run jq as a REPL on any json. It allows to incrementally alter your DSL without having to continually call jq. This is similar to jid/jiq but a little more powerful. It includes functions to change the preview to output raw, compact (or not), and some other things. It is essentially similar to jqplay but local. I didn't use jid/jiq because jid uses go-simplejson, which is nowhere near as powerful as jq, and jiq seemed very buggy when I used it and it felt like it was hacked together. Plus there was no where to change jq's arguments while running it. I'm sure this function can be improved on, but this has been good enough for me so far. Also, I run gojq[2] instead of jq. It is a drop-in replacement for jq but is written in Go, and has some improvements over jq such as bug fixes, support for yaml input, and it also provides more helpful error messages. [0] https://github.com/hoshsadiq/dot_files/blob/master/zshrc.d/misc/fuzzy.zsh#L27-L159 https://github.com/hoshsadiq/dot_files/blob/master/zshrc.d/m... [1] https://github.com/junegunn/fzf https://github.com/junegunn/fzf [2] https://github.com/itchyny/gojq/ https://github.com/itchyny/gojq/

rattray 5y ago
This looks cool - would you be willing to share it in a brew-installable format?
- hoshsadiq 5y ago
  Not particularly. I don't use a Mac, but I'd be happy to separate out the function into it's own script so it can be downloaded and put in your $PATH. I personally use zinit to manage individual files from random repos.
fisxoj 5y ago
If anyone is an emacs user and this sounds compelling, I recommend counsel-jq[0] for the sort of feedback loop described here. [0]: https://github.com/200ok-ch/counsel-jq https://github.com/200ok-ch/counsel-jq

matthewtovbin 5y ago

Forget about jq. JSONata is much more powerful - http://docs.jsonata.org/overview.html http://docs.jsonata.org/overview.html

sidpatil 5y ago
jq is a single binary which I can easily just drop into a remote server or throw into my ~/.local/bin, where as JSONata requires NPM.
turbocon 5y ago
Yea what's going on here, you have a couple of other comments pushing Jsonata? You an author or something?
- matthewtovbin 5y ago
  Not at all. After using jq in the past and seeing both it's potential and flaws I was looking for something more powerful and easy to use, so I stumbled upon JSONata and became a fan.
deleted 5y ago
[deleted]
maleldil 5y ago
It doesn't appear to have a CLI, which is the whole point of jq
- matthewtovbin 5y ago
  There is one - https://github.com/blgm/jfq https://github.com/blgm/jfq

yewenjie 5y ago

Can someone comment on how does `jq` compare with `fx`? - https://github.com/antonmedv/fx https://github.com/antonmedv/fx

baldeagle 5y ago

The most powerful thing about jq, IMO is that is can alter some of the json without having to parse and convert to an object all of the json. It is like using data transformation lasers.

jeffbee 5y ago

I often use jq for random hacks but for something like this I would turn to XPath. I know that sounds impossibly retro, but XPath 3.1 for JSON is awesome and its language makes so much more sense than jq’s. There are several good implementations, and all of them are faster than jq, too.

tejtm 5y ago

To better appreciate the structure of the document the author is dealing with (and to cast a bit of light on which words are variables in the document and which are `jq` syntax. I offer a shameless plug to a one liner well, I would but the result is "too long for a HN comment" so here a bunch is sniped out of the middle (unedited the result would currently be 140 lines) curl -s https://hn.algolia.com/api/v1/items/27941108 | ~/bin/json2jqpath.jq . .author .children .children|.[] .children|.[]|.author .children|.[]|.children .children|.[]|.children|.[] .children|.[]|.children|.[]|.author .children|.[]|.children|.[]|.children .children|.[]|.children|.[]|.children|.[] .children|.[]|.children|.[]|.children|.[]|.author .children|.[]|.children|.[]|.children|.[]|.children .children|.[]|.children|.[]|.children|.[]|.children|.[] <snip> ... </snip> .children|.[]|.children|.[]|.created_at .children|.[]|.children|.[]|.created_at_i .children|.[]|.children|.[]|.id .children|.[]|.children|.[]|.options .children|.[]|.children|.[]|.parent_id .children|.[]|.children|.[]|.points .children|.[]|.children|.[]|.story_id .children|.[]|.children|.[]|.text .children|.[]|.children|.[]|.title .children|.[]|.children|.[]|.type .children|.[]|.children|.[]|.url .children|.[]|.created_at .children|.[]|.created_at_i .children|.[]|.id .children|.[]|.options .children|.[]|.parent_id .children|.[]|.points .children|.[]|.story_id .children|.[]|.text .children|.[]|.title .children|.[]|.type .children|.[]|.url .created_at .created_at_i .id .options .parent_id .points .story_id .text .title .type .url [0] https://github.com/TomConlin/json_to_paths https://github.com/TomConlin/json_to_paths

woodruffw 5y ago
Thanks for linking this! I've wanted exactly this script, many times.
naniwaduni 5y ago
Looks a bit like `jq -rc '[path(..)|map(strings//"[]")]|unique[]|"."+join("|.")'`, but might I suggest `jq -rc '[paths|map(("."+strings)//"[]")|join("")]|unique[]'`?
- tejtm 5y ago
  There is no need to guess, you can just look in the repo. I like the explicit separators when I'm reading them instead of writing them. Helps the individual steps stand out for me. Is there a reason other than a more compact line?

thangalin 5y ago

An introductory script showing the jq, REST, and JSON trinity: https://github.com/DaveJarvis/github-email/blob/master/github-email.sh https://github.com/DaveJarvis/github-email/blob/master/githu...

deleted 5y ago

[deleted]

triska 5y ago

JSON data is also a valid Prolog term, and the declarative programming language Prolog is ideally suited for handling tree-shaped data. Using for example Scryer Prolog, we can conveniently relate the data to a flat list of items with Prolog's built-in grammar mechanism, definite clause grammars (DCGs): flat_json(JSON) --> { JSON = {A,B,C,D,E,F,_:Cs} }, [{A,B,C,D,E,F}], flat_items(Cs). flat_items([]) --> []. flat_items([I|Is]) --> { I = {(A,B,C,D,E,_:Cs)} }, [{A,B,C,D,E}], flat_items(Cs), flat_items(Is). Sample query, using the example JSON data from the article: ?- JSON = { "id": 27941108, "created_at": "2021-07-24T14:15:05.000Z", "type": "story", "author": "edward", "title": "Fun with Unix domain sockets", "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/", "children": [ { "id": 27942287, "created_at": "2021-07-24T16:31:18.000Z", "type": "comment", "author": "DesiLurker", "text": "one lesser known...", "children": [] }, { "id": 27944615, "created_at": "2021-07-24T21:26:33.000Z", "type": "comment", "author": "galaxyLogic", "text": "I read this from Wikipedia...", "children": [ { "id": 27944746, "created_at": "2021-07-24T21:49:07.000Z", "type": "comment", "author": "hughrr", "text": "Yes although I ...", "children": [] } ] } ] }, phrase(flat_json(JSON), Cs), maplist(portray_clause, Cs). yielding the flat list of entries, as desired: [{("id":27941108,"created_at":"2021-07-24T14:15:05.000Z","type":"story","author":"edward",...)}, {("id":27942287,"created_at":"2021-07-24T16:31:18.000Z","type":"comment",...)}, {("id":27944615,"created_at":"2021-07-24T21:26:33.000Z","type":"comment",...)}, {("id":27944746,"created_at":"2021-07-24T21:49:07.000Z","type":"comment",...)}]

ec109685 5y ago
Anyway to avoid the need to enumerate the json fields that come before children with those placeholders? Otherwise, it will be brittle to modifications. Prolog was the one language I couldn’t get my head around in the programming languages class I took at school.

mftb 5y ago

I'm happy to see that I'm not alone in my struggles with jq. I wanted to love it right out of the box. It appears to be very well engineered, but over and over again I have struggled with it's syntax. What I think I want is a syntax closer to css selectors. What I think I'm going to have to do is really stop and learn jq. It looks like some of the links in here may help.

ur-whale 5y ago

There's a real need for a tool like jq, but jq unfortunately isn't it. What I mean is this: the functionality offered by jq (parsing json on the command line and extracting what you need from it) is really needed in many modern data processing tasks, but jq's DSL is one of the most horrible thing I've had to learn in recent years. The only way a casual user of that thing can hope to succeed in actually crafting a working jq query is pray that there is a stack overflow topic answering his exact need. Here's the way I use jq 99% of the time: cat somefile.json | jq . | <long pipeline of traditional, well though-out unix text processing tools such as sed, grep, awk, cut, etc...> And when this doesn't cut it, I write a python script.

seppel 5y ago
You might prefer gron for this use case: https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron

quickthrower2 5y ago

Jq is a Command-line JSON processor, if like me you didn’t know and thought it might be Jquery abbreviated

smitty1e 5y ago

Jq cries out for a good O'Reilly cookbook.

arecurrence 5y ago

Jq has a playground that’s great for learning how to use it https://jqplay.org/ https://jqplay.org/ I often build my Jq steps in the playground before committing.

ittan 5y ago

Could the title be changed to specify `Extracting JSON objects recursively with jq` please?

TeeMassive 5y ago

In a job I did years ago, we had to deal with errors caused by small changes made in huge object trees in C# in a domain that requires handling thousands of parameters that all influence each other. Fortunately there was a Visual Studio extension that could export objects into JSON/XML/C# representation; then all I had to do was to diff the object dump. Was often called a "Jedi Wizard" for using this trick lol

sillysaurusx 5y ago

I wish jq could be used in Python. It would be so nice. Imagine writing a simple jq string and running that, rather than writing complicated loops for examining json data. It does exist as a library for Python, but unfortunately it links to jq’s C library. That means if I use it, then whoever uses my program has to compile C code whenever they pip install my program. I’m unwilling to do that, because history shows that as t approaches infinity, the chance of C code failing to compile during a pip install approaches 100%. If anyone’s looking for a challenge, please implement all of jq’s logic in pure Python. It would be so wonderful…

0xbadcafebee 5y ago

As a systems engineer working with containers, while I find jq really annoying to use, I much prefer it as a dependency to Python or Perl. Not only is it smaller, it's easier to install in weird environments. Say you need to jump into a running container that isn't running as root, doesn't have sudo/su, and doesn't even have packaging tools. The only practical way to quickly install a debugging tool is either 1) volume-mount a file, 2) copy in a file, or 3) curl/wget. So a small(ish) static binary is the best thing you can hope for. I've actually built a dozen static versions of common debugging tools just to make this easier. If I had more free time I would port jq's semantics to BusyBox and gain that much more utility out of that lovely little environment.

otikik 5y ago

Jq's query language feels like regular expressions for JSON. With the same amount of benefits and drawbacks.

xg15 5y ago

Instead of using recurse(), you can also define a function in jq and call that one recursively. E.g., the example from the OP could also be written as: def my_func: del(.children), (.children[] | my_func) ; [my_func] (The function first emits its input object without the "children" field, then emits each element of the "children" array separately, after passing each through a recursive call of the function. The ; char ends the function definition. Finally, the function is called once on the top-level input and all emitted results are collected into an array.) This is more verbose than recurse() but may sometimes be easier to understand if you want to know what is going on.