3 ms·

I needed this tutorial 6 months ago (and 6 months before that, and 6 months before that). :D Highly recommend looking at and maybe including "gron"[1] in this

I needed this tutorial 6 months ago (and 6 months before that, and 6 months before that). :D Highly recommend looking at and maybe including "gron"[1] in this as a very nice complement to jq. It fills in some use cases in a very straightforward way that are pretty cumbersome in jq, such as finding a field deeply nested in an optional parent. [1] https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron

donut 6y ago
Agreed, gron is a fantastic tool. Earler today I was looking at some deeply nested structures which had leaf nodes with a field named "fileURL", and values that were mostly "https:// https://" but some were "http:// http://". I needed to see how many, etc. cat file.json | gron | grep fileURL | grep http | grep -v https ... and presto, I had only four such nodes. Would've been a ton more work to get there with just jq.
- Arnavion 6y ago
  Based on your description, it sounds like: <file.json jq '.. | .fileURL? | select(startswith("http://"))' -r ... would've done the job? Or, if you can't remember `startswith`: <file.json jq '.. | .fileURL?' -r | grep '^http://'
  - donut 6y ago
    Nice and terse :-) Right, the ".." part was hard to remember because it's something like: { "entries": [ { "fields": { "fileURL": { "en-US": "https://..." }, ... }, ... }, ... ] } and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this: <file.json jq -r '.entries[] | .fields | select(.fileURL) | .fileURL["en-US"] | select(startswith("http:// http://"))' And later I had to do the same for other fields that ended in URL (websiteURL, etc.) Anyway, gron made it simpler to get a quick summary of what I needed because it represents every node in the tree as a separate line that has the path in it, which is perfect for grep. I still use jq more than gron :-)
    - Arnavion 6y ago
      >Right, the ".." part was hard to remember because it's something like: You misunderstand. The command I wrote is meant to be used as I wrote it. `..` is not a placeholder for you to replace. https://stedolan.github.io/jq/manual/#RecursiveDescent%3a%2e%2e https://stedolan.github.io/jq/manual/#RecursiveDescent%3a%2e... >and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this: Sure, so then it's: <file.json jq '.. | select(.fileURL?).fileURL["en-US"] | select(startswith("http://"))' -r
      - donut 6y ago
        Oh, I had no idea ".." was an operator. That's a pretty cool feature, combined with the "?" to match optional objects. Thanks for explaining!
sequoia 6y ago
oh wow cool I didn't know about gron!! Is it alright if I add this comment to the bottom of my post?
- heleninboodler 6y ago
  Of course, although I wouldn't be surprised if a jq expert comes along and gives an easy way to deal with deeply nested children of an optional parent, making my comment moot. :D
  - sequoia 6y ago
    Ha, I never let that stop me! You can do some of the stuff in my post other ways, like using the `--jsonpath` switch in kubectl, but 1. I prefer learning jq rather than the idiosyncratic syntax/flag of each tool & 2. Let 1000 flowers bloom. I figure its OK if there's more than one way to do something!
- busfahrer 6y ago
  gron is great! And because the 80% use case of "gron -> grep -> gron -u to turn it back into JSON" is so common, I use these two convenience wrappers that make it easier. I called them grop because its' gron + grep: https://github.com/busfahrer/grop https://github.com/busfahrer/grop
tejtm 6y ago
not a jq expert but do a) get to explore random huge files sometimes as json. b) may care when the structure (but not the values) change. so a jq utility[] that spits out all the jq paths for a json files helps in both cases. [] https://github.com/TomConlin/json_to_paths https://github.com/TomConlin/json_to_paths

donut 6y ago

Agreed, gron is a fantastic tool. Earler today I was looking at some deeply nested structures which had leaf nodes with a field named "fileURL", and values that were mostly "https:// https://" but some were "http:// http://". I needed to see how many, etc. cat file.json | gron | grep fileURL | grep http | grep -v https ... and presto, I had only four such nodes. Would've been a ton more work to get there with just jq.

Arnavion 6y ago
Based on your description, it sounds like: <file.json jq '.. | .fileURL? | select(startswith("http://"))' -r ... would've done the job? Or, if you can't remember `startswith`: <file.json jq '.. | .fileURL?' -r | grep '^http://'
- donut 6y ago
  Nice and terse :-) Right, the ".." part was hard to remember because it's something like: { "entries": [ { "fields": { "fileURL": { "en-US": "https://..." }, ... }, ... }, ... ] } and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this: <file.json jq -r '.entries[] | .fields | select(.fileURL) | .fileURL["en-US"] | select(startswith("http:// http://"))' And later I had to do the same for other fields that ended in URL (websiteURL, etc.) Anyway, gron made it simpler to get a quick summary of what I needed because it represents every node in the tree as a separate line that has the path in it, which is perfect for grep. I still use jq more than gron :-)
  - Arnavion 6y ago
    >Right, the ".." part was hard to remember because it's something like: You misunderstand. The command I wrote is meant to be used as I wrote it. `..` is not a placeholder for you to replace. https://stedolan.github.io/jq/manual/#RecursiveDescent%3a%2e%2e https://stedolan.github.io/jq/manual/#RecursiveDescent%3a%2e... >and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this: Sure, so then it's: <file.json jq '.. | select(.fileURL?).fileURL["en-US"] | select(startswith("http://"))' -r
    - donut 6y ago
      Oh, I had no idea ".." was an operator. That's a pretty cool feature, combined with the "?" to match optional objects. Thanks for explaining!

sequoia 6y ago

oh wow cool I didn't know about gron!! Is it alright if I add this comment to the bottom of my post?

heleninboodler 6y ago
Of course, although I wouldn't be surprised if a jq expert comes along and gives an easy way to deal with deeply nested children of an optional parent, making my comment moot. :D
- sequoia 6y ago
  Ha, I never let that stop me! You can do some of the stuff in my post other ways, like using the `--jsonpath` switch in kubectl, but 1. I prefer learning jq rather than the idiosyncratic syntax/flag of each tool & 2. Let 1000 flowers bloom. I figure its OK if there's more than one way to do something!
busfahrer 6y ago
gron is great! And because the 80% use case of "gron -> grep -> gron -u to turn it back into JSON" is so common, I use these two convenience wrappers that make it easier. I called them grop because its' gron + grep: https://github.com/busfahrer/grop https://github.com/busfahrer/grop

tejtm 6y ago

not a jq expert but do a) get to explore random huge files sometimes as json. b) may care when the structure (but not the values) change. so a jq utility[] that spits out all the jq paths for a json files helps in both cases. [] https://github.com/TomConlin/json_to_paths https://github.com/TomConlin/json_to_paths