7 ms·
Gron: Make JSON greppable
- g105b 3y agoLooks like an interesting tool. I use jq for any JSON related task, but it can often be finicky and complex when I just need to get at a value or search for something. Looks like gron would be a nice addition to my workflow with JSON tasks.
- stbenjam 3y agoThis is brilliant, I can't wait to have a json problem to try it out.
- rauljara 3y agoHmmm… at first glance, this feels like I’d use it for the same sorts of things I’d use jq for, only easier to use but also way less powerful. Jq does have a little bit of a learning curve necessary to get good use out of it, so I could see this being a nice quick tool for people who don’t want to make that investment. Having already learned jq, I’m not sure why I would reach for gron, but maybe I’m missing something.
- hyperthesis 3y agoNot missing, retaining something: the details of jq. Many developers find this difficult for usual and rarely used tools. See also bash.
- derwildemomo 3y agoGreat idea, definitely makes sense when you have that kind of problem. Also the username of the author made me chuckle, bonus points for that.
- omeid2 3y agoAs someone who has been using jq for years, my first instinct was why not jq? and while it answers it in the README as well, it is not very clear until you compare the output to jq. $ gron "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | fgrep "commit.author" json[0].commit.author = {}; json[0].commit.author.date = "2016-07-02T10:51:21Z"; json[0].commit.author.email = "mail@tomnomnom.com"; json[0].commit.author.name = "Tom Hudson"; And with jq: $ curl "https://api.github.com/repos/tomnomnom/gron/commits?per_page=1" | jq ".[].commit.author" { "name": "Tom Hudson", "email": "mail@tomnomnom.com", "date": "2022-04-13T14:23:37Z" } The jq version isn't greppable, as you can't do `| grep '.author.email'` for example.
- inferiorhuman 3y agoTheoretically jq can be coerced into printing similar output. Realistically though if you've already written that jq query and you wanted the email field you'd just append .email. https://unix.stackexchange.com/questions/561460/how-to-print-path-and-key-values-of-json-file#answer-760188 https://unix.stackexchange.com/questions/561460/how-to-print...
- hyperthesis 3y agoFlattening facilitates arbitrary ad hoc transformation.
- ivanhoe 3y ago> The jq version isn't greppable, as you can't do `| grep '.author.email'` for example. Truth to be told with jq you don't need to grap it, you can grab just emails directly. I find gron a lot more useful for grep -v, that is for filtering out the parts that you don't need. Super easy to clean up data.
- zimpenfish 3y ago`gron` is super useful when you don't directly know the structure of some JSON - gives you a nice simple path to locate things you can then construct a `jq` query to deal with (since e.g. dealing with multiple items in the same list can be a faff with `gron`.)
- swah 3y agoAnother option: jless. You can then copy the path and use in jq with "yq".
- __MatrixMan__ 3y agoI think of gron and jq as siblings: gron + grep for discovery, then you use gron's output (for the desired line) as jq's input, which shows you what you were after. With jq alone, you have to already understand the structure, which isn't always a given if you're combing through k8s manifests for instance.
- slt2021 3y agoThis is awesome! I am going to try to use it to make surgical edits to the terraform state file, in rare cases when I have to. Some terraform providers would rather delete and recreate resource, while a simple edit would do the trick for me
- salted-fry 3y agoI use gron a lot, because I can never remember how to use jq to do anything fancy but can usually make awk work. (I may be unusual in that department, in that I actually like awk) One warning to note is that gron burns RAM. I've killed 32GB servers working with 15MB JSON files. (I think gron -u is even worse, but my memory is a bit fuzzy here). https://github.com/adamritter/fastgron https://github.com/adamritter/fastgron as an alternative has been pretty good to me in terms of performance, I think both in speed and RAM usage.
- treyd 3y agoWhat could possibly be so memory-intensive about gron? I suppose it could make sense for ungronning, but not in the forward direction.
- nojs 3y agoDeeply nested structures would get expanded a lot
- haileys 3y agoIt buffers all of its output statements in memory before writing to stdout: https://github.com/tomnomnom/gron/blob/master/main.go#L204 https://github.com/tomnomnom/gron/blob/master/main.go#L204
- codetrotter 3y agoWhy?
- Eumenes 3y agocan see this being useful for pentesting, jq is fine but always have to look at documentation and relearn it every few weeks/months
- dang 3y agoRelated: Gron – Make JSON Greppable - https://news.ycombinator.com/item?id=25006277 https://news.ycombinator.com/item?id=25006277 - Nov 2020 (91 comments) Gron: A command line tool that makes JSON greppable - https://news.ycombinator.com/item?id=16727665 https://news.ycombinator.com/item?id=16727665 - April 2018 (51 comments)
- okbrown 3y agoSo what's the issue with jq?
- mongol 3y agoThe idea is pretty brilliant, but it is not a new one. I remember a similar program about 20 years ago when XML was all the rage. I don't recall the name but something like "py*" I think. Edit: I found it. It was called pyx. https://www.xml.com/pub/2000/03/15/feature/index.html https://www.xml.com/pub/2000/03/15/feature/index.html https://xmlstar.sourceforge.net/doc/UG/ch04s07.html https://xmlstar.sourceforge.net/doc/UG/ch04s07.html
- xonix 3y agoMy pure-AWK variant: https://github.com/xonixx/gron.awk https://github.com/xonixx/gron.awk (features complete JSON parser in AWK).
- kermatt 3y agoHas this been tested against Mawk? I often see performance benefits with Mawk, when the Awk script does not need any Gawk specific functions. I'm looking at the test suite, trying to figure out how to get it to emit test failure details: ./makesure test_suite mawk -f run_json_test_suit.awk ... Successes: 186 Fails: 152
- xonix 3y agoIt has: https://github.com/xonixx/gron.awk/blob/e5040a7b1384c5839dcad76d9b8cd78d81402813/run_json_test_suit.awk#L41 https://github.com/xonixx/gron.awk/blob/e5040a7b1384c5839dca... Strange, should be much less failures: https://github.com/xonixx/gron.awk/blob/e5040a7b1384c5839dcad76d9b8cd78d81402813/run_json_test_suit.tush.txt#L342 https://github.com/xonixx/gron.awk/blob/e5040a7b1384c5839dca... Could you please file a bug with the details?
- LVB 3y agogron is one of my favorite tools because I don't do this type of searching as much anymore and can't remember the options for more advanced tools (jq, etc). I can easily and confidently compose from gron -> grep though.
- hyperthesis 3y agoCould a jq have a more C-like syntax? JSON is already C-like https://www.json.org/json-en.html https://www.json.org/json-en.html and jq uses dot-separated paths for chaining name/value accesses.
- markbnj 3y agoThumbs up for gron. Been using it for a couple of years to get the jsonpath to a property I need. It's super handy with kubectl and other ctls of the sort.
- loadzero 3y agoI use this in my `~/.jq` when I have a problem like this. def flat_json_keys: [leaf_paths as $path | {"key": $path | map(if (type=="string") then (if (test("([?:\\W]+)")) then "['"+.+"']" else . end) else "["+tostring+"]" end) | join(".") | gsub(".\\[";"[") , "value": getpath($path)}] | from_entries; def ukeys: keys_unsorted; Use like so: cat wat.json | jq flat_json_keys
- inshadows 3y ago[dead]
- tanelpoder 3y agoI like how the description is just 3 words and clearly communicates what this tool is about.
- wonnage 3y agoI wonder if you could accomplish the same with structural search tools like Comby (https://comby.dev/ https://comby.dev/) - with the bonus that you can target specific levels of beating since it's syntax-aware
- effnorwood 3y ago[dead]
- Wicher 3y agoA plug for one of my own projects, something gron-like (but more general) for Python: https://pypi.org/project/jsonmason/ https://pypi.org/project/jsonmason/ "maybe doing something depending on context for some nodes in a nested structure by using transforms or side-effects while iterating over the regularized representation of the structure's nodes" It's a library but also includes CLI utilities that do the same as gron. Well, I hope without that memory ballooning problem described in sibling comments.
- ourmandave 3y agoWhy shouldn't I just use jq? jq is awesome, and a lot more powerful than gron, but with that power comes complexity. gron aims to make it easier to use the tools you already know, like grep and sed. I know grep, but sed is one of those I always have to look up whenever I have to escape a weird character or something.