9 ms·

Gron: A command line tool that makes JSON greppable

247 points by oftenwrong 9y ago

derimagia 9y ago
If you'd like to see some more tools for dealing with structured text take a look, https://github.com/dbohdan/structured-text-tools https://github.com/dbohdan/structured-text-tools a pretty nice list.
adrianN 9y ago
I find jq very useful for tasks like this. But it's more awk for JSON than grep for JSON.
- jedisct1 9y ago
  And rq has even easier syntax.
  - cryptonector 9y ago
    Link?
    - bklaasen 9y ago
      https://github.com/dflemstr/rq/blob/master/README.md https://github.com/dflemstr/rq/blob/master/README.md
      - cryptonector 9y ago
        Thanks!
        erric 9y ago
        Never had heard of rq, so thanks to the person above for that. I also use jp from JMESpath. I like it because both azure and aws cli use the JMESpath way of dealing with JSON so filtering between two different cloud providers is at least a _little_ easier when hunting needles in haystacks. For some compare and contrast between: jsonpath, jq, and jp these were for getting AWS ec2 instance IDs: cat foo.json | jsonpath -p $.InstanceProfiles.[*].RoleId cat foo.json | jq .InstanceProfiles[].Roles[].RoleId cat foo.json | jp InstanceProfiles[].InstanceProfileId apologies to mobile users!
  - stormbrew 9y ago
    I.. really don't agree. I had to use rq because I was working with a toml file and I could not for the life of me figure out even the basic things I was trying to do. The documentation was sparse and unclear compared to jq's relatively concise but clear documentation. In the end I wound up just using rq to convert the toml to JSON so I could use jq on it. It was a while ago so I suppose it's possible I just used it too early in its life or something though.
krat0sprakhar 9y ago
If you use jq[0] and are wondering why Gron, the answer is at the very bottom of the readme: jq is awesome, and a lot more powerful than gron, but with that power comes complexity. gron aims to make it easier to use the tools you already know, like grep and sed. gron's primary purpose is to make it easy to find the path to a value in a deeply nested JSON blob when you don't already know the structure; much of jq's power is unlocked only once you know that structure. [0] - https://stedolan.github.io/jq/ https://stedolan.github.io/jq/
- jypepin 9y ago
  and this sounds awesome, I'll definitely test Gron. I love jq, but oh boy how complex it is. I find its syntax getting very confusing very quickly as soons as you try to be a little fancy.
  - cryptonector 9y ago
    It's a full-blown, dynamically-typed functional programming language. Well, not quite full-blown: it's missing closures of indefinite extent, but still.
- cryptonector 9y ago
  $ jq -c tostream <<<'{"a":[{"b":2}]}' [["a",0,"b"],2] [["a",0,"b"]] [["a",0]] [["a"]] $ However, filtering that and then reconstructing JSON from that is... not possible at this time: $ jq -c tostream <<<'{"a":[{"b":2}]}'|jq -crn 'fromstream(inputs)' {"a":[{"b":2}]} $ $ jq -c tostream <<<'{"a":[{"b":2}]}'|grep b|jq -crn 'fromstream(inputs)' $ :( The reason is that tostream and fromstream can handle multiple top-level JSON texts, since jq normally does too, but then there's an ambiguity issue to resolve by having a sort of an object terminator. Filtering tostream's output with grep loses the terminators, and so fromstream cannot operate normally. But it should be possible to define a function that does allow this, by, e.g., requiring just one top-level JSON text. The other thing is that a path-based encoding that does not require quotes and commas would be handier -- tostream's output is itself JSON, so it's not shell-friendly. This is gron's brilliant innovation: it's got a path-based encoding of JSON that is easy to deal with in a shell script. (Mind you, I'm not sure that using brackets to denote array indices is all that easy to use, but the need to disambiguate object keys that look like numbers is critical. Also, there's an ambiguity as to keys that have embedded periods ('.') in them. And lastly, even gron can't shake off the string quotes for values.) That jq has the builtin functionality needed to do the same is not good enough if it doesn't actually do it out of the box.
xenomachina 9y ago
A few years ago I made a Python script that does the same thing, minus the reverse mode of converting the assignments back into JSON: https://github.com/xenomachina/jsflat https://github.com/xenomachina/jsflat I've found flattening JSON in this way not only useful for line based tools like grep, but also for understanding unfamiliar JSON. Sometimes it's nice to be able to see the whole path down to the value you're looking at.
- deleted 9y ago
  [deleted]
  - deleted 9y ago
    [deleted]
kbenson 9y ago
This is cool... but it appears to also work as an HTTP client as well? I'm not sure why the additional complexity of including this is needed. I maintain that any HTTP client functionality that supports enough options to be useful is complex and if it supports so few options that it's a toy, why include it? There is non-negligible overhead in keeping track of what shell tools can make network requests, and not being correct and up to date in that area has been the cause of numerous bugs and security issues in tool sets and programs that include and utilize a component like this without realizing it. To many, this might sound like some overly nit-picky complaint, but I maintain if you ask just about anyone that's been in the trenches as a sysadmin for more than a couple years whether they think saving the few characters it takes to use curl and pipe is a good trade-off for the possible unintended consequences of some developer shelling out to this without proper validation from some webapp, they'll tell you no. This tool seems awesome, but keep it simple. It's not like it's a GUI tool and it's hard to pass data between programs. A pipe is perfect here.
- vbsteven 9y ago
  I came here to say the same thing. It looks like a great tool which I will use frequently on large json files but I don't get why the author bothered to implement a basic HTTP client with some curl-like options when it can just as well read from stdin.
  - erric 9y ago
    Maybe in a container build environment like CircleCI where having a small concise toolset is desirable?
- cup-of-tea 9y ago
  The author just hasn't fully grokked the command line yet.
  - kbenson 9y ago
    That seems overly harsh. It's a cool utility, and the author even shows it being used with pipes in numerous ways. I'm just arguing that including non-essential features is a trade-off. Often it's just complexity for ease of use, but I think this one has some possible security implications, and what you get for that may not be worthwhile.
beaugunderson 9y ago
There's some prior art; namely jsonpipe/jsonunpipe: https://github.com/zacharyvoase/jsonpipe https://github.com/zacharyvoase/jsonpipe
kriomant 9y ago
Alternative: use ogrep (https://github.com/kriomant/ogrep-rs https://github.com/kriomant/ogrep-rs / https://github.com/kriomant/ogrep https://github.com/kriomant/ogrep) on pretty-printed JSON.
soheilpro 9y ago
I have written a similar tool called catj [1]. I mostly use it when I need to construct JSON expressions when working with jq. [1] https://github.com/soheilpro/catj https://github.com/soheilpro/catj
yason 9y ago
This kind of tools borderline the very interesting threshold where writing your own tool can be less of a mental effort than discovering, learning and keeping track of these small utilities separately. I've written a similar tool in Python, both for JSON and XML. Especially the JSON version was dead simple, probably fits on a single screen and took 15 minutes to test and write. Surely it didn't have any "features" but it does the job of letting me grep json. Gron is probably 10x more versatile and actually comes with useful features but I'd really have to have pressing needs to do transformations of JSON on a regular basis to switch over. The same applies to libraries in programming languages. There is a very vague threshold, depending on the expressiveness of the language and operating environment as well as the hardness of the problem itself, where it either makes sense to write your own library or reuse an existing one.
bm1362 9y ago
If you’re stuck on a foreign box or don’t want to install a new tool: cat xyz | python -mjson.tool | grep foo
- jwilk 9y ago
  Python's json.tool is a pretty-printer. It's not anything like gron. (json_pp is another pretty-printer that is likely to be already on your system.)
- pm90 9y ago
  This assumes that you have python installed ;)
billsmithaustin 9y ago
I might have chosen a name that sounds less like Cron, but regardless, I can see how Gron would be useful.
- deleted 9y ago
  [deleted]
deleted 9y ago
[deleted]
deleted 9y ago
[deleted]
acobster 9y ago
Thanks for this! "Grep for absolute path" is the use-case that the otherwise awesome `jq` doesn't address well. I've found myself having to iteratively drill down to find the field that has the data I need. One of those "eh, I'll automate this [poorly] someday..." things. :D
tedivm 9y ago
I wrote a tool called jsonsmash[0][1] that's meant for a more explorative view of the JSON files. It basically exposes the data in a minishell, complete with `ls` (with a ton of the standard flags), `cd`, `pwd`, `cat` (which outputs in yaml), and some others. My main use case was to read some json files that were far too big to load in standard editors (210mb+) and for that it has worked great. [0] https://www.npmjs.com/package/jsonsmash https://www.npmjs.com/package/jsonsmash [1] https://blog.tedivm.com/open-source/2017/05/introducing-jsonsmash-work-with-large-json-files-easily/ https://blog.tedivm.com/open-source/2017/05/introducing-json...
- pronik 9y ago
  This sounds awfully similar to Augeas CLI. Not that I endorse Augeas in any capacity...
coldacid 9y ago
Looks like I have a new Chocolatey package to create and push up when I get home tonight.
aiCeivi9 9y ago
json.Host = "headers.jsontest.com"; json["User-Agent"] = "curl/7.43.0"; Why not use simple notation for eveything without '.' in key?
- TomNomNom 9y ago
  Author of the tool here. The output is designed to be valid JavaScript, which doesn't allow certain characters in unquoted object keys, like the dash in User-Agent. Using JavaScript's rules for quoting keys makes it a lot easier to specify the grammar (and therefore write the parser); and makes it trivial to 'parse' the output using JavaScript should you want to. There must be _some_ rules in place for when to quote the key (e.g. when there is a dot, equals sign, square brace etc in the key name), so I see no reason to adopt something custom and potentially error-prone when a known-good set of rules already exists. Hope that answers your question well enough!

247 points by oftenwrong 9y ago

If you'd like to see some more tools for dealing with structured text take a look, https://github.com/dbohdan/structured-text-tools https://github.com/dbohdan/structured-text-tools a pretty nice list.

adrianN 9y ago

I find jq very useful for tasks like this. But it's more awk for JSON than grep for JSON.

jedisct1 9y ago
And rq has even easier syntax.
- cryptonector 9y ago
  Link?
  - bklaasen 9y ago
    https://github.com/dflemstr/rq/blob/master/README.md https://github.com/dflemstr/rq/blob/master/README.md
    - cryptonector 9y ago
      Thanks!
      - erric 9y ago
        Never had heard of rq, so thanks to the person above for that. I also use jp from JMESpath. I like it because both azure and aws cli use the JMESpath way of dealing with JSON so filtering between two different cloud providers is at least a _little_ easier when hunting needles in haystacks. For some compare and contrast between: jsonpath, jq, and jp these were for getting AWS ec2 instance IDs: cat foo.json | jsonpath -p $.InstanceProfiles.[*].RoleId cat foo.json | jq .InstanceProfiles[].Roles[].RoleId cat foo.json | jp InstanceProfiles[].InstanceProfileId apologies to mobile users!
- stormbrew 9y ago
  I.. really don't agree. I had to use rq because I was working with a toml file and I could not for the life of me figure out even the basic things I was trying to do. The documentation was sparse and unclear compared to jq's relatively concise but clear documentation. In the end I wound up just using rq to convert the toml to JSON so I could use jq on it. It was a while ago so I suppose it's possible I just used it too early in its life or something though.

krat0sprakhar 9y ago

If you use jq[0] and are wondering why Gron, the answer is at the very bottom of the readme: jq is awesome, and a lot more powerful than gron, but with that power comes complexity. gron aims to make it easier to use the tools you already know, like grep and sed. gron's primary purpose is to make it easy to find the path to a value in a deeply nested JSON blob when you don't already know the structure; much of jq's power is unlocked only once you know that structure. [0] - https://stedolan.github.io/jq/ https://stedolan.github.io/jq/

jypepin 9y ago
and this sounds awesome, I'll definitely test Gron. I love jq, but oh boy how complex it is. I find its syntax getting very confusing very quickly as soons as you try to be a little fancy.
- cryptonector 9y ago
  It's a full-blown, dynamically-typed functional programming language. Well, not quite full-blown: it's missing closures of indefinite extent, but still.
cryptonector 9y ago
$ jq -c tostream <<<'{"a":[{"b":2}]}' [["a",0,"b"],2] [["a",0,"b"]] [["a",0]] [["a"]] $ However, filtering that and then reconstructing JSON from that is... not possible at this time: $ jq -c tostream <<<'{"a":[{"b":2}]}'|jq -crn 'fromstream(inputs)' {"a":[{"b":2}]} $ $ jq -c tostream <<<'{"a":[{"b":2}]}'|grep b|jq -crn 'fromstream(inputs)' $ :( The reason is that tostream and fromstream can handle multiple top-level JSON texts, since jq normally does too, but then there's an ambiguity issue to resolve by having a sort of an object terminator. Filtering tostream's output with grep loses the terminators, and so fromstream cannot operate normally. But it should be possible to define a function that does allow this, by, e.g., requiring just one top-level JSON text. The other thing is that a path-based encoding that does not require quotes and commas would be handier -- tostream's output is itself JSON, so it's not shell-friendly. This is gron's brilliant innovation: it's got a path-based encoding of JSON that is easy to deal with in a shell script. (Mind you, I'm not sure that using brackets to denote array indices is all that easy to use, but the need to disambiguate object keys that look like numbers is critical. Also, there's an ambiguity as to keys that have embedded periods ('.') in them. And lastly, even gron can't shake off the string quotes for values.) That jq has the builtin functionality needed to do the same is not good enough if it doesn't actually do it out of the box.

xenomachina 9y ago

A few years ago I made a Python script that does the same thing, minus the reverse mode of converting the assignments back into JSON: https://github.com/xenomachina/jsflat https://github.com/xenomachina/jsflat I've found flattening JSON in this way not only useful for line based tools like grep, but also for understanding unfamiliar JSON. Sometimes it's nice to be able to see the whole path down to the value you're looking at.

deleted 9y ago
[deleted]
- deleted 9y ago
  [deleted]

kbenson 9y ago

This is cool... but it appears to also work as an HTTP client as well? I'm not sure why the additional complexity of including this is needed. I maintain that any HTTP client functionality that supports enough options to be useful is complex and if it supports so few options that it's a toy, why include it? There is non-negligible overhead in keeping track of what shell tools can make network requests, and not being correct and up to date in that area has been the cause of numerous bugs and security issues in tool sets and programs that include and utilize a component like this without realizing it. To many, this might sound like some overly nit-picky complaint, but I maintain if you ask just about anyone that's been in the trenches as a sysadmin for more than a couple years whether they think saving the few characters it takes to use curl and pipe is a good trade-off for the possible unintended consequences of some developer shelling out to this without proper validation from some webapp, they'll tell you no. This tool seems awesome, but keep it simple. It's not like it's a GUI tool and it's hard to pass data between programs. A pipe is perfect here.

vbsteven 9y ago
I came here to say the same thing. It looks like a great tool which I will use frequently on large json files but I don't get why the author bothered to implement a basic HTTP client with some curl-like options when it can just as well read from stdin.
- erric 9y ago
  Maybe in a container build environment like CircleCI where having a small concise toolset is desirable?
cup-of-tea 9y ago
The author just hasn't fully grokked the command line yet.
- kbenson 9y ago
  That seems overly harsh. It's a cool utility, and the author even shows it being used with pipes in numerous ways. I'm just arguing that including non-essential features is a trade-off. Often it's just complexity for ease of use, but I think this one has some possible security implications, and what you get for that may not be worthwhile.

beaugunderson 9y ago

There's some prior art; namely jsonpipe/jsonunpipe: https://github.com/zacharyvoase/jsonpipe https://github.com/zacharyvoase/jsonpipe

kriomant 9y ago

Alternative: use ogrep (https://github.com/kriomant/ogrep-rs https://github.com/kriomant/ogrep-rs / https://github.com/kriomant/ogrep https://github.com/kriomant/ogrep) on pretty-printed JSON.

soheilpro 9y ago

I have written a similar tool called catj [1]. I mostly use it when I need to construct JSON expressions when working with jq. [1] https://github.com/soheilpro/catj https://github.com/soheilpro/catj

yason 9y ago

This kind of tools borderline the very interesting threshold where writing your own tool can be less of a mental effort than discovering, learning and keeping track of these small utilities separately. I've written a similar tool in Python, both for JSON and XML. Especially the JSON version was dead simple, probably fits on a single screen and took 15 minutes to test and write. Surely it didn't have any "features" but it does the job of letting me grep json. Gron is probably 10x more versatile and actually comes with useful features but I'd really have to have pressing needs to do transformations of JSON on a regular basis to switch over. The same applies to libraries in programming languages. There is a very vague threshold, depending on the expressiveness of the language and operating environment as well as the hardness of the problem itself, where it either makes sense to write your own library or reuse an existing one.

bm1362 9y ago

If you’re stuck on a foreign box or don’t want to install a new tool: cat xyz | python -mjson.tool | grep foo

jwilk 9y ago
Python's json.tool is a pretty-printer. It's not anything like gron. (json_pp is another pretty-printer that is likely to be already on your system.)
pm90 9y ago
This assumes that you have python installed ;)

billsmithaustin 9y ago

I might have chosen a name that sounds less like Cron, but regardless, I can see how Gron would be useful.

deleted 9y ago
[deleted]

deleted 9y ago

[deleted]

deleted 9y ago

[deleted]

acobster 9y ago

Thanks for this! "Grep for absolute path" is the use-case that the otherwise awesome `jq` doesn't address well. I've found myself having to iteratively drill down to find the field that has the data I need. One of those "eh, I'll automate this [poorly] someday..." things. :D

tedivm 9y ago

I wrote a tool called jsonsmash[0][1] that's meant for a more explorative view of the JSON files. It basically exposes the data in a minishell, complete with `ls` (with a ton of the standard flags), `cd`, `pwd`, `cat` (which outputs in yaml), and some others. My main use case was to read some json files that were far too big to load in standard editors (210mb+) and for that it has worked great. [0] https://www.npmjs.com/package/jsonsmash https://www.npmjs.com/package/jsonsmash [1] https://blog.tedivm.com/open-source/2017/05/introducing-jsonsmash-work-with-large-json-files-easily/ https://blog.tedivm.com/open-source/2017/05/introducing-json...

pronik 9y ago
This sounds awfully similar to Augeas CLI. Not that I endorse Augeas in any capacity...

coldacid 9y ago

Looks like I have a new Chocolatey package to create and push up when I get home tonight.

aiCeivi9 9y ago

json.Host = "headers.jsontest.com"; json["User-Agent"] = "curl/7.43.0"; Why not use simple notation for eveything without '.' in key?

TomNomNom 9y ago
Author of the tool here. The output is designed to be valid JavaScript, which doesn't allow certain characters in unquoted object keys, like the dash in User-Agent. Using JavaScript's rules for quoting keys makes it a lot easier to specify the grammar (and therefore write the parser); and makes it trivial to 'parse' the output using JavaScript should you want to. There must be _some_ rules in place for when to quote the key (e.g. when there is a dot, equals sign, square brace etc in the key name), so I see no reason to adopt something custom and potentially error-prone when a known-good set of rules already exists. Hope that answers your question well enough!