9 ms·

I know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I can manage `jq .`, `jq .foo` and `jq -r`, but beyond that, the DSL is j

I know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I can manage `jq .`, `jq .foo` and `jq -r`, but beyond that, the DSL is just opaque to me. And every time I try to learn, I get lost in a maze of a manpage: https://manpages.debian.org/jq https://manpages.debian.org/jq I mean there are useful tips in there, but I rarely find what I need. I usually find it simpler to write a Python script (which ships with `json` because it's "batteries included") and operate on lists and dicts the normal way... It's longer, but at least I don't need to learn a new programming language (if we can call jq that...)

jldugger 6y ago
AFAICT, it's basically xpath applied to json. Which isn't helpful, but perhaps explains why it's so obtuse.
- de_watcher 6y ago
  And I still have no idea why jq is slow af.
  - IshKebab 6y ago
    Seems pretty fast to me considering it is reading JSON. JSON is difficult to read quickly.
- lucideer 6y ago
  I know xpath pretty well, and cannot wrap my head around jq's DSL. It might have borrowed from elements of xpath's more obtuse expressions, but simple xpath seems much simpler (to me) than simple jq.
kureikain 6y ago
I cannot remember the syntax and have to search for it every time but it's very powerful. You basically give a DSL to process json. I can do map, filter and even math. The reason it's popular partly because it's available from both Centos/Debian based. Just use `yum` and `apt`. Most of time, you maybe limited with what have on a container. And `jq` may even pre-install by DevOps. Second is it play well with unix tool, pipe and friend.
ur-whale 6y ago
I can totally relate. I think the only way would be a long web page with a massive list of examples to "solve problem X". But the DSL is ...byzantine, to say the least.
dragonwriter 6y ago
> [...] but at least I don't need to learn a new programming language (if we can call jq that...) jq has looping, branching, recursion (with tail recursion optimization), variable assignment, modules, I/O... Yeah, it's a programming language.
cube2222 6y ago
I agree, jq is very advanced when everything most people want is simple json transformation and highlighting. I've made jql[0] for that reason. Check it out, it's less featureful, but much simpler with a uniform lispy syntax. [0]: https://github.com/cube2222/jql https://github.com/cube2222/jql
re 6y ago
> I know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I don't think this is uncommon. People generally reach for jq for simple path expression evaluation against JSON objects, and never get deeper into it than that. It seems like the kind of thing you could re-implement yourself in an afternoon. However, as soon as you start taking advantage of some of the more complex functionality--say, a program like `.entries[] | select(.size > 1024) | .name`--there's a disquieting feeling of "what the hell is actually happening here?" I took some time a while back to really dig into how jq programs work, and was surprised to discover how deep and powerful jq really is, while being built out of some very simple fundamental building blocks. Most of the jq builtin functions are actually implemented in jq itself[1]; even more of them could be, but aren't for the sake of performance. And the power and self-consistency of jq made sense when I found out that the creator Stephen Dolan is an "actual" Computer Scientist (in the academic sense) with extensive programming language research experience[2]. The main obstacle I see for most people is the lack of of an accessible introduction to how jq actually works when it comes to streams, function calls, generators, and backtracking. They're explained somewhat in the docs and on the project wiki[3][4][5], but in a fairly blunt way that assumes existing familiarity with terminology and concepts. It takes some effort to learn initially, but once you understand it, everything falls into place. I'm hopeful that subsequent installments in this "mastering" series can explain how the jq model works in an approachable way. [1] https://github.com/stedolan/jq/blob/master/src/builtin.jq https://github.com/stedolan/jq/blob/master/src/builtin.jq [2] http://stedolan.net/research/ http://stedolan.net/research/ [3] https://stedolan.github.io/jq/manual/#Advancedfeatures https://stedolan.github.io/jq/manual/#Advancedfeatures [4] https://github.com/stedolan/jq/wiki/Advanced-Topics https://github.com/stedolan/jq/wiki/Advanced-Topics [5] https://github.com/stedolan/jq/wiki/Internals:-backtracking https://github.com/stedolan/jq/wiki/Internals:-backtracking
enriquto 6y ago
> the DSL is just opaque to me. May I suggest "gron"? As compared to jq it is simpler (good!) has less features (good!) has no DSL (good!) and it is less powerful (good!, in many cases). It is a tool that "expands" json into standalone lines, and you can further process them using the standard tools grep, sed, cut, sort, awk... EDIT: for example, if you have this json: { "outdir" : "out3", "data" : [ {"img" : "img_01.jpg"}, {"img" : "img_02.jpg"} ], "roi" : { "x" : 150, "y" : 150, "w" : 700, "h" : 700 }, "margin_h": 20, "margin_v": 5, "tile_size" : 300, "resolution": 0.5 } running it through gron produces this outdir = "out3" data[0].img = "img_01.jpg" data[1].img = "img_02.jpg" roi.x = 150 roi.y = 150 roi.w = 700 roi.h = 700 margin_h = 20 margin_v = 5 tile_size = 300 resolution = 0.5 that some people find really convenient to deal with.
- fold_left 6y ago
  https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
- ddgflorida 6y ago
  This online tool also produces assignment statements from JSON: https://www.convertjson.com/json-path-list.htm https://www.convertjson.com/json-path-list.htm (note: creator here)
1_player 6y ago
Let's just say it: jq is an amazing tool, but the DSL is just bad. This is not to shame the developers but to voice our issues with it since this seems to be the prevailing sentiment, perhaps some brave soul will try and simplify it for human brains.
kitd 6y ago
Jq is really powerful but I think the bit it gets wrong is in array handling and filtering arrays. It is such a common thing that some syntactic sugar would be justified, but jq requires you to reach for map() and select() to do even the simplest stuff, and I can never remember if I should filter the `.array` or `.array[]`. Yq [1] syntax for that is much better, and yq can accept JSON files too. [1] https://github.com/mikefarah/yq https://github.com/mikefarah/yq
- otabdeveloper4 6y ago
  Yeah, the fact that jq has both 'streams of values' and 'arrays of values', and that the two are somehow completely orthogonal and support different operations is a huge design wart, in my opinion.
seanwilson 6y ago
Yeah, I find jq similar to writing regexes: I always have to look up the syntax, only get it working after some confusion why my patterns aren't matching, then forget it all in a few days so have to relearn it again later. Any more alternates to Python or Node that are lightweight to add to a project? I'd rather use Node but that's a massive dependency for a non-Node project.
- code-faster 6y ago
  You can use any language with json support, which is most languages. In python, you can have a jq like experience with def f(data): # put code here data = json.load(sys.stdin) output = f(data) print(json.dumps(output))
code-faster 6y ago
Is there anything in my tutorial that I can help clarify for you?
mason55 6y ago
Agree and I also struggle with the docs. The examples are so trivial that they’re useless if you’re trying to use the examples to learn how to use them in context. I think the examples for any() are any(true, false) and any(true, true). Fair play but not super helpful if I’m new and trying to understand how this fits into a typical jq filter.
waynesonfire 6y ago
This resonates with me. I don't need more example of basic selects.. json is a tree structure and jq is a tool that allows us to select elements from a tree and transform them. I would like to see a document that talks about jq from this more abstract perspective and what capabilities exist in this context. For example, can I select two arbitrary branches? Can I merge / move branches? Once we have a grasp of tree manipulation capabilities that are made available through the DSL we can lookup the syntax. Can I select a node and apply filters to it's children? There are so many ways in which a tree can be manipulated and I personally need a higher level description of what I can accomplish with jq.
thiht 6y ago
The man page might not be the best way to learn jq. Did you read the documentation on has website? It's pretty comprehensive, with straightforward examples. Spending one hour or so actually learning jq is a good time investment.

jldugger 6y ago

AFAICT, it's basically xpath applied to json. Which isn't helpful, but perhaps explains why it's so obtuse.

de_watcher 6y ago
And I still have no idea why jq is slow af.
- IshKebab 6y ago
  Seems pretty fast to me considering it is reading JSON. JSON is difficult to read quickly.
lucideer 6y ago
I know xpath pretty well, and cannot wrap my head around jq's DSL. It might have borrowed from elements of xpath's more obtuse expressions, but simple xpath seems much simpler (to me) than simple jq.

kureikain 6y ago

I cannot remember the syntax and have to search for it every time but it's very powerful. You basically give a DSL to process json. I can do map, filter and even math. The reason it's popular partly because it's available from both Centos/Debian based. Just use `yum` and `apt`. Most of time, you maybe limited with what have on a container. And `jq` may even pre-install by DevOps. Second is it play well with unix tool, pipe and friend.

ur-whale 6y ago

I can totally relate. I think the only way would be a long web page with a massive list of examples to "solve problem X". But the DSL is ...byzantine, to say the least.

dragonwriter 6y ago

> [...] but at least I don't need to learn a new programming language (if we can call jq that...) jq has looping, branching, recursion (with tail recursion optimization), variable assignment, modules, I/O... Yeah, it's a programming language.

cube2222 6y ago

I agree, jq is very advanced when everything most people want is simple json transformation and highlighting. I've made jql[0] for that reason. Check it out, it's less featureful, but much simpler with a uniform lispy syntax. [0]: https://github.com/cube2222/jql https://github.com/cube2222/jql

re 6y ago

> I know I might be a dissenting opinion here, but I can never wrap my head around `jq`. I don't think this is uncommon. People generally reach for jq for simple path expression evaluation against JSON objects, and never get deeper into it than that. It seems like the kind of thing you could re-implement yourself in an afternoon. However, as soon as you start taking advantage of some of the more complex functionality--say, a program like `.entries[] | select(.size > 1024) | .name`--there's a disquieting feeling of "what the hell is actually happening here?" I took some time a while back to really dig into how jq programs work, and was surprised to discover how deep and powerful jq really is, while being built out of some very simple fundamental building blocks. Most of the jq builtin functions are actually implemented in jq itself[1]; even more of them could be, but aren't for the sake of performance. And the power and self-consistency of jq made sense when I found out that the creator Stephen Dolan is an "actual" Computer Scientist (in the academic sense) with extensive programming language research experience[2]. The main obstacle I see for most people is the lack of of an accessible introduction to how jq actually works when it comes to streams, function calls, generators, and backtracking. They're explained somewhat in the docs and on the project wiki[3][4][5], but in a fairly blunt way that assumes existing familiarity with terminology and concepts. It takes some effort to learn initially, but once you understand it, everything falls into place. I'm hopeful that subsequent installments in this "mastering" series can explain how the jq model works in an approachable way. [1] https://github.com/stedolan/jq/blob/master/src/builtin.jq https://github.com/stedolan/jq/blob/master/src/builtin.jq [2] http://stedolan.net/research/ http://stedolan.net/research/ [3] https://stedolan.github.io/jq/manual/#Advancedfeatures https://stedolan.github.io/jq/manual/#Advancedfeatures [4] https://github.com/stedolan/jq/wiki/Advanced-Topics https://github.com/stedolan/jq/wiki/Advanced-Topics [5] https://github.com/stedolan/jq/wiki/Internals:-backtracking https://github.com/stedolan/jq/wiki/Internals:-backtracking

enriquto 6y ago

> the DSL is just opaque to me. May I suggest "gron"? As compared to jq it is simpler (good!) has less features (good!) has no DSL (good!) and it is less powerful (good!, in many cases). It is a tool that "expands" json into standalone lines, and you can further process them using the standard tools grep, sed, cut, sort, awk... EDIT: for example, if you have this json: { "outdir" : "out3", "data" : [ {"img" : "img_01.jpg"}, {"img" : "img_02.jpg"} ], "roi" : { "x" : 150, "y" : 150, "w" : 700, "h" : 700 }, "margin_h": 20, "margin_v": 5, "tile_size" : 300, "resolution": 0.5 } running it through gron produces this outdir = "out3" data[0].img = "img_01.jpg" data[1].img = "img_02.jpg" roi.x = 150 roi.y = 150 roi.w = 700 roi.h = 700 margin_h = 20 margin_v = 5 tile_size = 300 resolution = 0.5 that some people find really convenient to deal with.

fold_left 6y ago
https://github.com/tomnomnom/gron https://github.com/tomnomnom/gron
ddgflorida 6y ago
This online tool also produces assignment statements from JSON: https://www.convertjson.com/json-path-list.htm https://www.convertjson.com/json-path-list.htm (note: creator here)

1_player 6y ago

Let's just say it: jq is an amazing tool, but the DSL is just bad. This is not to shame the developers but to voice our issues with it since this seems to be the prevailing sentiment, perhaps some brave soul will try and simplify it for human brains.

kitd 6y ago

Jq is really powerful but I think the bit it gets wrong is in array handling and filtering arrays. It is such a common thing that some syntactic sugar would be justified, but jq requires you to reach for map() and select() to do even the simplest stuff, and I can never remember if I should filter the `.array` or `.array[]`. Yq [1] syntax for that is much better, and yq can accept JSON files too. [1] https://github.com/mikefarah/yq https://github.com/mikefarah/yq

otabdeveloper4 6y ago
Yeah, the fact that jq has both 'streams of values' and 'arrays of values', and that the two are somehow completely orthogonal and support different operations is a huge design wart, in my opinion.

seanwilson 6y ago

Yeah, I find jq similar to writing regexes: I always have to look up the syntax, only get it working after some confusion why my patterns aren't matching, then forget it all in a few days so have to relearn it again later. Any more alternates to Python or Node that are lightweight to add to a project? I'd rather use Node but that's a massive dependency for a non-Node project.

code-faster 6y ago
You can use any language with json support, which is most languages. In python, you can have a jq like experience with def f(data): # put code here data = json.load(sys.stdin) output = f(data) print(json.dumps(output))

code-faster 6y ago

Is there anything in my tutorial that I can help clarify for you?

mason55 6y ago

Agree and I also struggle with the docs. The examples are so trivial that they’re useless if you’re trying to use the examples to learn how to use them in context. I think the examples for any() are any(true, false) and any(true, true). Fair play but not super helpful if I’m new and trying to understand how this fits into a typical jq filter.

waynesonfire 6y ago

This resonates with me. I don't need more example of basic selects.. json is a tree structure and jq is a tool that allows us to select elements from a tree and transform them. I would like to see a document that talks about jq from this more abstract perspective and what capabilities exist in this context. For example, can I select two arbitrary branches? Can I merge / move branches? Once we have a grasp of tree manipulation capabilities that are made available through the DSL we can lookup the syntax. Can I select a node and apply filters to it's children? There are so many ways in which a tree can be manipulated and I personally need a higher level description of what I can accomplish with jq.

thiht 6y ago

The man page might not be the best way to learn jq. Did you read the documentation on has website? It's pretty comprehensive, with straightforward examples. Spending one hour or so actually learning jq is a good time investment.