23 ms·

This is great, JQ is brilliant. I love JQ so much we implemented a subset of JQ in Clojure so that our users could use it to munge/filter data in our product (

by d_t_w 3y ago

This is great, JQ is brilliant. I love JQ so much we implemented a subset of JQ in Clojure so that our users could use it to munge/filter data in our product (JVM and browser based Kafka tooling). One of the most fun coding pieces I've done, though I am a bit odd and I love writing grammars (big shoutout to Instaparse![1]). I learned through my implementation that JQ is a LISP-2[2] which surprised me as it didn't feel obvious from the grammar. [1] https://github.com/Engelberg/instaparse https://github.com/Engelberg/instaparse [2] https://github.com/jqlang/jq/wiki/jq-Language-Description#:~:text=The%20jq%20language%20is%20a,their%20use%20after%20being%20deallocated https://github.com/jqlang/jq/wiki/jq-Language-Description#:~....

chatmasta 3y ago
I can't stand jq. I realize this is an unpopular opinion, and our codebase at work has plenty of jq in the bash scripts, some of it even code that I wrote. I begrudgingly use it when it's the best option for me. But something about it rubs me the wrong way - I think it's the unintuitive query syntax and the need to search for every minute step of what I'm trying to do, and the frequency with which that leads to cryptic answers that I can only decipher if I am some sort of jq expert. But I have this instinctive reaction to all DSL languages that embed themselves into strings, like htmx and tailwind (both embedded in attribute string values). I realize some people like it, and it's a well-made piece of software, and I will even admit that sometimes there is no better choice. But I guess I just hate that it's necessary? I guess I could also admit it's the least-bad option, in the sense that it's a vast improvement over various sed/awk/cut monstrosities when it comes to parsing JSON in bash. Certainly once you find the right incantation, it's perfect - it transforms some raw stdin into parsed JSON that you can manipulate into exactly what you need. But for me, it ranks right next to regex in terms of "things I (don't) want to see in my code." I hate that the jq command is always some indecipherable string in the middle of the script. The only real alternative I've ever used is piping to a Python program that I define inline in a heredoc, but that ends up being at least as nasty as the JQ script.
- michaelcampbell 3y ago
  `jq` and `GNU Parallel` share a world in my brain where I know they're wonderful tools, but I spend more time grokking the syntax of each one as rarely as I need either, than just writing a bash/sed/awk/perl, ruby, or python script to do what I need.
- cryptonector 3y ago
  Python is so much harder to process JSON data in than jq, that that is how I got into working with and on jq almost a decade ago.
  - chatmasta 3y ago
    Yeah, Python is like 10-20x the number of lines required to do the same thing as jq (especially with the boilerplate of consuming stdin), but that's also why it's more readable. But generally I agree - I would choose jq over some weird bash/python hybrid most of the time. I just wish it was more immediately readable.
    - lathiat 3y ago
      I've found Ruby much nicer for writing dirty parsing logic like this in a "real" language, it lets you be more terse and "DRY" than Python. Which in bigger software projects doesn't hurt me as much but when I'm primarily trying to write something that otherwise would be well handled by SQL or JQ I found Ruby the better middleground for me.
    - cryptonector 3y ago
      Simple jq programs are easy to read because simple jq programs are just path expressions, and the jq language is optimized to make path expressions easy to read. Path expressions like .[].commit | select(.author == "Tom Hudson") which basically says "find all commits by Tom Hudson" in the input. `.[]` iterates all the values in its input (whether the input be an array or an object). `.commit` gets the value of the "commit" key in the input object. You concatenate path expressions with `|`, and array/object index expressions you can just concatenate w/o `|`, so `.[]` and `.commit` can be `.[] | .commit` and also `.[].commit`. Calls to functions like `select()` whose bodies are path expressions are.. also path expressions. Perhaps the most brilliant thing about jq is that you can assign to arbitrarily complex path expressions, so you can: (.[].commit | select(.author == "Tom Hudson")) = "Anon" The syntax is strange probably because of this trying to make path expressions so trivial and readable. jq programs get hard to read mainly when you go beyond path expressions, especially when you start doing reductions. The problem is that it resembles point free programming in Haskell, which is really not for everyone. The other thing is that jq is very much a functional programming language, and that takes getting used to.
dekelpilli 3y ago
Just adding that a JQ implementation already exists on the JVM[1], although it's not 100% complete. [1] https://github.com/eiiches/jackson-jq https://github.com/eiiches/jackson-jq
mnutt 3y ago
jq is great for letting users munge their data; we do something similar letting users provision an incoming webhook endpoint, send us arbitrary json data, and set up mappings to do useful things with their data, along with regression tests, monitoring, etc. jq makes the majority of cases straightforward (basically json dot notation) and the long tail possible.
mingodad 3y ago
I also just add a JQ parser/grammar to the online LALR(1)/FLEX grammar editor/tester at https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ select "Jq parser (partially working)" from examples then click "Parse" to see a parser tree of the source in "Input source". Any feedback is welcome !
- cryptonector 3y ago
  Hey, that's an awesome playground! Thank you for this!
- yoz 3y ago
  Thanks to this, I found your fantastic list of interesting and powerful parser projects: https://mingodad.github.io/parsertl-playground/ https://mingodad.github.io/parsertl-playground/ Thank you so much for compiling it! A related question for you and anyone else into this kind of tooling: if you had to automate some structural edits across a codebase that contains a wide range of popular languages (say: C++, C#, Java, Ruby, Python), and you had to do it with a single tool, which tool would you use?
Dionakra 3y ago
Is your product available yet? I would be glad to try it out!
- d_t_w 3y ago
  Yes, sure is! https://kpow.io/ https://kpow.io/ You can get started with the Community Edition for free, it includes our JQ implementation (we call it kJQ - https://docs.kpow.io/features/data-inspect/kjq-filters https://docs.kpow.io/features/data-inspect/kjq-filters) Just shout if you need any help.

by d_t_w 3y ago

chatmasta 3y ago

I can't stand jq. I realize this is an unpopular opinion, and our codebase at work has plenty of jq in the bash scripts, some of it even code that I wrote. I begrudgingly use it when it's the best option for me. But something about it rubs me the wrong way - I think it's the unintuitive query syntax and the need to search for every minute step of what I'm trying to do, and the frequency with which that leads to cryptic answers that I can only decipher if I am some sort of jq expert. But I have this instinctive reaction to all DSL languages that embed themselves into strings, like htmx and tailwind (both embedded in attribute string values). I realize some people like it, and it's a well-made piece of software, and I will even admit that sometimes there is no better choice. But I guess I just hate that it's necessary? I guess I could also admit it's the least-bad option, in the sense that it's a vast improvement over various sed/awk/cut monstrosities when it comes to parsing JSON in bash. Certainly once you find the right incantation, it's perfect - it transforms some raw stdin into parsed JSON that you can manipulate into exactly what you need. But for me, it ranks right next to regex in terms of "things I (don't) want to see in my code." I hate that the jq command is always some indecipherable string in the middle of the script. The only real alternative I've ever used is piping to a Python program that I define inline in a heredoc, but that ends up being at least as nasty as the JQ script.

michaelcampbell 3y ago
`jq` and `GNU Parallel` share a world in my brain where I know they're wonderful tools, but I spend more time grokking the syntax of each one as rarely as I need either, than just writing a bash/sed/awk/perl, ruby, or python script to do what I need.
cryptonector 3y ago
Python is so much harder to process JSON data in than jq, that that is how I got into working with and on jq almost a decade ago.
- chatmasta 3y ago
  Yeah, Python is like 10-20x the number of lines required to do the same thing as jq (especially with the boilerplate of consuming stdin), but that's also why it's more readable. But generally I agree - I would choose jq over some weird bash/python hybrid most of the time. I just wish it was more immediately readable.
  - lathiat 3y ago
    I've found Ruby much nicer for writing dirty parsing logic like this in a "real" language, it lets you be more terse and "DRY" than Python. Which in bigger software projects doesn't hurt me as much but when I'm primarily trying to write something that otherwise would be well handled by SQL or JQ I found Ruby the better middleground for me.
  - cryptonector 3y ago
    Simple jq programs are easy to read because simple jq programs are just path expressions, and the jq language is optimized to make path expressions easy to read. Path expressions like .[].commit | select(.author == "Tom Hudson") which basically says "find all commits by Tom Hudson" in the input. `.[]` iterates all the values in its input (whether the input be an array or an object). `.commit` gets the value of the "commit" key in the input object. You concatenate path expressions with `|`, and array/object index expressions you can just concatenate w/o `|`, so `.[]` and `.commit` can be `.[] | .commit` and also `.[].commit`. Calls to functions like `select()` whose bodies are path expressions are.. also path expressions. Perhaps the most brilliant thing about jq is that you can assign to arbitrarily complex path expressions, so you can: (.[].commit | select(.author == "Tom Hudson")) = "Anon" The syntax is strange probably because of this trying to make path expressions so trivial and readable. jq programs get hard to read mainly when you go beyond path expressions, especially when you start doing reductions. The problem is that it resembles point free programming in Haskell, which is really not for everyone. The other thing is that jq is very much a functional programming language, and that takes getting used to.

dekelpilli 3y ago

Just adding that a JQ implementation already exists on the JVM[1], although it's not 100% complete. [1] https://github.com/eiiches/jackson-jq https://github.com/eiiches/jackson-jq

mnutt 3y ago

jq is great for letting users munge their data; we do something similar letting users provision an incoming webhook endpoint, send us arbitrary json data, and set up mappings to do useful things with their data, along with regression tests, monitoring, etc. jq makes the majority of cases straightforward (basically json dot notation) and the long tail possible.

mingodad 3y ago

I also just add a JQ parser/grammar to the online LALR(1)/FLEX grammar editor/tester at https://mingodad.github.io/parsertl-playground/playground/ https://mingodad.github.io/parsertl-playground/playground/ select "Jq parser (partially working)" from examples then click "Parse" to see a parser tree of the source in "Input source". Any feedback is welcome !

cryptonector 3y ago
Hey, that's an awesome playground! Thank you for this!
yoz 3y ago
Thanks to this, I found your fantastic list of interesting and powerful parser projects: https://mingodad.github.io/parsertl-playground/ https://mingodad.github.io/parsertl-playground/ Thank you so much for compiling it! A related question for you and anyone else into this kind of tooling: if you had to automate some structural edits across a codebase that contains a wide range of popular languages (say: C++, C#, Java, Ruby, Python), and you had to do it with a single tool, which tool would you use?

Dionakra 3y ago

Is your product available yet? I would be glad to try it out!

d_t_w 3y ago
Yes, sure is! https://kpow.io/ https://kpow.io/ You can get started with the Community Edition for free, it includes our JQ implementation (we call it kJQ - https://docs.kpow.io/features/data-inspect/kjq-filters https://docs.kpow.io/features/data-inspect/kjq-filters) Just shout if you need any help.