8 ms·

Show HN: gq – like jq or zq, but you use Go

107 points by errnil 4y ago

Hi HN. I've gotten pretty tired of needing to learn a custom programming language for a tool I use once a week or less. So I figured, it might be easier to pick up if `jq` used a programming language I already know. Voila, gq.

eatonphil 4y ago
Nice work! I also learned about gojq [0] yesterday too. A Go clone of jq that you can also embed in Go code. [0] https://github.com/itchyny/gojq https://github.com/itchyny/gojq Edit: oh unlike jq/gojq your tool doesn't implement the jq language it allows you to script in Go. Cool! And interesting to see your use of the Yaegi interpreter for interpreting the Go. I'm curious, did you find it faster to interpret with Yaegi vs just writing the Go code to a temp file and build/running that? I imagine at some point for larger datasets the latter approach would be faster?
- errnil 4y ago
  Precisely. The first pass generated a temp file and compiled/ran it, but that lead to annoying pauses when trying to run simple scripts (probably 500ms latency? Maybe a tad more?). Yaegi cut that latency dramatically. I think you’re right that this may make it slower for larger scripts or for sufficiently large inputs, however the biggest bottleneck for me with jq isn’t performance but learning it’s language repeatedly.
- tedunangst 4y ago
  yaegi is pretty fast. Not compiled fast, but it can chug through some pretty large tree walks without trouble. (Using it for walking html processing.) Half the work is parsing, which is using the compiled library.
  - eatonphil 4y ago
    Curious what is your scenario for using yaegi and not Go itself? You allow users to run arbitrary Go somewhere?
    - tedunangst 4y ago
      Instant reloads for changes.
bachmeier 4y ago
Looks nice. I might even use it if I wrote Go programs.
johnday 4y ago
I'd give it maybe a day and a half before `hq` makes its way onto the front page. No guesses which language the `h` stands for :)
- ldh 4y ago
  Surely `rq` can't be far behind.
  - wey-gu 4y ago
    Haha 100% sure there will be rq…
    - eatonphil 4y ago
      I think the joke was that it already exists. https://github.com/dflemstr/rq https://github.com/dflemstr/rq
      - ducktective 4y ago
        rq is criminally underrated. It is a universal convertor between json, avro, cbor, message-pack, toml, yaml and csv. It misses XML though, and activity in development ;)
      - wey-gu 4y ago
        wow, thanks!
  - mynameisash 4y ago
    I was going to post this the other day, but my project[0] isn't quite the same as jq. I didn't even know about jq when I had a need for some easy JSONL text extraction. No aggregation or complex querying -- just extracting from arrays/objects in my data. Sorry to disappoint? :) [0] https://github.com/aeshirey/jsx https://github.com/aeshirey/jsx
  - adolph 4y ago
    Two maybe: one for the systems folk and another for data scientologists
mdaniel 4y ago
this tool may interest the same audience: https://github.com/hairyhenderson/gomplate https://github.com/hairyhenderson/gomplate I find it especially handy for poc-ing helm snippets, although regrettably they don't have the same "standard library" of functions
ricardobeat 4y ago
Go is a fine language, but not great for scripting. The verbosity shows in the examples, it’s basically skipping five or six lines of code to read from stdin and import a functional library. Learning the jq syntax will have a much better payoff in the long run.
- nh23423fefe 4y ago
  | gq 'j.Filter(func(n *Node) bool { return strings.Contains(n.String(), "good/") })' | jq 'map(select(contains("good/")))'
  - mccanne 4y ago
    zq good/ -
  - throwaway894345 4y ago
    I like jq, but the parent is right that it's hard to wrap one's head around the way jq thinks about things and how that model maps to its grammar. Even in this example, I couldn't tell you why `map()` wraps everything else, and I always get tripped up on when to use `|` or how to go back and forth between a list and a stream. I don't think the answer is `gq`, but I certainly understand why people try out other languages.
    - nh23423fefe 4y ago
      every filter operates on its input. in this case the input was an array. every map implementation ive seen has type function(a,b) -> list(a) -> list(b) select(contains("good/")) is a function from string -> string|void so map plus a voidable function equals a filter | just means connect 2 filters if you have a stream you turn it into a array with the array constructor [] if you have an array you turn it into a stream with the producer operator .[] the api is super orthogonal and well designed. but you do have to understand the model
      - throwaway894345 4y ago
dmitriid 4y ago
Ok. Is this the new "todo list" or "hacker news clone" trend, but for tools? :) We need competing implementations in more languages! (Ignore my snarky tone because: why not?)
pipeline_peak 4y ago
Why not just pipe into a python program that uses a mature JSON parser? These “*q” things are really boring.
- eatonphil 4y ago
  Is go's json parser less mature than python's?
  - pipeline_peak 4y ago
    I was under the assumption that this thing is using it's own parser, I guess it doesn't, idc. The point I'm trying to make is that JQ depends on the user investing the time to know an esoteric domain language. This thing is fed by Go scripts that get interpreted, at that rate just use a powerful shell language like Python.
    - tedunangst 4y ago
      What if I'm more familiar with go and python is an esoteric language?
      - pipeline_peak 4y ago
        In the world of shell scripts, python isn't an esoteric language....
- joshbaptiste 4y ago
  because many folks just want a single binary and not a full language library
  - pipeline_peak 4y ago
    If you are using whateverq and writing a shell script, everyone else using it needs to have whateverq installed as well....
deleted 4y ago
[deleted]
macintux 4y ago
Discussion on zq from yesterday: https://news.ycombinator.com/item?id=31166956 https://news.ycombinator.com/item?id=31166956
deleted 4y ago
[deleted]
zamalek 4y ago
> Im tired of learning new programming languages. Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous. * buildah shows just how much of a headache Dockerfiles are. We already have a whole ecosystem of awesome scripting languages, such as bash, fish, zx, etc. Instead, we have to deal with the incredibly constrained (and harmfully magic) behavior of docker build. * the whole implerative YAML ecosystem. I have been playing around with Dagger (CUE, a language built to solve real problems), and what a joy compared implerative CI. * lest we forget how JavaScript came to be: https://thenewstack.io/brendan-eich-on-creating-javascript-in-10-days-and-what-hed-do-differently-today/ https://thenewstack.io/brendan-eich-on-creating-javascript-i...
- yessirwhatever 4y ago
  > Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous. I don't understand this mentality. Some programmers enjoy writing programs and some enjoy learning about programming languages. No one is telling all programmers they they need to learn about every programming language, but why paint the whole endeavor as frivolous?
  - zamalek 4y ago
    From what you quoted: > Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do.
  - dijit 4y ago
    I've been a sysadmin (now, devops, SRE, whatever) for over 15 years now, so when you say: > No one is telling all programmers they they need to learn about every programming language I cringe a little. There are about 25 different DSL's in my head from configuring various desired state configuration systems or services which use DSLs, there's countless config file formats ranging for ZONE files to TOML to YAML to HCL and on and on and on. Makefiles, Dockerfiles, Ninja files, Jenkinsfiles etc;etc;etc;etc; DevOps of Yore had to know bash and perl, then it was Ruby, then python, now Go. The treadmill continues, it's _necessary_ to learn those things because you _must_ interact with those things. That's before we get into special languages like AWK which had significant prominence when I was beginning life as a sysadmin too. Nobody is forcing programmers to know every language, but honestly, there's a lot that sysadmins had to learn, and with the rise of DevOps, developers are the SysAdmins of tomorrow: so you have to know what we knew.
pensatoio 4y ago
I love that you wrote this last night in response to the recent front page. :D What a cool project!
- errnil 4y ago
  :) Thank you! It was fun to write.
voidon 4y ago
This is cool and I agree about not needing to learn yet another language. I love jq, but I always have to look at the docs. I recently learned about ctrl-x-e (after 25 years of shell-use!) to edit the one-liners in $EDITOR, so verbose go code might actually work.
- johnisgood 4y ago
  Ctrl-x Ctrl-e for me.
vemv 4y ago
I keep using "My jq killer" as I commented back in 2016: https://news.ycombinator.com/item?id=13091469 https://news.ycombinator.com/item?id=13091469
- sleepybrett 4y ago
  Now my build/ci/cd container needs fucking ruby in it .. no thanks. I broke myself from my 'just use perl instead of awk/sed' for the very same reason. The smaller those containers are the faster my pipeline runs.
renewiltord 4y ago
I use https://github.com/itchyny/gojq https://github.com/itchyny/gojq . I like `jq`'s functional style, but I don't like the way it breaks number precision (or used to). However, for deep `jq` selects I use `<file.json | gron | grep key -C n | norg` where `norg` is `gron --ungron`. I also prefer `gron` for ad-hoc pipelines and `jq` for longer-lasting ones.
sleepybrett 4y ago
Firstmover goes to `jq` all these other tools might be great but honestly it might just be better to use the dev time to add features to jq. For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it. As a guy who does a bunch of devops and works with a bunch of teams to improve their pipelines etc. jq is a tool they all have and i always find in pipelines having to install the new jq flavor of the month every six months is terrible.
- kitd 4y ago
  For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it. As it happens, gojq, mentioned elsewhere in this thread, adds this feature, and quite a few others. Jq has served the industry admirably, but there's always room for improvement.
cabirum 4y ago
Do we even need a tool like that? I mean, just use the language you know to do something with the json you have. Alternatively: the tool should use sql.
- dhagz 4y ago
  Just load the JSON into Mongo/Cassandra and query that. /s
IceWreck 4y ago
Can someone explain why people are creating so many CLI tools (with a very weird and custom syntax) to extract JSON using a very complicated one line command when you can make one off scripts to do the same using python/go/ruby/node/whatever ?
- QuikAccount 4y ago
  I'd argue that it is less about practicality and more about learning a language by recreating a CLI tool.
- UncleEntity 4y ago
  I wrapped boost::property_tree in a python module to read in a rss xml feed and store the items in a json file to track the TV shows I download all because the pure python version was kind of slow. The amount of time I spent on getting all that working is probably a lot less than waiting a few extra seconds on a python script I run every few days so…
- NAR8789 4y ago
  oneliners are generally nicer for fast data exploration than one-off scripts. With a oneliner I can get started instantly, and build incrementally. With a one-off script my first step is boilerplate input handling. Maybe a nice middle ground would be for more language runtimes to have an equivalent to perl's `-p` flag. https://perldoc.perl.org/perlrun#-p https://perldoc.perl.org/perlrun#-p This flag is so convenient that I mostly use `perl -pe` instead of `sed` for regex search-and-replace.
kbrazil 4y ago
Similar in concept to jello[0] which works like jq but uses python syntax. [0] https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello
- techdragon 4y ago
  Thank you for one of my new favourite tools!
rubyist5eva 4y ago
can someone make `bfq` (brainf*ck)?

107 points by errnil 4y ago

eatonphil 4y ago

Nice work! I also learned about gojq [0] yesterday too. A Go clone of jq that you can also embed in Go code. [0] https://github.com/itchyny/gojq https://github.com/itchyny/gojq Edit: oh unlike jq/gojq your tool doesn't implement the jq language it allows you to script in Go. Cool! And interesting to see your use of the Yaegi interpreter for interpreting the Go. I'm curious, did you find it faster to interpret with Yaegi vs just writing the Go code to a temp file and build/running that? I imagine at some point for larger datasets the latter approach would be faster?

errnil 4y ago
Precisely. The first pass generated a temp file and compiled/ran it, but that lead to annoying pauses when trying to run simple scripts (probably 500ms latency? Maybe a tad more?). Yaegi cut that latency dramatically. I think you’re right that this may make it slower for larger scripts or for sufficiently large inputs, however the biggest bottleneck for me with jq isn’t performance but learning it’s language repeatedly.
tedunangst 4y ago
yaegi is pretty fast. Not compiled fast, but it can chug through some pretty large tree walks without trouble. (Using it for walking html processing.) Half the work is parsing, which is using the compiled library.
- eatonphil 4y ago
  Curious what is your scenario for using yaegi and not Go itself? You allow users to run arbitrary Go somewhere?
  - tedunangst 4y ago
    Instant reloads for changes.

bachmeier 4y ago

Looks nice. I might even use it if I wrote Go programs.

johnday 4y ago

I'd give it maybe a day and a half before `hq` makes its way onto the front page. No guesses which language the `h` stands for :)

ldh 4y ago
Surely `rq` can't be far behind.
- wey-gu 4y ago
  Haha 100% sure there will be rq…
  - eatonphil 4y ago
    I think the joke was that it already exists. https://github.com/dflemstr/rq https://github.com/dflemstr/rq
    - ducktective 4y ago
      rq is criminally underrated. It is a universal convertor between json, avro, cbor, message-pack, toml, yaml and csv. It misses XML though, and activity in development ;)
    - wey-gu 4y ago
      wow, thanks!
- mynameisash 4y ago
  I was going to post this the other day, but my project[0] isn't quite the same as jq. I didn't even know about jq when I had a need for some easy JSONL text extraction. No aggregation or complex querying -- just extracting from arrays/objects in my data. Sorry to disappoint? :) [0] https://github.com/aeshirey/jsx https://github.com/aeshirey/jsx
- adolph 4y ago
  Two maybe: one for the systems folk and another for data scientologists

mdaniel 4y ago

this tool may interest the same audience: https://github.com/hairyhenderson/gomplate https://github.com/hairyhenderson/gomplate I find it especially handy for poc-ing helm snippets, although regrettably they don't have the same "standard library" of functions

ricardobeat 4y ago

Go is a fine language, but not great for scripting. The verbosity shows in the examples, it’s basically skipping five or six lines of code to read from stdin and import a functional library. Learning the jq syntax will have a much better payoff in the long run.

nh23423fefe 4y ago
| gq 'j.Filter(func(n *Node) bool { return strings.Contains(n.String(), "good/") })' | jq 'map(select(contains("good/")))'
- mccanne 4y ago
  zq good/ -
- throwaway894345 4y ago
  I like jq, but the parent is right that it's hard to wrap one's head around the way jq thinks about things and how that model maps to its grammar. Even in this example, I couldn't tell you why `map()` wraps everything else, and I always get tripped up on when to use `|` or how to go back and forth between a list and a stream. I don't think the answer is `gq`, but I certainly understand why people try out other languages.
  - nh23423fefe 4y ago
    every filter operates on its input. in this case the input was an array. every map implementation ive seen has type function(a,b) -> list(a) -> list(b) select(contains("good/")) is a function from string -> string|void so map plus a voidable function equals a filter | just means connect 2 filters if you have a stream you turn it into a array with the array constructor [] if you have an array you turn it into a stream with the producer operator .[] the api is super orthogonal and well designed. but you do have to understand the model
    - throwaway894345 4y ago

dmitriid 4y ago

Ok. Is this the new "todo list" or "hacker news clone" trend, but for tools? :) We need competing implementations in more languages! (Ignore my snarky tone because: why not?)

pipeline_peak 4y ago

Why not just pipe into a python program that uses a mature JSON parser? These “*q” things are really boring.

eatonphil 4y ago
Is go's json parser less mature than python's?
- pipeline_peak 4y ago
  I was under the assumption that this thing is using it's own parser, I guess it doesn't, idc. The point I'm trying to make is that JQ depends on the user investing the time to know an esoteric domain language. This thing is fed by Go scripts that get interpreted, at that rate just use a powerful shell language like Python.
  - tedunangst 4y ago
    What if I'm more familiar with go and python is an esoteric language?
    - pipeline_peak 4y ago
      In the world of shell scripts, python isn't an esoteric language....
joshbaptiste 4y ago
because many folks just want a single binary and not a full language library
- pipeline_peak 4y ago
  If you are using whateverq and writing a shell script, everyone else using it needs to have whateverq installed as well....

deleted 4y ago

[deleted]

macintux 4y ago

Discussion on zq from yesterday: https://news.ycombinator.com/item?id=31166956 https://news.ycombinator.com/item?id=31166956

deleted 4y ago

[deleted]

zamalek 4y ago

> Im tired of learning new programming languages. Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous. * buildah shows just how much of a headache Dockerfiles are. We already have a whole ecosystem of awesome scripting languages, such as bash, fish, zx, etc. Instead, we have to deal with the incredibly constrained (and harmfully magic) behavior of docker build. * the whole implerative YAML ecosystem. I have been playing around with Dagger (CUE, a language built to solve real problems), and what a joy compared implerative CI. * lest we forget how JavaScript came to be: https://thenewstack.io/brendan-eich-on-creating-javascript-in-10-days-and-what-hed-do-differently-today/ https://thenewstack.io/brendan-eich-on-creating-javascript-i...

yessirwhatever 4y ago
> Hear hear! Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do. But these one-off languages are becoming ridiculous. I don't understand this mentality. Some programmers enjoy writing programs and some enjoy learning about programming languages. No one is telling all programmers they they need to learn about every programming language, but why paint the whole endeavor as frivolous?
- zamalek 4y ago
  From what you quoted: > Building bona-fide compilers that have interesting concepts, or just for fun, is a worthwhile thing to do.
- dijit 4y ago
  I've been a sysadmin (now, devops, SRE, whatever) for over 15 years now, so when you say: > No one is telling all programmers they they need to learn about every programming language I cringe a little. There are about 25 different DSL's in my head from configuring various desired state configuration systems or services which use DSLs, there's countless config file formats ranging for ZONE files to TOML to YAML to HCL and on and on and on. Makefiles, Dockerfiles, Ninja files, Jenkinsfiles etc;etc;etc;etc; DevOps of Yore had to know bash and perl, then it was Ruby, then python, now Go. The treadmill continues, it's _necessary_ to learn those things because you _must_ interact with those things. That's before we get into special languages like AWK which had significant prominence when I was beginning life as a sysadmin too. Nobody is forcing programmers to know every language, but honestly, there's a lot that sysadmins had to learn, and with the rise of DevOps, developers are the SysAdmins of tomorrow: so you have to know what we knew.

pensatoio 4y ago

I love that you wrote this last night in response to the recent front page. :D What a cool project!

errnil 4y ago
:) Thank you! It was fun to write.

voidon 4y ago

This is cool and I agree about not needing to learn yet another language. I love jq, but I always have to look at the docs. I recently learned about ctrl-x-e (after 25 years of shell-use!) to edit the one-liners in $EDITOR, so verbose go code might actually work.

johnisgood 4y ago
Ctrl-x Ctrl-e for me.

vemv 4y ago

I keep using "My jq killer" as I commented back in 2016: https://news.ycombinator.com/item?id=13091469 https://news.ycombinator.com/item?id=13091469

sleepybrett 4y ago
Now my build/ci/cd container needs fucking ruby in it .. no thanks. I broke myself from my 'just use perl instead of awk/sed' for the very same reason. The smaller those containers are the faster my pipeline runs.

renewiltord 4y ago

I use https://github.com/itchyny/gojq https://github.com/itchyny/gojq . I like `jq`'s functional style, but I don't like the way it breaks number precision (or used to). However, for deep `jq` selects I use `<file.json | gron | grep key -C n | norg` where `norg` is `gron --ungron`. I also prefer `gron` for ad-hoc pipelines and `jq` for longer-lasting ones.

sleepybrett 4y ago

Firstmover goes to `jq` all these other tools might be great but honestly it might just be better to use the dev time to add features to jq. For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it. As a guy who does a bunch of devops and works with a bunch of teams to improve their pipelines etc. jq is a tool they all have and i always find in pipelines having to install the new jq flavor of the month every six months is terrible.

kitd 4y ago
For instance json is a subset of yaml, maybe internally jq they could use yaml libraries, consume either, output either json or input format by default and give me a flag for yaml output if I want it. As it happens, gojq, mentioned elsewhere in this thread, adds this feature, and quite a few others. Jq has served the industry admirably, but there's always room for improvement.

cabirum 4y ago

Do we even need a tool like that? I mean, just use the language you know to do something with the json you have. Alternatively: the tool should use sql.

dhagz 4y ago
Just load the JSON into Mongo/Cassandra and query that. /s

IceWreck 4y ago

Can someone explain why people are creating so many CLI tools (with a very weird and custom syntax) to extract JSON using a very complicated one line command when you can make one off scripts to do the same using python/go/ruby/node/whatever ?

QuikAccount 4y ago
I'd argue that it is less about practicality and more about learning a language by recreating a CLI tool.
UncleEntity 4y ago
I wrapped boost::property_tree in a python module to read in a rss xml feed and store the items in a json file to track the TV shows I download all because the pure python version was kind of slow. The amount of time I spent on getting all that working is probably a lot less than waiting a few extra seconds on a python script I run every few days so…
NAR8789 4y ago
oneliners are generally nicer for fast data exploration than one-off scripts. With a oneliner I can get started instantly, and build incrementally. With a one-off script my first step is boilerplate input handling. Maybe a nice middle ground would be for more language runtimes to have an equivalent to perl's `-p` flag. https://perldoc.perl.org/perlrun#-p https://perldoc.perl.org/perlrun#-p This flag is so convenient that I mostly use `perl -pe` instead of `sed` for regex search-and-replace.

kbrazil 4y ago

Similar in concept to jello[0] which works like jq but uses python syntax. [0] https://github.com/kellyjonbrazil/jello https://github.com/kellyjonbrazil/jello

techdragon 4y ago
Thank you for one of my new favourite tools!

rubyist5eva 4y ago

can someone make `bfq` (brainf*ck)?