5 ms·

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed

by lapser 4y ago

I have actually fully replaced my jq installation with gojq (including an `ln -s gojq jq`) for a few years, and no script has broken so far. I'm super impressed by the jq compatibility. If you are going down this route, do be careful with performance. I don't know which is more performant as I've never really had to work with large data sets, but I can't help but feel jq will be faster than gojq in such case. I have no benchmarks backing this up, but who knows, maybe someone will benchmark both. One of my favourite features is the fact that error messages are actually legible, unlike jq.

brundolf 4y ago
It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json https://github.com/davesnx/query-json After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr https://github.com/brundonsmith/jqr
- cerved 4y ago
  jq is particularly bad at large stream progressing
- oever 4y ago
  Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... A fast tool to split a json file like that into a format with one json file per line would already help a lot.
  - brundolf 4y ago
    Mine can :)
hyperpallium2 4y ago
jq uses bison (gnu's yacc), which is a nightmare for error diagnosis. Additionally, the founder (though brilliant - or maybe because brilliant) wouldn't accept improvements in error reporting.
aarchi 4y ago
To see if gojq works even with complex jq programs, I tested it on my wsjq[0] Whitespace language interpreter, which uses most of the advanced jq features. It impressively appears to support the full jq language, though I uncovered a bug[1] in gojq. gojq's arbitrary-precision integer support will be useful (jq just uses 64-bit floating-point), though I suspect it will have performance regressions, since it uses math/big, instead of GMP. [0]: https://github.com/andrewarchi/wsjq https://github.com/andrewarchi/wsjq [1]: https://github.com/itchyny/gojq/issues/186 https://github.com/itchyny/gojq/issues/186

by lapser 4y ago

brundolf 4y ago

It's very possible it could be faster; jq seems to actually be fairly unoptimized. This implementation in OCaml was featured on HN a while back and it trashes the original jq in performance: https://github.com/davesnx/query-json https://github.com/davesnx/query-json After seeing that one I did my own (less-complete) version in Rust and managed to squeeze out even more performance in the operations it supports: https://github.com/brundonsmith/jqr https://github.com/brundonsmith/jqr

cerved 4y ago
jq is particularly bad at large stream progressing
oever 4y ago
Working with large json files is hard to parallelize. Just filtering the objects in a root array can take very long. jqr and gojq both die with OOM when running on large files like https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz https://dumps.wikimedia.org/wikidatawiki/entities/latest-all... A fast tool to split a json file like that into a format with one json file per line would already help a lot.
- brundolf 4y ago
  Mine can :)

hyperpallium2 4y ago

jq uses bison (gnu's yacc), which is a nightmare for error diagnosis. Additionally, the founder (though brilliant - or maybe because brilliant) wouldn't accept improvements in error reporting.

aarchi 4y ago

To see if gojq works even with complex jq programs, I tested it on my wsjq[0] Whitespace language interpreter, which uses most of the advanced jq features. It impressively appears to support the full jq language, though I uncovered a bug[1] in gojq. gojq's arbitrary-precision integer support will be useful (jq just uses 64-bit floating-point), though I suspect it will have performance regressions, since it uses math/big, instead of GMP. [0]: https://github.com/andrewarchi/wsjq https://github.com/andrewarchi/wsjq [1]: https://github.com/itchyny/gojq/issues/186 https://github.com/itchyny/gojq/issues/186