5 ms·

Is jq slow? I have only worked with datasets up to 1mb but I’ve never had a performance issue that wasn’t attributed to my error.

minaguib 6y ago
Yes I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions. jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.
nicoburns 6y ago
jq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file). The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.
- davesnx 6y ago
  Right, q doesn't support streaming so it will manage a 90GB JSON. I should specify that on the performance section, Thanks!
- arethuza 6y ago
  Out of interest, what created a 90GB JSON file?
  - lllr_finger 6y ago
    Not OP, but I routinely call a specific HTTP API for millions of entities or pull down entire Kafka topics - all in JSON format. For various reasons those are the canonical sources of data and/or the most performant, so I end up ripping through GBs and GBs of JSON when troubleshooting/reporting on things.
  - nicoburns 6y ago
    A Firebase Realtime Database backup file
  - hobofan 6y ago
    I don't think it's quite 90GB, but I've processed Wikidata dumps in the same order of magnitude before (which are one JSON object per line) with jq, and it could've certainly been faster.
    - arethuza 6y ago
      I was wondering about that - whether they are one single JSON array/object or one per line.
de_watcher 6y ago
It's very slow. It was immediately standing out in our automated tests when we've added a json protocol to our system and used jq to test some assertions.
davesnx 6y ago
Nobody said that jq is slow.
jonstewart 6y ago
Yes. I was looking to embed it in a tool, but decided against it after looking at its implementation. It parses the expression with a stack and executes it directly, and its JSON parsing is much the same. I doubt the parsing would be close to competitive with RapidJSON, let alone simd-json. The conditionals and pointer chasing of such an implementation are stumbling blocks to performance. The C code is clean enough as C code goes, but fairly monolithic. And it’s C, so it’s not noticeably slow until you start processing GB. But it would probably take a rewrite to improve its performance significantly.

minaguib 6y ago

Yes I often have to pluck out attributes from streams of json records (1 json object per line) - often millions/billions. jq is almost always the bottleneck in the pipeline at 100% CPU - so much so that we often add an fgrep to the left side of the pipeline to minimize the input to jq as much as possible.

nicoburns 6y ago

jq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file). The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.

davesnx 6y ago
Right, q doesn't support streaming so it will manage a 90GB JSON. I should specify that on the performance section, Thanks!
arethuza 6y ago
Out of interest, what created a 90GB JSON file?
- lllr_finger 6y ago
  Not OP, but I routinely call a specific HTTP API for millions of entities or pull down entire Kafka topics - all in JSON format. For various reasons those are the canonical sources of data and/or the most performant, so I end up ripping through GBs and GBs of JSON when troubleshooting/reporting on things.
- nicoburns 6y ago
  A Firebase Realtime Database backup file
- hobofan 6y ago
  I don't think it's quite 90GB, but I've processed Wikidata dumps in the same order of magnitude before (which are one JSON object per line) with jq, and it could've certainly been faster.
  - arethuza 6y ago
    I was wondering about that - whether they are one single JSON array/object or one per line.

de_watcher 6y ago

It's very slow. It was immediately standing out in our automated tests when we've added a json protocol to our system and used jq to test some assertions.

davesnx 6y ago

Nobody said that jq is slow.

jonstewart 6y ago

Yes. I was looking to embed it in a tool, but decided against it after looking at its implementation. It parses the expression with a stack and executes it directly, and its JSON parsing is much the same. I doubt the parsing would be close to competitive with RapidJSON, let alone simd-json. The conditionals and pointer chasing of such an implementation are stumbling blocks to performance. The C code is clean enough as C code goes, but fairly monolithic. And it’s C, so it’s not noticeably slow until you start processing GB. But it would probably take a rewrite to improve its performance significantly.