4 ms·
jq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file). The main weakness seems to be
by nicoburns 6y ago
jq is pretty fast in my experience. But there have been cases where I've wanted it to be faster (dealing with a 90GB JSON file).
The main weakness seems to be streaming use cases (not having the whole file in memory at once). These are supported, but the syntax is quite awkward.
- davesnx 6y agoRight, q doesn't support streaming so it will manage a 90GB JSON. I should specify that on the performance section, Thanks!
- arethuza 6y agoOut of interest, what created a 90GB JSON file?
- lllr_finger 6y agoNot OP, but I routinely call a specific HTTP API for millions of entities or pull down entire Kafka topics - all in JSON format. For various reasons those are the canonical sources of data and/or the most performant, so I end up ripping through GBs and GBs of JSON when troubleshooting/reporting on things.
- nicoburns 6y agoA Firebase Realtime Database backup file
- hobofan 6y agoI don't think it's quite 90GB, but I've processed Wikidata dumps in the same order of magnitude before (which are one JSON object per line) with jq, and it could've certainly been faster.
- arethuza 6y agoI was wondering about that - whether they are one single JSON array/object or one per line.