3 ms·
jq is convenient, but I don't see the draw in building data processing pipelines on it. It's like writing complex software in shell. Recently, I found myself w
by quotemstr 1y ago
jq is convenient, but I don't see the draw in building data processing pipelines on it. It's like writing complex software in shell.
Recently, I found myself wanting to do a join by filename on two sets of about 300,000 files. Tried bashing my head against jq with INDEX and various tricks and couldn't get the runtime below minutes.
Then I just gave up, fired up Python, loaded the dataset into Pandas, and did a join. Completed too fast to notice.
- porridgeraisin 1y ago> bash Did the `join` command help?
- quotemstr 1y agoNo. join in jq is just syntactic sugar over object property lookup anyway. Turns out index() was the expensive step.
- porridgeraisin 1y agoI meant the coreutils join command.
- ghProTip 1y agoCreated this tiny JavaScript project and published on npm for when you just can't be arsed remembering/learning JQ: https://www.npmjs.com/package/jsling https://www.npmjs.com/package/jsling . Not shilling it so much as saying.. IMHO, for a lot of use cases, there are far better approaches than dealing with the occasional JQ struggle bus ride.