3 ms·
I love idea of "json lines" which is implicitly here, in jq. You can convert giant json array of objects to "json lines" and back with jq. A lot of tools could
by codesnik 6y ago
I love idea of "json lines" which is implicitly here, in jq. You can convert giant json array of objects to "json lines" and back with jq.
A lot of tools could produce them, and then you can mix and match jq, simple unix grep, head etc, because each unix line of your stream is a complete parsable json, guaranteed.
back in a day I've loaded a lot of data to postgresql and elastic search after preprocessing it with very simple but powerful chain of CSV parsers (I've used CSVfix), jq, sort, grep, etc.
- chrisweekly 6y agoThanks, codesnik -- good tip! My favorite tool in this area is `lnav` (https://lnav.org https://lnav.org), a scriptable / chainable mini-ETL with embedded sqlite. Oh and tiny correction (intended as helpful not nit-picking), the idiom is "back in the day" not "back in a day". :)
- kemitchell 6y agoThis concept has a name: "ndjson", for "newline-delimited JSON". http://ndjson.org/ http://ndjson.org/ Some tools from the Node.js universe. `npm install $etc`. https://www.npmjs.com/package/ndjson-cli https://www.npmjs.com/package/ndjson-cli
- enw 6y agondjson has more than a few times saved the scalability concerns of my technical designs.
- matt-attack 6y agoI use https://www.npmjs.com/package/concatjson https://www.npmjs.com/package/concatjson to get a Stream interface in node.
- kh_hk 6y agojson-lines is as much of a name for the concept as ndjson is https://jsonlines.org https://jsonlines.org https://github.com/ndjson/ndjson.github.io/issues/1 https://github.com/ndjson/ndjson.github.io/issues/1
- rlt 6y agoIt sure would be cool if all the common Unix tools had JSON lines / ndjson output mode. It could also be done as a library of adapters. e.x. something like j ls Would run “ls”, parse the output, and re-output it as JSON lines.