3 ms·
The big benefit of JSON lines over a JSON array that you can easily append new records to your file by just adding new bytes at the end.
by krisoft 3y ago
The big benefit of JSON lines over a JSON array that you can easily append new records to your file by just adding new bytes at the end.
- masklinn 3y agoMore generally, the benefit of json lines is that it's trivially streamable, you don't need a streaming json serializer or parser, because each record can be serialized or deserialized by a common eager non-streaming parser.
- beardyw 3y agoIf you have a streaming parser it spits out what it's got and it waits for more data (or an end which may never come). Mostly you would be extracting what you needed from the JSON as it went along. Wondering if jq works that way?
- rzzzt 3y agojq supports both modes. You can use --slurp to pull all newline-separated input into a big array before it starts to process it, otherwise it should produce a result for each complete object parsed (I think): https://jqlang.github.io/jq/manual/#invoking-jq https://jqlang.github.io/jq/manual/#invoking-jq Edit: ah, the streaming described in your comment is also available via --stream, but the format presented to the selection-filtering-transformation stage will be a bit different.
- remus 3y agoAh yes, that is a big practical advantage, thanks for explaining.