4 ms·
If the data can conveniently be split in to lines, I wonder if there is an advantage to this format over a single json list e.g. [ {"name":"Bob"},
by remus 3y ago
If the data can conveniently be split in to lines, I wonder if there is an advantage to this format over a single json list e.g.
[
{"name":"Bob"},
{"name":"Alice"},
{"name":"Alison"}
]
You're already working with json anyway so you could just stay in that world rather than mixing new lines and json.
- est 3y agomost JSON parser would have difficulty decoding an array with 1 million rows. For example in Python you could easily eat up all or RAM and swap and gets OOM'd
- zipping1549 3y agoIf we're having milions of line of anything, just go with an actual db, no?
- masklinn 3y agoHow does "an actual db" help you with an event stream e.g. over a socket? Are you going to stream WAL records and ask the client to merge that into their local database?
- rented_mule 3y agoNo, this is often used for logging. In my last job, we were generating on the order of a million lines like this per hour in the log I worked with most often (many millions per hour if you count all the logs in the company). Each line of that log was between a few hundred bytes and ~100 KB. It would take a lot of work to support that in a database. That log was processed by many Spark jobs (some in streaming fashion, others in batch). "Small" summaries were added to a columnar database each night. The main table in that database contained hundreds of columns and billions of rows for more interactive analysis.
- baq 3y agoWhy would I need a db for a meager million rows? Even excel handles a million rows
- DonHopkins 3y agoAn actual db like BigQuery supports importing jsonl directly. https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json https://cloud.google.com/bigquery/docs/loading-data-cloud-st...
- zX41ZdbW 3y agoClickHouse supports it for actual INSERT queries. So instead of INSERT ... VALUES, you write INSERT ... FORMAT JSONEachRow and stream your data. https://clickhouse.com/docs/en/sql-reference/statements/insert-into https://clickhouse.com/docs/en/sql-reference/statements/inse...
- Expurple 3y agoThis doesn't work with streaming, e.g. JSON logs
- krisoft 3y agoThe big benefit of JSON lines over a JSON array that you can easily append new records to your file by just adding new bytes at the end.
- masklinn 3y agoMore generally, the benefit of json lines is that it's trivially streamable, you don't need a streaming json serializer or parser, because each record can be serialized or deserialized by a common eager non-streaming parser.
- beardyw 3y agoIf you have a streaming parser it spits out what it's got and it waits for more data (or an end which may never come). Mostly you would be extracting what you needed from the JSON as it went along. Wondering if jq works that way?
- rzzzt 3y agojq supports both modes. You can use --slurp to pull all newline-separated input into a big array before it starts to process it, otherwise it should produce a result for each complete object parsed (I think): https://jqlang.github.io/jq/manual/#invoking-jq https://jqlang.github.io/jq/manual/#invoking-jq Edit: ah, the streaming described in your comment is also available via --stream, but the format presented to the selection-filtering-transformation stage will be a bit different.
- remus 3y agoAh yes, that is a big practical advantage, thanks for explaining.
- deleted 3y ago[deleted]
- beardyw 3y agoI agree. I think the objections result from JSON having it's roots in JavaScript as an object. I built a streaming JSON parser for a microcontroller with very limited memory. It wasn't too hard to do though it probably doesn't cover edge cases. But it can be done if you change your mindset.
- jbverschoor 3y agoRoots? JSON IS JavaScript. Its roots are: var data = eval(jsonString); This is exactly why it sucks in many ways, but also why it's pretty great.
- oulipo 3y agowell actually they are the same if you say (pseudocode) ``` const jsonLParser = (streamOfLines) => { for (line in streamOfLines) { if line.trim() === "[" || line.trim() === "]" -> skip if line.endswith(',') -> line = line.trimEnd(',') yield JSON.parse(line) } } ``` this allows your API to be consumed both by JSON parsers and JSONL parsers