3 ms·
I really like just encoding each object as JSON and then concatinating them with a new line between. Allows parsing and streaming without any special libraries
by jlundberg 1y ago
I really like just encoding each object as JSON and then concatinating them with a new line between.
Allows parsing and streaming without any special libraries and allow for an unlimited amount of data (with objects being reasonably sized).
Usually gives these files the .jsonlines suffix when stored on disk.
Allows for batch process without requiring huge amounts of memory.
- kondro 1y agoMe too, and it's quite a common technique. https://en.wikipedia.org/wiki/JSON_streaming https://en.wikipedia.org/wiki/JSON_streaming
- 0x6c6f6c 1y agoBased on this thread that's called NDJSON Newline Delimited JSON TIL
- keitmo 1y agoIt's also known as JSONL (JSON Lines).
- tracker1 1y agoI'm pretty sure jsonl was a bit earlier as a term, but ndjson is now the more prominent term used for this... been using this approach for years though, when I first started using Mongo/Elastic for denormalized data, I'd also backup that same data to S3 as .jsonl.gz Leaps and bounds better than XMl at least.
- deleted 1y ago[deleted]
- codesnik 1y agojq allows to covert normal json document to jsonlines and back, though it does it much faster if it can slurp an original doc into memory (no --stream option)