5 ms·
I've been using jsonlines for logfiles from about 5 years before it acquired a name. With line delimited logfile objects, it's easy to grep for a string of int
by greglindahl 9y ago
I've been using jsonlines for logfiles from about 5 years before it acquired a name.
With line delimited logfile objects, it's easy to grep for a string of interest and then only parse the lines that match -- much more efficient than parsing an entire logfile to pick out 0.01% of the lines.
That's not the usecase talked about in this article, but it is a usecase that's important to me.
- beau 9y agoIt's also easy to load them into Google's BigQuery: https://cloud.google.com/bigquery/docs/loading-data#supported_data_formats https://cloud.google.com/bigquery/docs/loading-data#supporte...
- stingraycharles 9y agoI was just about to name that as an example. Newline separated JSON is a so much simpler solution to the problem than streaming JSON parsers. They're easier to understand, and it's trivial to implement. From a purity point of view, sure, streaming JSON is more "correct", but I wouldn't call it the better solution.
- beached_whale 9y agoit should be trivial to write a tool to stream a json file with proper newlines
- greglindahl 9y agoNot sure what you're getting at -- my tools that generate json logfiles do trivially add newlines where I want them. jq's "compact" mode will reformat files that don't have newlines where I want them.
- beached_whale 9y agoI mean that the tools to add the newlines to a compact json file with minimal whitespace are not going to be the bottle neck. I am fairly sure that grep uses a forward streaming access pattern and not random access so inserting a relatively simple command into the pipeline to insert newlines, like jq, probably isn't going to slow it down. The tool itself wouldn't be difficult to write, or like you said, just use jq
- pkulak 9y ago> That's not the usecase talked about in this article, but it is a usecase that's important to me. Yeah, and I agree with you 100%. But lot's of things that are great for log storage aren't appropriate for an API.
- hiccuphippo 9y agoTogether with jq, jsonlines make for great logging.
- bringtheaction 9y agohttp://jsonlines.org/ http://jsonlines.org/ > Each Line is a Valid JSON Value This seems like the better option I agree.
- smnscu 9y agoGoogle's lead me to this one today, lol: http://ndjson.org/ http://ndjson.org/ I'd make fun of NIH but I'm so guilty of it too.
- ivanhoe 9y agoJsonlines is also an excellent intermediary format for working with any huge datasets that are just too big to be loaded into the memory all at once, and where data is hierarchical so csv is not an option. Compared to XML you save a lot in the total file size, and it's just as easy to parse.
- laumars 9y agoI've just googled what jsonlines was and it turns out i've been using it for a while in log files too where I wanted JSON objects but needed it to behave like random access memory (exactly like you described with grep). I hadn't even realised it was a formal thing. Thank you