3 ms·
For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of fail
by sebcat 10y ago
For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of failure when accepting input from external sources. There should be no difference between:
{"foo": 1}{"bar": 2}
and
{"foo": 1}
{"bar": 2}
or any variations of common whitespace between the objects. Just parse objects incrementally, one at a time. Most JSON libraries supports this.
As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case.
Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.
- tracker1 10y agoIt's simply easier to determine when an object ends with a delimiter... {"dtm":"2016-...",...,meta:{...}} That requires more complicated input processing/streaming than read until '\n' then parse the record...
- eatonphil 10y agoNot meaningfully more complicated. You know when you've reached the end of your object of you are actually parsing it, so you also know when the next one can begin.
- tracker1 10y agoWhen you're working with a platform that has native JSON parsing, it generally doesn't support streams as described... so you either count braces and quotes, create a stream processor or use a delimiter... a delimiter is an easy enough solution, also allows for easier viewing in a text editor. It's about being pragmatic.
- JdeBP 10y agoActually, no, one is not necessarily parsing it. Xyr point was that using line delimiters allows one to retain one-log-record-per line semantics and use the log stream with text processing tools that understand those semantics but that do not understand JSON.