8 ms·

Newline Delimited JSON

47 points by xingyzt 5y ago

throwaway894345 5y ago
I was wondering how this compared to jsonlines/jsonl https://jsonlines.org https://jsonlines.org > This page describes the JSON Lines text format, also called newline-delimited JSON. And then I saw this at the bottom of the ndjson.org page: > Site forked from jsonlines.org So is this a fork of the site only, or a fork of the jsonlines standard?
- SahAssar 5y ago
  Yeah, the only difference I can see from the specs are that ndjson SHOULD be served with the type `application/x-ndjson` while jsonl does not seem to specify a mediatype. There is similar issues in both projects: - https://github.com/wardi/jsonlines/issues/22 https://github.com/wardi/jsonlines/issues/22 - https://github.com/ndjson/ndjson-spec/issues/35 https://github.com/ndjson/ndjson-spec/issues/35
- divbzero 5y ago
  I was curious whether .jsonl or .ndjson is used more often. Based on frequency in GitHub it looks like both are popular but .jsonl enjoys slightly wider usage: extension:jsonl 62K results [1] extension:ndjson 41K results [2] [1]: https://github.com/search?q=extension:jsonl https://github.com/search?q=extension:jsonl [2]: https://github.com/search?q=extension:ndjson https://github.com/search?q=extension:ndjson
BugsJustFindMe 5y ago
This is an awful lot of foofaraw to basically say "no newlines in your stream of discrete messages except between messages". The proposal appears to have absolutely nothing to do with JSON, and the only connection is that their example messages happen to be JSON but could just as easily be anything else.
- hirundo 5y ago
  It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally specifying the syntax alternative is worthwhile.
  - BugsJustFindMe 5y ago
    > It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.
    - jmull 5y ago
      The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
    - wolfgang42 5y ago
      The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.
callesgg 5y ago
I know I have used something like this code before, it is basicly what they are talking about. json_decode(“[“.implode($lines,”,\n”).”]”)
- wolfgang42 5y ago
  Sure, that's one way to parse this format. The advantage is that individual records can also be manipulated by JSON-agnostic code, so you can easily do things like stream processing.
spullara 5y ago
This has been Twitter's stream format since they had a stream format for their firehose in probably 2007.
- bigtones 5y ago
  Right, it's pretty much the standard already.
jmull 5y ago
Makes sense to me. I've been using this and calling it "jsonlog" format, but now I can switch to a hopefully more standard name.
jasonpeacock 5y ago
And here I am wrapping my results in a result object line a newb: { results: [ { ... }, { ... }, { ... } ] }
- cosmotic 5y ago
  or just: [ { ... }, { ... }, { ... }, ]
- iddqd 5y ago
  Line separated JSON has its uses, for example when streaming data.
  - BugsJustFindMe 5y ago
    The only reason people care about newlines for a stream is because they've arbitrarily chosen to fetch bytes from the stream until the next newline sequence (readline instead of read). But you could just as easily look for a different sequence, like the ASCII record separator character which was invented exactly for this task, and then you wouldn't have to destructively strip newlines from your input.
    - nerdponx 5y ago
      I'll switch from \n to \x1E when cat, head, text editors, etc. start supporting the latter as a synonym for the former.
    - morelisp 5y ago
      We tell our customers our streaming API uses jsonlines for documentation purposes, but we actually just decode in a loop until EOF, and 400 at the first decode error. No separators necessary at all.
  - amelius 5y ago
    Ok, but what if you want to transmit two streams? You'd get something like Interlaced-newline-delimited-JSON. And so the list of required file formats knows no end.
kazinator 5y ago
I've done a similar thing with Lisp sometimes: implementing hacks so that a list in a file always looks like this: ((lorem ipsum ..) (quorat est demonstrandum ...) ... ...) Only major line-breaks between top-level elements.
monkey26 5y ago
Haven't we already been doing this for years? JQ has supported this for a long time as well.
- wolfgang42 5y ago
  NDJSON is a fork of https://jsonlines.org/ https://jsonlines.org/, which has been around since 2013. I’m sure the concept is even older, but it can be helpful to have a name and a spec (no matter how small) for everyone to agree on.
- nerdponx 5y ago
  Yes, but having a spec (however trivial) means I can tell someone to use "nd-json" and send them a link to the spec, without having to debate about escaping, utf-8, etc.
- deleted 5y ago
  [deleted]
tengbretson 5y ago
I could be wrong, but isnt the ElasticSearch API implemented like this?
- CommieBobDole 5y ago
  I know the Elasticsearch bulk API uses it - I can see why, because you're sending one request that basically says, "Hey, do all of these separate operations at once", but I've cursed it a few times because it makes it a slight pain to build bulk requests programmatically.
kosolam 5y ago
Err... Am I the only one that see json objects on multiple lines? And that’s it...
- bryanlarsen 5y ago
  Exactly. It gives a name and a mime-type to a trivial and commonly used format. That's it. But useful none the less.
alpb 5y ago
Why not throw in the inline comments feature to the mix if you're gonna end up using a different spec/different parser anyway? This is an 8+ year old spec and I don't think it has caught much adoption. If you're really into hot having commas, maybe you can do a post-processing step with sed or in-memory string replacement to add/remove "," and \n? Then you're still getting the same JSON experience under the covers where programs process it?
- wolfgang42 5y ago
  The idea behind this spec is that it’s an ultra-simple framing format for a stream of JSON documents; you would use the same JSON parser as always but e.g. inside a loop that reads each line from the input. It has lots of adoption, but since it’s such a simple format many of those have invented it independently and don’t call it by name.
- laurencerowe 5y ago
  > Why not throw in the inline comments feature to the mix if you're gonna end up using a different spec/different parser anyway? Doing so would complicate adoption, since parsing is essentially just: for line in file: obj = json.loads(line) > This is an 8+ year old spec and I don't think it has caught much adoption It's seen huge adoption, essentially becoming the standard logging format for many Node JS applications. Usually I've heard it referred to as JSON Lines: https://jsonlines.org https://jsonlines.org
  - nerdponx 5y ago
    The name "JSON Lines" is a lot better in my opinion. I hope the two Git repo maintainers can agree on some kind of merge, but they've had almost a decade to do it and neither one seems interested. There's also an RFC (7464) for more or less the same thing, except using ASCII record separators instead of newline characters: https://datatracker.ietf.org/doc/html/rfc7464 https://datatracker.ietf.org/doc/html/rfc7464 And yes, I have used this at $JOB and I intend to use it in future ${JOB}s. That's adoption.
    - 5y ago
maerF0x0 5y ago
basically json streams? https://en.wikipedia.org/wiki/JSON_streaming https://en.wikipedia.org/wiki/JSON_streaming In my day job we ndjson pretty extensively and it's been pretty good to us. It has far better characteristics than passing a giant json array of objects (you have to hit the closing bracket on the array to know if it's valid)
dheera 5y ago
Why not just use YAML, which accomplishes effectively the same goals, and already has a decent amount of traction? In general I use YAML for human-edited configuration files, because they generate clean diffs, and JSON for machine-to-machine communication, because of the wealth of obscenely optimized parsnips for every language.
- nerdponx 5y ago
  I don't think YAML accomplishes the same goals. I like YAML, and I think people hate on it more than it deserves (at least, in its current form). And yes, you can stream YAML documents by concatenating them with document separators. But ND-JSON/JSONLines/RFC-7464 is mostly a machine-to-machine format anyway. Maybe we should be able to customize the delimiter? The media type "application/stream+json; separator=\n" would be JSONLines/ND-JSON, "application/stream+json; separator=\x1E" would be RFC 7464, "application/stream+json; separator=\n...\n" would theoretically be valid YAML, etc.

47 points by xingyzt 5y ago

I was wondering how this compared to jsonlines/jsonl https://jsonlines.org https://jsonlines.org > This page describes the JSON Lines text format, also called newline-delimited JSON. And then I saw this at the bottom of the ndjson.org page: > Site forked from jsonlines.org So is this a fork of the site only, or a fork of the jsonlines standard?

SahAssar 5y ago
Yeah, the only difference I can see from the specs are that ndjson SHOULD be served with the type `application/x-ndjson` while jsonl does not seem to specify a mediatype. There is similar issues in both projects: - https://github.com/wardi/jsonlines/issues/22 https://github.com/wardi/jsonlines/issues/22 - https://github.com/ndjson/ndjson-spec/issues/35 https://github.com/ndjson/ndjson-spec/issues/35
divbzero 5y ago
I was curious whether .jsonl or .ndjson is used more often. Based on frequency in GitHub it looks like both are popular but .jsonl enjoys slightly wider usage: extension:jsonl 62K results [1] extension:ndjson 41K results [2] [1]: https://github.com/search?q=extension:jsonl https://github.com/search?q=extension:jsonl [2]: https://github.com/search?q=extension:ndjson https://github.com/search?q=extension:ndjson

BugsJustFindMe 5y ago

This is an awful lot of foofaraw to basically say "no newlines in your stream of discrete messages except between messages". The proposal appears to have absolutely nothing to do with JSON, and the only connection is that their example messages happen to be JSON but could just as easily be anything else.

hirundo 5y ago
It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally specifying the syntax alternative is worthwhile.
- BugsJustFindMe 5y ago
  > It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.
  - jmull 5y ago
    The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
  - wolfgang42 5y ago
    The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.

callesgg 5y ago

I know I have used something like this code before, it is basicly what they are talking about. json_decode(“[“.implode($lines,”,\n”).”]”)

wolfgang42 5y ago
Sure, that's one way to parse this format. The advantage is that individual records can also be manipulated by JSON-agnostic code, so you can easily do things like stream processing.

spullara 5y ago

This has been Twitter's stream format since they had a stream format for their firehose in probably 2007.

bigtones 5y ago
Right, it's pretty much the standard already.

jmull 5y ago

Makes sense to me. I've been using this and calling it "jsonlog" format, but now I can switch to a hopefully more standard name.

jasonpeacock 5y ago

And here I am wrapping my results in a result object line a newb: { results: [ { ... }, { ... }, { ... } ] }

cosmotic 5y ago
or just: [ { ... }, { ... }, { ... }, ]
iddqd 5y ago
Line separated JSON has its uses, for example when streaming data.
- BugsJustFindMe 5y ago
  The only reason people care about newlines for a stream is because they've arbitrarily chosen to fetch bytes from the stream until the next newline sequence (readline instead of read). But you could just as easily look for a different sequence, like the ASCII record separator character which was invented exactly for this task, and then you wouldn't have to destructively strip newlines from your input.
  - nerdponx 5y ago
    I'll switch from \n to \x1E when cat, head, text editors, etc. start supporting the latter as a synonym for the former.
  - morelisp 5y ago
    We tell our customers our streaming API uses jsonlines for documentation purposes, but we actually just decode in a loop until EOF, and 400 at the first decode error. No separators necessary at all.
- amelius 5y ago
  Ok, but what if you want to transmit two streams? You'd get something like Interlaced-newline-delimited-JSON. And so the list of required file formats knows no end.

kazinator 5y ago

I've done a similar thing with Lisp sometimes: implementing hacks so that a list in a file always looks like this: ((lorem ipsum ..) (quorat est demonstrandum ...) ... ...) Only major line-breaks between top-level elements.

monkey26 5y ago

Haven't we already been doing this for years? JQ has supported this for a long time as well.

wolfgang42 5y ago
NDJSON is a fork of https://jsonlines.org/ https://jsonlines.org/, which has been around since 2013. I’m sure the concept is even older, but it can be helpful to have a name and a spec (no matter how small) for everyone to agree on.
nerdponx 5y ago
Yes, but having a spec (however trivial) means I can tell someone to use "nd-json" and send them a link to the spec, without having to debate about escaping, utf-8, etc.
deleted 5y ago
[deleted]

tengbretson 5y ago

I could be wrong, but isnt the ElasticSearch API implemented like this?

CommieBobDole 5y ago
I know the Elasticsearch bulk API uses it - I can see why, because you're sending one request that basically says, "Hey, do all of these separate operations at once", but I've cursed it a few times because it makes it a slight pain to build bulk requests programmatically.

kosolam 5y ago

Err... Am I the only one that see json objects on multiple lines? And that’s it...

bryanlarsen 5y ago
Exactly. It gives a name and a mime-type to a trivial and commonly used format. That's it. But useful none the less.

alpb 5y ago

Why not throw in the inline comments feature to the mix if you're gonna end up using a different spec/different parser anyway? This is an 8+ year old spec and I don't think it has caught much adoption. If you're really into hot having commas, maybe you can do a post-processing step with sed or in-memory string replacement to add/remove "," and \n? Then you're still getting the same JSON experience under the covers where programs process it?

wolfgang42 5y ago
The idea behind this spec is that it’s an ultra-simple framing format for a stream of JSON documents; you would use the same JSON parser as always but e.g. inside a loop that reads each line from the input. It has lots of adoption, but since it’s such a simple format many of those have invented it independently and don’t call it by name.
laurencerowe 5y ago
> Why not throw in the inline comments feature to the mix if you're gonna end up using a different spec/different parser anyway? Doing so would complicate adoption, since parsing is essentially just: for line in file: obj = json.loads(line) > This is an 8+ year old spec and I don't think it has caught much adoption It's seen huge adoption, essentially becoming the standard logging format for many Node JS applications. Usually I've heard it referred to as JSON Lines: https://jsonlines.org https://jsonlines.org
- nerdponx 5y ago
  The name "JSON Lines" is a lot better in my opinion. I hope the two Git repo maintainers can agree on some kind of merge, but they've had almost a decade to do it and neither one seems interested. There's also an RFC (7464) for more or less the same thing, except using ASCII record separators instead of newline characters: https://datatracker.ietf.org/doc/html/rfc7464 https://datatracker.ietf.org/doc/html/rfc7464 And yes, I have used this at $JOB and I intend to use it in future ${JOB}s. That's adoption.
  - 5y ago

maerF0x0 5y ago

basically json streams? https://en.wikipedia.org/wiki/JSON_streaming https://en.wikipedia.org/wiki/JSON_streaming In my day job we ndjson pretty extensively and it's been pretty good to us. It has far better characteristics than passing a giant json array of objects (you have to hit the closing bracket on the array to know if it's valid)

dheera 5y ago

Why not just use YAML, which accomplishes effectively the same goals, and already has a decent amount of traction? In general I use YAML for human-edited configuration files, because they generate clean diffs, and JSON for machine-to-machine communication, because of the wealth of obscenely optimized parsnips for every language.

nerdponx 5y ago
I don't think YAML accomplishes the same goals. I like YAML, and I think people hate on it more than it deserves (at least, in its current form). And yes, you can stream YAML documents by concatenating them with document separators. But ND-JSON/JSONLines/RFC-7464 is mostly a machine-to-machine format anyway. Maybe we should be able to customize the delimiter? The media type "application/stream+json; separator=\n" would be JSONLines/ND-JSON, "application/stream+json; separator=\x1E" would be RFC 7464, "application/stream+json; separator=\n...\n" would theoretically be valid YAML, etc.