6 ms·

This is an awful lot of foofaraw to basically say "no newlines in your stream of discrete messages except between messages". The proposal appears to have absolu

This is an awful lot of foofaraw to basically say "no newlines in your stream of discrete messages except between messages". The proposal appears to have absolutely nothing to do with JSON, and the only connection is that their example messages happen to be JSON but could just as easily be anything else.

hirundo 5y ago
It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally specifying the syntax alternative is worthwhile.
- BugsJustFindMe 5y ago
  > It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.
  - jmull 5y ago
    The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
  - wolfgang42 5y ago
    The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.
    - BugsJustFindMe 5y ago
      > you do still need to deal with the situation where the character shows up in your data JSON doesn't support binary data without conversion to string first.
      - wolfgang42 5y ago
        I thought that you said that JSON was incidental and unimportant? My point is that using RS does still force escaping, in the general case; it’s not a panacea. You still need to figure out how to deal with your delimiter character appearing in the data. The way JSON Lines does this is by specifying the use of (a slightly limited subset of) JSON.
        BugsJustFindMe 5y ago
        > I thought that you said that JSON was incidental and unimportant? It is for this, but if we're already declaring a restriction to JSON then we're likewise declaring a restriction to unicode. All delimited data requires escaping if your delimiter can be part of a no-delimit sequence. At least a character explicitly intended for delimiting isn't part of any no-delimit sequences representable in unicode whereas \n is.
iaabtpbtpnn 5y ago
The reason why it works with JSON is that JSON strings are not allowed to contain real newline characters. The spec says they have to be escaped. So newline works as a field delimiter, but only because JSON excludes it from field values.
- BugsJustFindMe 5y ago
  Every compactible format allows physically marking line breaks. JSON isn't very special in this regard.
  - iaabtpbtpnn 5y ago
    I am not familiar with the phrase "compactible format", but it's not about allowing, it's about requiring. I can be sure that my JSON message contains no literal newlines, because if it did then it wouldn't be JSON. Contrast with any format that allows literal newlines in values, for example CSV...
    - BugsJustFindMe 5y ago
      > I am not familiar with the phrase "compactible format" You know how C++ lines end with semicolons (or how Javascript lines _can_ end that way if you want)? And how that means you can put all of the instructions in a function on one hyperlong line? That's compactible. Any format where you are able to smush all the lines within a functional group together into one long line is compactible. You can use the ASCII record separator character instead of newlines if you prefer. Separating records without colliding with newlines is literally what it's for.
      - zvr 5y ago
        In the two languages mentioned, semicolons play a different role: statement terminator vs. statements separator .
    - recursive 5y ago
      newlines are explicitly allowed in json. https://www.json.org/json-en.html https://www.json.org/json-en.html See "whitespace". In order to be used in ndjson, valid json must have its newlines removed.
nerdponx 5y ago
Sure, except then you don't know what the format of each message in the stream is. This way you know that it's JSON. Nobody is stopping anyone from making ND-XML, ND-Sexpr, ND-Messagepack, et al.
- BugsJustFindMe 5y ago
  Knowing simply that something uses JSON syntax is itself insufficient to actually do anything with the data. Sorry, but you cannot escape communicating semantic understanding by throwing a "parts are JSON" mimetype header on it. JSON is just not a self-describing format, so you really can't escape the fact that it's only ever useful when either talking to yourself or when the sender is already saying "and when I say 'age' that is a number of months and means..." In both cases you already know that the messages are formatted with JSON.
  - nerdponx 5y ago
    There are 3 layers here: 1. Records separated by ASCII "\n" characters, wherein each record cannot contain an "\n" character 2. Each record is UTF-8 text in JSON format 3. The schema of each record is <something> ND-JSON takes care of the first two. You would still have to use JSONSchema for the third. This is not very different from using JSON without the "ND" part. Except: 1. Now there's the "ND" part, and I have something standardized I can use for quick and easy understanding/agreement when collaborating, especially outside my organization. 2. Plain JSON supports exactly 1 record per stream or file. ND-JSON supports any natural number of records, or an infinite number of records. That's useful to me. I really don't understand the resistance here. Nobody is trotting this out claiming its some hot new thing. It's an unoffensive standardization of a pretty basic data format that people have been using for 10+ years. If you have no use for it, then just ignore it. Consider that the people using it might not be morons; they might just have different needs that are different from yours.
    - BugsJustFindMe 5y ago
      > 2. Plain JSON supports exactly 1 record per stream or file. Unless you wrap it in an array, which is all this is doing.

hirundo 5y ago

It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally specifying the syntax alternative is worthwhile.

BugsJustFindMe 5y ago
> It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.
- jmull 5y ago
  The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
- wolfgang42 5y ago
  The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.
  - BugsJustFindMe 5y ago
    > you do still need to deal with the situation where the character shows up in your data JSON doesn't support binary data without conversion to string first.
    - wolfgang42 5y ago
      I thought that you said that JSON was incidental and unimportant? My point is that using RS does still force escaping, in the general case; it’s not a panacea. You still need to figure out how to deal with your delimiter character appearing in the data. The way JSON Lines does this is by specifying the use of (a slightly limited subset of) JSON.
      - BugsJustFindMe 5y ago
        > I thought that you said that JSON was incidental and unimportant? It is for this, but if we're already declaring a restriction to JSON then we're likewise declaring a restriction to unicode. All delimited data requires escaping if your delimiter can be part of a no-delimit sequence. At least a character explicitly intended for delimiting isn't part of any no-delimit sequences representable in unicode whereas \n is.

iaabtpbtpnn 5y ago

The reason why it works with JSON is that JSON strings are not allowed to contain real newline characters. The spec says they have to be escaped. So newline works as a field delimiter, but only because JSON excludes it from field values.

BugsJustFindMe 5y ago
Every compactible format allows physically marking line breaks. JSON isn't very special in this regard.
- iaabtpbtpnn 5y ago
  I am not familiar with the phrase "compactible format", but it's not about allowing, it's about requiring. I can be sure that my JSON message contains no literal newlines, because if it did then it wouldn't be JSON. Contrast with any format that allows literal newlines in values, for example CSV...
  - BugsJustFindMe 5y ago
    > I am not familiar with the phrase "compactible format" You know how C++ lines end with semicolons (or how Javascript lines _can_ end that way if you want)? And how that means you can put all of the instructions in a function on one hyperlong line? That's compactible. Any format where you are able to smush all the lines within a functional group together into one long line is compactible. You can use the ASCII record separator character instead of newlines if you prefer. Separating records without colliding with newlines is literally what it's for.
    - zvr 5y ago
      In the two languages mentioned, semicolons play a different role: statement terminator vs. statements separator .
  - recursive 5y ago
    newlines are explicitly allowed in json. https://www.json.org/json-en.html https://www.json.org/json-en.html See "whitespace". In order to be used in ndjson, valid json must have its newlines removed.

nerdponx 5y ago

Sure, except then you don't know what the format of each message in the stream is. This way you know that it's JSON. Nobody is stopping anyone from making ND-XML, ND-Sexpr, ND-Messagepack, et al.

BugsJustFindMe 5y ago
Knowing simply that something uses JSON syntax is itself insufficient to actually do anything with the data. Sorry, but you cannot escape communicating semantic understanding by throwing a "parts are JSON" mimetype header on it. JSON is just not a self-describing format, so you really can't escape the fact that it's only ever useful when either talking to yourself or when the sender is already saying "and when I say 'age' that is a number of months and means..." In both cases you already know that the messages are formatted with JSON.
- nerdponx 5y ago
  There are 3 layers here: 1. Records separated by ASCII "\n" characters, wherein each record cannot contain an "\n" character 2. Each record is UTF-8 text in JSON format 3. The schema of each record is <something> ND-JSON takes care of the first two. You would still have to use JSONSchema for the third. This is not very different from using JSON without the "ND" part. Except: 1. Now there's the "ND" part, and I have something standardized I can use for quick and easy understanding/agreement when collaborating, especially outside my organization. 2. Plain JSON supports exactly 1 record per stream or file. ND-JSON supports any natural number of records, or an infinite number of records. That's useful to me. I really don't understand the resistance here. Nobody is trotting this out claiming its some hot new thing. It's an unoffensive standardization of a pretty basic data format that people have been using for 10+ years. If you have no use for it, then just ignore it. Consider that the people using it might not be morons; they might just have different needs that are different from yours.
  - BugsJustFindMe 5y ago
    > 2. Plain JSON supports exactly 1 record per stream or file. Unless you wrap it in an array, which is all this is doing.