4 ms·

It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally s

It looks each line in the stream represents a JSON statement (only hashes?) in an array, even without being enclosed in an array and comma delimited. Formally specifying the syntax alternative is worthwhile.

BugsJustFindMe 5y ago
> It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.
- jmull 5y ago
  The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
- wolfgang42 5y ago
  The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.
  - BugsJustFindMe 5y ago
    > you do still need to deal with the situation where the character shows up in your data JSON doesn't support binary data without conversion to string first.
    - wolfgang42 5y ago
      I thought that you said that JSON was incidental and unimportant? My point is that using RS does still force escaping, in the general case; it’s not a panacea. You still need to figure out how to deal with your delimiter character appearing in the data. The way JSON Lines does this is by specifying the use of (a slightly limited subset of) JSON.
      - BugsJustFindMe 5y ago
        > I thought that you said that JSON was incidental and unimportant? It is for this, but if we're already declaring a restriction to JSON then we're likewise declaring a restriction to unicode. All delimited data requires escaping if your delimiter can be part of a no-delimit sequence. At least a character explicitly intended for delimiting isn't part of any no-delimit sequences representable in unicode whereas \n is.
        wolfgang42 5y ago
        > we're likewise declaring a restriction to unicode. U+1E is a Unicode code point, and therefore requires escaping inside of a ‘no-delimit’ sequence of Unicode characters.

BugsJustFindMe 5y ago

> It looks each line in the stream represents a JSON statement Only incidentally. There's literally nothing about the format, and I hesitate to give it the implied gravitas by calling it a format, that makes the JSON part important in any way. The lines are just individual messages. JSON happens to fit, but so do many other syntaxes. What's an array? It's an abstract sequence of items. What's a delimited stream? It's a sequence of items. Do the lines need to be JSON? Who the hell cares. If you're using JSON then they're JSON. If you aren't, then they aren't. Honestly, the most important part of their description is "for passing messages between cooperating processes", because JSON is terrible for blindly consuming messages without cooperative explanation from the sender (http://seriot.ch/parsing_json.php http://seriot.ch/parsing_json.php). Given that you can't actually do anything with JSON data (or most data, JSON is again not extremely special in this regard) without a concrete explanation from the sender of the expected interpretation for every field, can you tell me what you think the important difference is between "json messages delimited by newlines" as proposed here with its own custom mimetype but without any other information and "newline delimited messages that can be parsed with a json parser"? They'd have been better off using the ASCII record separator character instead of newline. That's both already a standard and also doesn't force escaping to prevent collisions with normal text.

jmull 5y ago
The point of specifying JSON is so a client can parse the the stream of messages. This is as useful/useless as JSON itself.
wolfgang42 5y ago
The record separator ship sailed long ago. The advantage of newline is that it’s easy to use with a bunch of standard tools like less/head/tail/your text editor. Besides, you do still need to deal with the situation where the character shows up in your data; using an uncommon codepoint just makes it much easier to ignore until it blows up in your face.
- BugsJustFindMe 5y ago
  > you do still need to deal with the situation where the character shows up in your data JSON doesn't support binary data without conversion to string first.
  - wolfgang42 5y ago
    I thought that you said that JSON was incidental and unimportant? My point is that using RS does still force escaping, in the general case; it’s not a panacea. You still need to figure out how to deal with your delimiter character appearing in the data. The way JSON Lines does this is by specifying the use of (a slightly limited subset of) JSON.
    - BugsJustFindMe 5y ago
      > I thought that you said that JSON was incidental and unimportant? It is for this, but if we're already declaring a restriction to JSON then we're likewise declaring a restriction to unicode. All delimited data requires escaping if your delimiter can be part of a no-delimit sequence. At least a character explicitly intended for delimiting isn't part of any no-delimit sequences representable in unicode whereas \n is.
      - wolfgang42 5y ago
        > we're likewise declaring a restriction to unicode. U+1E is a Unicode code point, and therefore requires escaping inside of a ‘no-delimit’ sequence of Unicode characters.