4 ms·

Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but i

Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why. Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.

icebraining 9y ago
Newlines can't appear in JSON strings (they must be encoded as \n).
- mikeash 9y ago
  But newlines can appear in JSON anywhere whitespace is allowed otherwise. Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?
  - rhacker 9y ago
    But newlines can't appear in JSON-Lines which is what this is really based off of.
  - jcrites 9y ago
    I’ve often used the jsonlines format for application logging, though I didn’t have a name for it until today. I appreciate this format because, as a log file, I can use all of my standard tools to work with it (grep awk etc.). Since it’s a text file, I can use standard text editors to view and search the contents. Since my application is the one that’s creating the log file, newlines will not appear or will be escaped. I can often accomplish my goal without parsing the objects, but when I need to, I can break out powerful tools like `jq` or RecordStream to query the contents in a structured way. I can also treat the log file as a streaming or archived data source and process it using database technologies like Redshift or Elasticsearch. I can’t think of another format that would be more convenient. I have worked with log file formats that are actual XML or JSON documents, and they are inconvenient in practice. I can’t think of another format that would be more convenient than this one for structured application logging.
  - IanCal 9y ago
    Lots of other tools work on a line by line basis.
beau 9y ago
I made this choice because so many tools support newline-delimited text. A quick search showed that others (Twitter, eBay) made the same choice, so I went with the flow. *Thank you for NSBlog, it's great!
- mikeash 9y ago
  You’re welcome! I think the tooling issue is for debugging and inspection, in which case you can just add a quick `tr` invocation to your pipeline to get it into a form they’ll understand.
zamalek 9y ago
> it would be nice if they’d explain why. Just to fill in the picture here: it's because the built-in parser has a very good chance of being faster than anything you could write[1]. In addition, it's code that you don't own and don't have to maintain. [1]: with exception to https://news.ycombinator.com/item?id=16413917 https://news.ycombinator.com/item?id=16413917
- mikeash 9y ago
  You’d use an existing streaming parser, not write your own.
  - aaronblohowiak 9y ago
    support for un-marshaling json into objects in a streaming way is spotty / inconvenient. it should be easier than it is.

icebraining 9y ago

Newlines can't appear in JSON strings (they must be encoded as \n).

mikeash 9y ago
But newlines can appear in JSON anywhere whitespace is allowed otherwise. Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?
- rhacker 9y ago
  But newlines can't appear in JSON-Lines which is what this is really based off of.
- jcrites 9y ago
  I’ve often used the jsonlines format for application logging, though I didn’t have a name for it until today. I appreciate this format because, as a log file, I can use all of my standard tools to work with it (grep awk etc.). Since it’s a text file, I can use standard text editors to view and search the contents. Since my application is the one that’s creating the log file, newlines will not appear or will be escaped. I can often accomplish my goal without parsing the objects, but when I need to, I can break out powerful tools like `jq` or RecordStream to query the contents in a structured way. I can also treat the log file as a streaming or archived data source and process it using database technologies like Redshift or Elasticsearch. I can’t think of another format that would be more convenient. I have worked with log file formats that are actual XML or JSON documents, and they are inconvenient in practice. I can’t think of another format that would be more convenient than this one for structured application logging.
- IanCal 9y ago
  Lots of other tools work on a line by line basis.

beau 9y ago

I made this choice because so many tools support newline-delimited text. A quick search showed that others (Twitter, eBay) made the same choice, so I went with the flow. *Thank you for NSBlog, it's great!

mikeash 9y ago
You’re welcome! I think the tooling issue is for debugging and inspection, in which case you can just add a quick `tr` invocation to your pipeline to get it into a form they’ll understand.

zamalek 9y ago

> it would be nice if they’d explain why. Just to fill in the picture here: it's because the built-in parser has a very good chance of being faster than anything you could write[1]. In addition, it's code that you don't own and don't have to maintain. [1]: with exception to https://news.ycombinator.com/item?id=16413917 https://news.ycombinator.com/item?id=16413917

mikeash 9y ago
You’d use an existing streaming parser, not write your own.
- aaronblohowiak 9y ago
  support for un-marshaling json into objects in a streaming way is spotty / inconvenient. it should be easier than it is.