7 ms·

I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed? Why invent a new protocol when JSON already exists? Streamin

• mikeash 9y ago
  Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why. Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.
  • icebraining 9y ago
    Newlines can't appear in JSON strings (they must be encoded as \n).
    • mikeash 9y ago
      But newlines can appear in JSON anywhere whitespace is allowed otherwise. Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?
      • rhacker 9y ago
        But newlines can't appear in JSON-Lines which is what this is really based off of.
      • jcrites 9y ago
        I’ve often used the jsonlines format for application logging, though I didn’t have a name for it until today. I appreciate this format because, as a log file, I can use all of my standard tools to work with it (grep awk etc.). Since it’s a text file, I can use standard text editors to view and search the contents. Since my application is the one that’s creating the log file, newlines will not appear or will be escaped. I can often accomplish my goal without parsing the objects, but when I need to, I can break out powerful tools like `jq` or RecordStream to query the contents in a structured way. I can also treat the log file as a streaming or archived data source and process it using database technologies like Redshift or Elasticsearch. I can’t think of another format that would be more convenient. I have worked with log file formats that are actual XML or JSON documents, and they are inconvenient in practice. I can’t think of another format that would be more convenient than this one for structured application logging.
      • IanCal 9y ago
        Lots of other tools work on a line by line basis.
  • beau 9y ago
    I made this choice because so many tools support newline-delimited text. A quick search showed that others (Twitter, eBay) made the same choice, so I went with the flow. *Thank you for NSBlog, it's great!
    • mikeash 9y ago
      You’re welcome! I think the tooling issue is for debugging and inspection, in which case you can just add a quick `tr` invocation to your pipeline to get it into a form they’ll understand.
  • zamalek 9y ago
    > it would be nice if they’d explain why. Just to fill in the picture here: it's because the built-in parser has a very good chance of being faster than anything you could write[1]. In addition, it's code that you don't own and don't have to maintain. [1]: with exception to https://news.ycombinator.com/item?id=16413917 https://news.ycombinator.com/item?id=16413917
    • mikeash 9y ago
      You’d use an existing streaming parser, not write your own.
      • aaronblohowiak 9y ago
        support for un-marshaling json into objects in a streaming way is spotty / inconvenient. it should be easier than it is.
• macspoofing 9y ago
  >I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed? JSON hasn't been designed for chunked interpretation. How would the client know when to start interpreting the received message? How could you tell the difference between a valid chunk or malformed response? >With this new-line-delimited JSON format all your clients HAVE to know about your new protocol Yes. It's probably not a good option for public. It should be a complement to a more general API. I would put this in the bucket of micro-optimization for very specific use-cases. Regardless, if you publish your API and document it, other developers should be able to consume it just fine. It's not rocket science.
  • beau 9y ago
    It works for Twitter: https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data https://developer.twitter.com/en/docs/tutorials/consuming-st...
    • platz 9y ago
      > carriage-return (\r\n) delimited JSON-endocded activities That's not JSON
      • Spivak 9y ago
        Why would you say that? > Insignificant whitespace is allowed before or after any of the six structural characters. Where whitespace may be space, carriage return, line feed, or horizontal tab. It's explicitly valid JSON. https://tools.ietf.org/html/rfc7159#section-2 https://tools.ietf.org/html/rfc7159#section-2
        • platz 9y ago
          ok - but my point is that it's a protocol, that is more specific than json i.e. streaming JSON in full generality has problems. if you want to restrict yourself to a subset and create a protocol, thats great - but it's not really "streaming JSON"
  • 9y ago
• tobyhinloopen 9y ago
  Embedding newlines..... hmmmmm \nI’m not sure how one would do that
• andrewprock 9y ago
  Streaming JSON is troublesome because the base object is unbounded. It's true that there are workhorses that handle this, but when debugging responses, ad hoc tooling is very useful. If you can't examine a response without using a streaming parser, the cost of maintenance goes up significantly. It's important to remember why we use JSON. It's not because it's well suited to transmission. It's because it's easy to reason about. If you want to move to a streaming format that is not easy to reason about, you may as well move to a binary format.
  • beau 9y ago
    curl is a streaming parser. Most tools know what to do with a newline.
    • andrewprock 9y ago
      I haven't checked, but does Chrome preview tab handle this non-standard format? That's the level where a great deal of JSON debugging occurs.
• beau 9y ago
  It is a pragmatic choice. Most languages can consume newline-delimited text natively. Do you really think getline() is going away?
  • euyyn 9y ago
    His point isn't that consuming data line by line is hard.
• greglindahl 9y ago
  I've been using jsonlines for logfiles from about 5 years before it acquired a name. With line delimited logfile objects, it's easy to grep for a string of interest and then only parse the lines that match -- much more efficient than parsing an entire logfile to pick out 0.01% of the lines. That's not the usecase talked about in this article, but it is a usecase that's important to me.
  • beau 9y ago
    It's also easy to load them into Google's BigQuery: https://cloud.google.com/bigquery/docs/loading-data#supported_data_formats https://cloud.google.com/bigquery/docs/loading-data#supporte...
    • stingraycharles 9y ago
      I was just about to name that as an example. Newline separated JSON is a so much simpler solution to the problem than streaming JSON parsers. They're easier to understand, and it's trivial to implement. From a purity point of view, sure, streaming JSON is more "correct", but I wouldn't call it the better solution.
  • beached_whale 9y ago
    it should be trivial to write a tool to stream a json file with proper newlines
    • greglindahl 9y ago
      Not sure what you're getting at -- my tools that generate json logfiles do trivially add newlines where I want them. jq's "compact" mode will reformat files that don't have newlines where I want them.
      • beached_whale 9y ago
        I mean that the tools to add the newlines to a compact json file with minimal whitespace are not going to be the bottle neck. I am fairly sure that grep uses a forward streaming access pattern and not random access so inserting a relatively simple command into the pipeline to insert newlines, like jq, probably isn't going to slow it down. The tool itself wouldn't be difficult to write, or like you said, just use jq
• skybrian 9y ago
  This makes more sense when working with files (rather than an API). You can use traditional Unix commands. Another reason might be that when working with hand-written files, the delimiter adds a bit of redundancy. This make errors like unbalanced parentheses easier to diagnose.
• gg3 9y ago
  I agree. Keeping this backward compatible would not only simplify client design but also accelerate the adoption. I am not really sure how this would be implemented, though.
• ricardobeat 9y ago
  JS (and JSON by extension) does not have multi-line strings, so line endings are already escaped. All you need to do is output ‘non-pretty’ json.
  • pkulak 9y ago
    Nice catch, thanks! I didn't realize that. Looks like no control characters are allowed in JSON string literals.
• makmanalp 9y ago
  Last time I tried streaming parsing it was with Oboe.js which wasn't bad but I remember it having approximately a 5x speed penalty. So you get your first item in your dataset faster, but the whole thing loads slower - a tradeoff that's not a no-brainer. I wonder how getline plus many repeated calls to json.parse work in comparison to that - I suspect the native parser probably has some startup overhead but still better than a JS parser. The other thing was that to get the full benefits of something like this (not necessarily about streaming parsers vs not) you had to rearrange the way your /whole/ stack works, streaming all the way from the database through all the backend layers to the frontend. It's satisfying when it works, but definitely a non-trivial amount of change.
• boubiyeah 9y ago
  It's impossible to stream a JSON Array
• jandrewrogers 9y ago
  Let me preface by saying that I don't have an immediate solution for this case, the details seem to be application specific. However, I will note that JSON is brutally inefficient by almost every computational metric that exists and it therefore is quite expensive unless your problem domain is trivial. I used to be in the position of spending -- literally -- millions of dollars per year on JSON because that was how people wanted to format the data. You can solve many engineering efficiency problems for millions of dollars/euros. Which is to say I don't have an answer for why streaming JSON isn't valid in this case, but I can also say that if it was up to me I would never us it in an application that mattered. It is much too expensive for many (most?) applications.