8 ms·

Use streaming JSON to reduce latency on mobile

189 points by beau 9y ago

rhacker 9y ago
Somewhat related is the JSON Lines "spec": http://jsonlines.org/ http://jsonlines.org/
jannes 9y ago
Does anyone know a JSON parser that parses an ArrayBuffer instead of strings? [1] JSON.parse() only accepts strings. The library that the article recommends also uses XMLHttpRequest with strings. [2] The reason I'm asking is the maximum string length in 32-bit Chrome. [1]: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/responseType https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequ... [2]: https://github.com/eBay/jsonpipe/blob/master/lib/net/xhr.js https://github.com/eBay/jsonpipe/blob/master/lib/net/xhr.js
- matharmin 9y ago
  If you want to parse more than 4GB of JSON in a browser, ArrayBuffer versus string isn't the most important criteria - you shouldn't parse a 4GB ArrayBuffer as a single chunk either. You can always convert between an ArrayBuffer and a string, but you'll want to compare parsers according to the ability to stream, as well as performance and memory efficiency.
- DvdGiessen 9y ago
  You could look into streaming parsers such as Oboe.js, which specifically support the use case of parsing JSON tree's larger than the available RAM[0]. Then again, when you're loading such huge JSON files into a 32-bit instance of Chrome, it is likely you should look for a totally different solution to your problem. [0]: http://oboejs.com/examples#loading-json-trees-larger-than-the-available-ram http://oboejs.com/examples#loading-json-trees-larger-than-th...
pkulak 9y ago
I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed? Why invent a new protocol when JSON already exists? Streaming JSON parsers aren't unicorns, they are horses (sorry). With this new-line-delimited JSON format all your clients HAVE to know about your new protocol. They have to stream the response bytes, split on new lines, unescape new lines in the payload (how are we doing that, btw?), etc. If a client doesn't care about streaming, it can't just sit on the response and parse it when it's done coming in. Or, how about if later on you upgrade the system so that the response is instant and streaming is no longer necessary? Then you move on to a new API and have to keep supporting this old streaming-but-not-really endpoint forever.
- mikeash 9y ago
  Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why. Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.
  - icebraining 9y ago
    Newlines can't appear in JSON strings (they must be encoded as \n).
    - mikeash 9y ago
      But newlines can appear in JSON anywhere whitespace is allowed otherwise. Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?
      - rhacker 9y ago
        But newlines can't appear in JSON-Lines which is what this is really based off of.
tuukkah 9y ago
I'd like to see this streaming JSON parser incorporated into GraphQL clients: http://oboejs.com http://oboejs.com
erikrothoff 9y ago
Really interesting stuff! What about simply opening a Websocket connection and using that to for all requests if connection latency is such an issue?
gumby 9y ago
I feel like I'm missing something here. Isn't that the point of using a JSON SAX parser instead of a DOM parser?
slig 9y ago
I have nothing but praise to this service. It's fast and efficient, and does the job without bloat. For instance, their iOS app weighs 888.8 KB! When it's common for simple apps to be 50 MB monsters, it's very refreshing to use something that has been developed with proper care.
- beau 9y ago
  Thanks!
zeger 9y ago
You say that using websockets was less reliable than streaming HTTPS, can you elaborate why? In my experience websockets are perfect to use for the use case you described, are there disadvantages?
- greenleafjacob 9y ago
  Websockets are also bidirectional.
- beau 9y ago
  I tried this a long time ago. At the time, certain queries were hard for a server to answer. Other clients connected to the zombie server stalled until something timed out. With HTTPS, a load balancer can direct new queries to servers that are responsive.
  - thinkloop 9y ago
    I was also curious about this (and thanks for the article, very informative!) Are you saying that at one point the servers would crash relatively often, which would leave sockets clients hanging, unless some complicated client-side code was written - whereas without sockets, a load-balancer could automatically switch clients to functional servers, without extra coding, and mitigate the issue? Isn't the problem the crashy servers?
    - beau 9y ago
      Sure, but why keep state when you don't need to? Now that HTTP/2 is ubiquitous, sending new GET requests is no more expensive than sending bytes through a WebSocket.
MonkeyDan 9y ago
Any benefit to using this over Server-Sent Events? (other than IE/Edge support)
fenwick67 9y ago
I did this a few years ago before I knew it was a "thing" and felt really proud that it actually worked. The use-case was we had a slow database query for basically map pins. The first ones pins come back in milliseconds, but the last ones would take seconds. The UI was vastly improved by streaming the data instead of waiting for it all to finish, and the server code was easy to implement. A different delimiter would have worked, but newlines are easy to see in a debugger.
nategri 9y ago
Why not just gzip the JSON? Should make complicated JSON around an order of magnitude smaller, and be more portable to boot.
- beau 9y ago
  I do gzip the streamed JSON. Try: $ curl -H "Accept-Encoding: gzip" --trace - "https://instantdomainsearch.com/services/vanity/apple?hash=866016287" https://instantdomainsearch.com/services/vanity/apple?hash=8...
Figs 9y ago
As someone who actually regularly uses a slow mobile connection (8 kilobytes per second!) with somewhat high ping (~90ms to Google), please don't do this thinking you're making my life significantly better. It barely makes a difference in performance once loaded, and the initial load time is ridiculously worse. I'd much rather you make your page work without JavaScript, kept the design light (as this page has otherwise done), and make your CSS cacheable. Right now, it takes over 5 seconds(!!) for this page to load because of all the freaking JavaScript it has to download! With JS off, the page loads almost immediately. With a keep-alive connection, subsequent loads over HTTPS are not particularly long, unlike what this article seems to think. (Hacker News is one of the FASTEST sites I can access, for example. Even on my crappy connection, pages load nearly instantly.) Simply letting me type, press enter, and wait 0.1~0.3 seconds for a new page response would not be a significantly worse experience -- however, due to the way the site is written, search doesn't work AT ALL with JS disabled. So, lots of engineering effort (compared to just serving up a new page) for little to no actual speed improvement, and a more brittle website that breaks completely on unusual configurations... Yeah. Please don't do this!
- ricardobeat 9y ago
  You wouldn’t wait 0.3s for the next page - that’d be 0.3s + a few seconds waiting for all the queries to return before showing anything. Streaming let’s you show results earlier. Most likely the page loads instantly because the server is not doing any real work, which is offloaded to the client/js.
  - Figs 9y ago
    I'm looking at the behavior of the search tool on the page, which makes a request to https://instantdomainsearch.com/services/all/<query https://instantdomainsearch.com/services/all/<query parameters> through AJAX and gets streamed lines of JSON back as a result. It's not doing DNS queries from JS or whatever. Looking at the behavior with curl and wireshark, what I see is that a full, new connection to the service does spend most of its time in DNS lookup and HTTPS handshake. It takes about 0.1~0.3s for the actual data to transfer. What the article is recommending is basically, don't make a request per JSON object. (It streamed back 69 objects for the query I tried.) Using one connection to transfer the information saves a lot of overhead -- and I don't have a problem with that part of the advice. What I mean is, instead of using JS at all to do this (and consequently triggering a 5 second initial page load, etc etc.), have the server build the page the traditional way, and send that -- that's still one connection for that data transfer, and with a light page design and keep alive connection, the page load time does not seem like it would be significantly different here (most of the time is going to be in that 0.1~0.3s for the query to execute regardless) -- but the initial page load time would be significantly faster on slow connections. If your queries do actually take many seconds, sure, maybe there would be a benefit there, but I'm not seeing the value on a page like this, and I really don't want people to take away the idea here that they should redesign their sites to use AJAX to "reduce latency on mobile" by default as it won't help, and in fact, tends to make things worse.
iamd3vil 9y ago
I think websockets is a much better use case for this if you don't want the reconnection overhead. Also since websockets are bidirectional, you can keep the connection open and send all requests through the connection as well as receive responses from the connection. Also you can send binary on websockets if you want to save bandwidth as well. We do this at work and it works pretty nicely.
wybiral 9y ago
Is there an advantage to using chunked streams like this over using WebSockets like this: https://github.com/wybiral/hookah https://github.com/wybiral/hookah (this will take newlines from a programs stdout and send them over WebSockets, even aggregating multiple streams into one).
bshacklett 9y ago
How is this handled from a UI perspective? As more applications are built around the idea of streaming data, I've found that UI elements tend to jump around, and I find myself clicking/tapping the wrong item more and more, because the item which had been under my thumb/cursor has jumped away just before I could activate it.
- beau 9y ago
  This is a good point. De-bounce, static results ordering, and static elements with placeholders help. Instant Domain Search has room to improve here.
osrec 9y ago
Why would you use this over web sockets?
Osiris 9y ago
Isn't this pretty similar to how you would use WebSocket frames to transfer individual JSON elements when a client is subscribed? At one job I had several years ago we came up with the same idea and use \n separated JSON elements as a streaming response. We also tossed around the idea of using WebSockets to stream large responses between services.
JensRantil 9y ago
Let's talk about reliability: The network is unrealiable; Firewalls might be broken, packets are dropped, IP-addresses may change, cellphones lose connection in subway tunnels. Simply calling "streaming reliable" without even defining what the "reliability" is protecting again, makes "reliable" an overstatement. IMHO the most reliable way to get data from point A to point B is likely by having a client actively polling for data, using a strict socket timeout. Data should be at-least once delivered. If JSONS should be called anything remotely "reliable" as periodically polling, at least it should have a strict timeout (not mentioned in the article) for receiving the next newline & it should handle replaying of non-acked messages. Otherwise I would call it far from "reliable".
- beau 9y ago
  Each request takes a second or two to complete. If you lose the connection, the client can send the same request again (with exponential back off).
stringham 9y ago
Should have (2017) in the title.
delaaxe 9y ago
Feels to me like the response content type shouldn't be "application/json" anymore (it is what's returned on that first example).
- beau 9y ago
  IIRC, earlier versions of WebKit wouldn't emit data from each chunk when I sent a custom content type. Or maybe there was some conflict with gzip, I forget. Now that those browsers are gone, maybe ndjson? jsons?
sajal83 9y ago
If the concern is HTTPS overhead, why not use HTTP/2 and send multiple requests? I think streaming would be useful only if the responses are stateful and it's hard to share it across requests.
- beau 9y ago
  Even with HTTP/2, sending a GET request is not free. Most of the benefit is that we can show the user results as they come in. Each query gets over 50 responses in random order from DNS, in-memory indexes of zone files, slower fuzzy searches over other data, and so on. Why wait to show them the .com result while .ca resolves?
cwt137 9y ago
How is this different than SSE https://html.spec.whatwg.org/multipage/comms.html#server-sent-events https://html.spec.whatwg.org/multipage/comms.html#server-sen... ? Or, why would someone choose this over SSE?

189 points by beau 9y ago

Somewhat related is the JSON Lines "spec": http://jsonlines.org/ http://jsonlines.org/

Does anyone know a JSON parser that parses an ArrayBuffer instead of strings? [1] JSON.parse() only accepts strings. The library that the article recommends also uses XMLHttpRequest with strings. [2] The reason I'm asking is the maximum string length in 32-bit Chrome. [1]: https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/responseType https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequ... [2]: https://github.com/eBay/jsonpipe/blob/master/lib/net/xhr.js https://github.com/eBay/jsonpipe/blob/master/lib/net/xhr.js

matharmin 9y ago
If you want to parse more than 4GB of JSON in a browser, ArrayBuffer versus string isn't the most important criteria - you shouldn't parse a 4GB ArrayBuffer as a single chunk either. You can always convert between an ArrayBuffer and a string, but you'll want to compare parsers according to the ability to stream, as well as performance and memory efficiency.
DvdGiessen 9y ago
You could look into streaming parsers such as Oboe.js, which specifically support the use case of parsing JSON tree's larger than the available RAM[0]. Then again, when you're loading such huge JSON files into a 32-bit instance of Chrome, it is likely you should look for a totally different solution to your problem. [0]: http://oboejs.com/examples#loading-json-trees-larger-than-the-available-ram http://oboejs.com/examples#loading-json-trees-larger-than-th...

pkulak 9y ago

I agree that a streaming response is cool, but why the dismissal of returning valid JSON, streamed? Why invent a new protocol when JSON already exists? Streaming JSON parsers aren't unicorns, they are horses (sorry). With this new-line-delimited JSON format all your clients HAVE to know about your new protocol. They have to stream the response bytes, split on new lines, unescape new lines in the payload (how are we doing that, btw?), etc. If a client doesn't care about streaming, it can't just sit on the response and parse it when it's done coming in. Or, how about if later on you upgrade the system so that the response is instant and streaming is no longer necessary? Then you move on to a new API and have to keep supporting this old streaming-but-not-really endpoint forever.

mikeash 9y ago
Yeah, that’s really weird. They mention streaming parsers, then immediately say “ignore them.” Why? They apparently think this solution is better somehow, but it would be nice if they’d explain why. Also, if you’re going to reinvent the wheel and make a custom framing format, why would you choose a delimeter that can legally appear in your content? Separating your JSON with newlines is complete madness. If you’re sending UTF-8 then you can trivially use a byte that cannot appear in the data, like 0xff, as the divider.
- icebraining 9y ago
  Newlines can't appear in JSON strings (they must be encoded as \n).
  - mikeash 9y ago
    But newlines can appear in JSON anywhere whitespace is allowed otherwise. Yes, you can say “we will avoid this by never generating JSON with newlines.” But why would you do that when you can easily pick a delimeter that’s guaranteed by the spec not to appear?
    - rhacker 9y ago
      But newlines can't appear in JSON-Lines which is what this is really based off of.

tuukkah 9y ago

I'd like to see this streaming JSON parser incorporated into GraphQL clients: http://oboejs.com http://oboejs.com

erikrothoff 9y ago

Really interesting stuff! What about simply opening a Websocket connection and using that to for all requests if connection latency is such an issue?

gumby 9y ago

I feel like I'm missing something here. Isn't that the point of using a JSON SAX parser instead of a DOM parser?

slig 9y ago

I have nothing but praise to this service. It's fast and efficient, and does the job without bloat. For instance, their iOS app weighs 888.8 KB! When it's common for simple apps to be 50 MB monsters, it's very refreshing to use something that has been developed with proper care.

beau 9y ago
Thanks!

zeger 9y ago

You say that using websockets was less reliable than streaming HTTPS, can you elaborate why? In my experience websockets are perfect to use for the use case you described, are there disadvantages?

greenleafjacob 9y ago
Websockets are also bidirectional.
beau 9y ago
I tried this a long time ago. At the time, certain queries were hard for a server to answer. Other clients connected to the zombie server stalled until something timed out. With HTTPS, a load balancer can direct new queries to servers that are responsive.
- thinkloop 9y ago
  I was also curious about this (and thanks for the article, very informative!) Are you saying that at one point the servers would crash relatively often, which would leave sockets clients hanging, unless some complicated client-side code was written - whereas without sockets, a load-balancer could automatically switch clients to functional servers, without extra coding, and mitigate the issue? Isn't the problem the crashy servers?
  - beau 9y ago
    Sure, but why keep state when you don't need to? Now that HTTP/2 is ubiquitous, sending new GET requests is no more expensive than sending bytes through a WebSocket.

MonkeyDan 9y ago

Any benefit to using this over Server-Sent Events? (other than IE/Edge support)

fenwick67 9y ago

I did this a few years ago before I knew it was a "thing" and felt really proud that it actually worked. The use-case was we had a slow database query for basically map pins. The first ones pins come back in milliseconds, but the last ones would take seconds. The UI was vastly improved by streaming the data instead of waiting for it all to finish, and the server code was easy to implement. A different delimiter would have worked, but newlines are easy to see in a debugger.

nategri 9y ago

Why not just gzip the JSON? Should make complicated JSON around an order of magnitude smaller, and be more portable to boot.

beau 9y ago
I do gzip the streamed JSON. Try: $ curl -H "Accept-Encoding: gzip" --trace - "https://instantdomainsearch.com/services/vanity/apple?hash=866016287" https://instantdomainsearch.com/services/vanity/apple?hash=8...

Figs 9y ago

As someone who actually regularly uses a slow mobile connection (8 kilobytes per second!) with somewhat high ping (~90ms to Google), please don't do this thinking you're making my life significantly better. It barely makes a difference in performance once loaded, and the initial load time is ridiculously worse. I'd much rather you make your page work without JavaScript, kept the design light (as this page has otherwise done), and make your CSS cacheable. Right now, it takes over 5 seconds(!!) for this page to load because of all the freaking JavaScript it has to download! With JS off, the page loads almost immediately. With a keep-alive connection, subsequent loads over HTTPS are not particularly long, unlike what this article seems to think. (Hacker News is one of the FASTEST sites I can access, for example. Even on my crappy connection, pages load nearly instantly.) Simply letting me type, press enter, and wait 0.1~0.3 seconds for a new page response would not be a significantly worse experience -- however, due to the way the site is written, search doesn't work AT ALL with JS disabled. So, lots of engineering effort (compared to just serving up a new page) for little to no actual speed improvement, and a more brittle website that breaks completely on unusual configurations... Yeah. Please don't do this!

ricardobeat 9y ago
You wouldn’t wait 0.3s for the next page - that’d be 0.3s + a few seconds waiting for all the queries to return before showing anything. Streaming let’s you show results earlier. Most likely the page loads instantly because the server is not doing any real work, which is offloaded to the client/js.
- Figs 9y ago
  I'm looking at the behavior of the search tool on the page, which makes a request to https://instantdomainsearch.com/services/all/<query https://instantdomainsearch.com/services/all/<query parameters> through AJAX and gets streamed lines of JSON back as a result. It's not doing DNS queries from JS or whatever. Looking at the behavior with curl and wireshark, what I see is that a full, new connection to the service does spend most of its time in DNS lookup and HTTPS handshake. It takes about 0.1~0.3s for the actual data to transfer. What the article is recommending is basically, don't make a request per JSON object. (It streamed back 69 objects for the query I tried.) Using one connection to transfer the information saves a lot of overhead -- and I don't have a problem with that part of the advice. What I mean is, instead of using JS at all to do this (and consequently triggering a 5 second initial page load, etc etc.), have the server build the page the traditional way, and send that -- that's still one connection for that data transfer, and with a light page design and keep alive connection, the page load time does not seem like it would be significantly different here (most of the time is going to be in that 0.1~0.3s for the query to execute regardless) -- but the initial page load time would be significantly faster on slow connections. If your queries do actually take many seconds, sure, maybe there would be a benefit there, but I'm not seeing the value on a page like this, and I really don't want people to take away the idea here that they should redesign their sites to use AJAX to "reduce latency on mobile" by default as it won't help, and in fact, tends to make things worse.

iamd3vil 9y ago

I think websockets is a much better use case for this if you don't want the reconnection overhead. Also since websockets are bidirectional, you can keep the connection open and send all requests through the connection as well as receive responses from the connection. Also you can send binary on websockets if you want to save bandwidth as well. We do this at work and it works pretty nicely.

wybiral 9y ago

Is there an advantage to using chunked streams like this over using WebSockets like this: https://github.com/wybiral/hookah https://github.com/wybiral/hookah (this will take newlines from a programs stdout and send them over WebSockets, even aggregating multiple streams into one).

bshacklett 9y ago

How is this handled from a UI perspective? As more applications are built around the idea of streaming data, I've found that UI elements tend to jump around, and I find myself clicking/tapping the wrong item more and more, because the item which had been under my thumb/cursor has jumped away just before I could activate it.

beau 9y ago
This is a good point. De-bounce, static results ordering, and static elements with placeholders help. Instant Domain Search has room to improve here.

osrec 9y ago

Why would you use this over web sockets?

Osiris 9y ago

Isn't this pretty similar to how you would use WebSocket frames to transfer individual JSON elements when a client is subscribed? At one job I had several years ago we came up with the same idea and use \n separated JSON elements as a streaming response. We also tossed around the idea of using WebSockets to stream large responses between services.

JensRantil 9y ago

Let's talk about reliability: The network is unrealiable; Firewalls might be broken, packets are dropped, IP-addresses may change, cellphones lose connection in subway tunnels. Simply calling "streaming reliable" without even defining what the "reliability" is protecting again, makes "reliable" an overstatement. IMHO the most reliable way to get data from point A to point B is likely by having a client actively polling for data, using a strict socket timeout. Data should be at-least once delivered. If JSONS should be called anything remotely "reliable" as periodically polling, at least it should have a strict timeout (not mentioned in the article) for receiving the next newline & it should handle replaying of non-acked messages. Otherwise I would call it far from "reliable".

beau 9y ago
Each request takes a second or two to complete. If you lose the connection, the client can send the same request again (with exponential back off).

stringham 9y ago

Should have (2017) in the title.

delaaxe 9y ago

Feels to me like the response content type shouldn't be "application/json" anymore (it is what's returned on that first example).

beau 9y ago
IIRC, earlier versions of WebKit wouldn't emit data from each chunk when I sent a custom content type. Or maybe there was some conflict with gzip, I forget. Now that those browsers are gone, maybe ndjson? jsons?

sajal83 9y ago

If the concern is HTTPS overhead, why not use HTTP/2 and send multiple requests? I think streaming would be useful only if the responses are stateful and it's hard to share it across requests.

beau 9y ago
Even with HTTP/2, sending a GET request is not free. Most of the benefit is that we can show the user results as they come in. Each query gets over 50 responses in random order from DNS, in-memory indexes of zone files, slower fuzzy searches over other data, and so on. Why wait to show them the .com result while .ca resolves?

cwt137 9y ago

How is this different than SSE https://html.spec.whatwg.org/multipage/comms.html#server-sent-events https://html.spec.whatwg.org/multipage/comms.html#server-sen... ? Or, why would someone choose this over SSE?