3 ms·

Not mentioned is a common pitfall with serving web sockets: it is very easy to end up in a situation where you may not recover from disruptions. The problem is

Not mentioned is a common pitfall with serving web sockets: it is very easy to end up in a situation where you may not recover from disruptions. The problem is that establishing a web socket, especially when using SPDY or HTTPS (which is typical), is much more expensive than maintaining a connected one. So a service will typically see a steady creep up in the number of connections, with a relatively low number of new connections per second, and it's probably been tuned for massive levels of concurrency, and that's all manageable. But then the service has a blip of some kind all of a sudden all of those web socket clients try to connect at the same time; disaster. At that point the service is incredibly overloaded, so many connections fail, which results in retries, which makes matters even worse still and fuels a vicious cycle. That situation might self-stabilize if browsers implemented exponential back off, but that's not typical either. So the service remains hosed until some kind of blood letting can be done to let clients back in at a manageable rate. Extremely pernicious, and something I've seen several times. When scaling web-sockets, it's important to ask: can I handle all of these clients reconnecting at the same time?

clarkmoody 11y ago
> browsers implemented exponential back off Browsers do not have any native reconnect behavior for WebSockets. You have to implement reconnect in your application, which usually means exponential falloff. There are a few open-source implementations of this very feature. Since you control the reconnect behavior, you could have clients wait a random amount of time from 1-5s before attempting the first connection. That way your reconnect storm is spread uniformly over a broader time frame.
CyberDildonics 11y ago
Sounds like web programming might end up becoming closer to real programming.
Can_Not 11y ago
Is there a formal term for this websocket reconnect stampede? What serverside stack and language were you using? What clientside language and libraries? How many users were active and how many CPU cores were dedicated to the websockets? Were you load balancing with or without sticky sessions? Was your websockets a public broadcast or did you do authentication checks for rooms/channels? Have you considered doing a formal writeup over this pitfal and how it was resolved for your project?

clarkmoody 11y ago

> browsers implemented exponential back off Browsers do not have any native reconnect behavior for WebSockets. You have to implement reconnect in your application, which usually means exponential falloff. There are a few open-source implementations of this very feature. Since you control the reconnect behavior, you could have clients wait a random amount of time from 1-5s before attempting the first connection. That way your reconnect storm is spread uniformly over a broader time frame.

CyberDildonics 11y ago

Sounds like web programming might end up becoming closer to real programming.

Can_Not 11y ago

Is there a formal term for this websocket reconnect stampede? What serverside stack and language were you using? What clientside language and libraries? How many users were active and how many CPU cores were dedicated to the websockets? Were you load balancing with or without sticky sessions? Was your websockets a public broadcast or did you do authentication checks for rooms/channels? Have you considered doing a formal writeup over this pitfal and how it was resolved for your project?