3 ms·

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially

by jhgg 10y ago

Neat. We've been using CF websockets for some time now on the Enterprise plan. If you're intending to use CF websockets, be prepared for random (and potentially massive) connection drops, and be sure you're architected to handle these disconnects gracefully. Cloudflare rolling restarts have caused hundreds of thousands of websocket connections to be disconnected in a matter of minutes for us. If you plan to operate at that scale, make sure you're able to tolerate the thundering herd of reconnecting clients as cloudflare disconnects them.

ddorian43 10y ago
Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?
- ianlevesque 10y ago
  It's not ideal for sure but you can use DNS to divvy it up by adding several IPs to your domain name.
  - jmtulloss 10y ago
    You can also do it out of band and have an API call that returns the server you're supposed to be connected to.
- al_chemist 10y ago
  If you use Elixir/Phoenix, you can have many servers running same code, connected with each other as Erlang nodes. When client wants to broadcast something in whole network, it is broadcasted locally but is also sent do all servers and servers broadcasts it to their local clients. It's very easy to write.
pencoyd 10y ago
As you know, when terminating a WebSocket connection due to releases CloudFlare now signals this action to both client and origin server by sending the 1001 status code (aka "going away", see section 7.4.1 of RFC 6455), so both sides are aware that the WebSocket termination is only a transient event, and that they can expect to immediately re-establish a connection again on retry. We're working on additional refinements to the release process to minimize disruptions. (I work at CloudFlare.)
- jhgg 10y ago
  Aye! Well aware. I actually think you added that for us specifically! Anyways, it's worth noting for anyone interested that they'll need to be able to handle spikey reconnects. In our case, we keep a buffer of the last N websocket messages sent to the client, and when the client reconnects, it sends the last sequence id that the client saw, and the server catches it up. https://discordapp.com/developers/docs/topics/gateway#resuming https://discordapp.com/developers/docs/topics/gateway#resumi...

by jhgg 10y ago

ddorian43 10y ago

Ugh, how do you shard websocket connections when you have more connection than 1 server can handle ?

ianlevesque 10y ago
It's not ideal for sure but you can use DNS to divvy it up by adding several IPs to your domain name.
- jmtulloss 10y ago
  You can also do it out of band and have an API call that returns the server you're supposed to be connected to.
al_chemist 10y ago
If you use Elixir/Phoenix, you can have many servers running same code, connected with each other as Erlang nodes. When client wants to broadcast something in whole network, it is broadcasted locally but is also sent do all servers and servers broadcasts it to their local clients. It's very easy to write.

pencoyd 10y ago

As you know, when terminating a WebSocket connection due to releases CloudFlare now signals this action to both client and origin server by sending the 1001 status code (aka "going away", see section 7.4.1 of RFC 6455), so both sides are aware that the WebSocket termination is only a transient event, and that they can expect to immediately re-establish a connection again on retry. We're working on additional refinements to the release process to minimize disruptions. (I work at CloudFlare.)

jhgg 10y ago
Aye! Well aware. I actually think you added that for us specifically! Anyways, it's worth noting for anyone interested that they'll need to be able to handle spikey reconnects. In our case, we keep a buffer of the last N websocket messages sent to the client, and when the client reconnects, it sends the last sequence id that the client saw, and the server catches it up. https://discordapp.com/developers/docs/topics/gateway#resuming https://discordapp.com/developers/docs/topics/gateway#resumi...