4 ms·

At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) w

by jhgg 4y ago

At Discord, we maintain long lived web-sockets, and need to be able to deploy our real-time elixir based services (which runs on the BEAM VM just like Erlang) without any disruption to user traffic. To do so, we've built an application specific process migration system (this is BEAM VM processes, not OS processes) that is able to essentially live migrate processes both between nodes and software versions, allowing us to essentially replace the engine of the train while it's running, and also be able to scale up and down our clusters as needed. We are able to roll out updates across a hundred million processes on our distributed system in half an hour with no disruption to service. These are for internal services. For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated.

bmitc 4y ago
Damn, that's cool. Does Discord have any blog posts talking about this?
- abrookewood 4y ago
  Just seconding this - would love to hear more about Discord's use of Elixir
  - worthless-trash 4y ago
    And thirding, I'd like to subscribe to this newsletter.
- masklinn 4y ago
  I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code. Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.
  - toast0 4y ago
    The bit about migrating to other nodes doesn't flow from the builtin code loading functionality. Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun. [1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
    - jhgg 4y ago
      This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable. We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.
slekker 4y ago
This is so cool! I often wondered how possible would it be to keep a TCP connection alive during a server restart. How is the connection severed and the client not being aware of it works?
- pyinstallwoes 4y ago
  Probably a layer of indirection. One layer acting as the mailbox for sessions which then communicates with whatever services handles the connection to the user client. Effectively builds up a buffer until the service tree is restarted. Pub/sub abstractions basically. I could be wrong.
- ecdavis 4y ago
  Depends what you mean by server. Some old game server processes would support a "copyover" function. They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket. Unsophisticated, but it works. It's completely transparent to the client because the connection is never actually closed. The worst the client might experience is some latency as their input gets buffered by the OS while the server process restarts. Of course the connection would not persist over a physical server restart (or something like a pod being killed or whatever). I imagine it's possible to move a connection between nodes, though, with a fairly similar process.
  - rramadass 4y ago
    >They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket Nope. When a Process exits for whatever reason, the OS closes and releases all resources connected with all File/Socket descriptors for that Process. You will have to architect your System to explicitly accommodate this scenario. Some techniques here : https://stackoverflow.com/questions/55006657/can-i-allow-my-server-process-to-restart-without-killing-existing-connections https://stackoverflow.com/questions/55006657/can-i-allow-my-...
rramadass 4y ago
>For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated. Is this based on Alex Snoeren's TCP Connection Migration draft ? - https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migrate-00 https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migr...
- jhgg 4y ago
  No, it's entirely in user-space. Our websocket protocol can just negotiate a resumption. https://discord.com/developers/docs/topics/gateway#resuming https://discord.com/developers/docs/topics/gateway#resuming
theptip 4y ago
Great context, thanks. Do you have a ballpark feel for the cost of just terminating connections? Is it mainly a tail latency thing where you might try to push a message down a socket that is currently reconnecting, and have to wait way longer than normal? Or a CPU/network overhead thing due to how many open connections you have, with a full restart causing a thundering herd of activity?

by jhgg 4y ago

bmitc 4y ago

Damn, that's cool. Does Discord have any blog posts talking about this?

abrookewood 4y ago
Just seconding this - would love to hear more about Discord's use of Elixir
- worthless-trash 4y ago
  And thirding, I'd like to subscribe to this newsletter.
masklinn 4y ago
I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code. Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.
- toast0 4y ago
  The bit about migrating to other nodes doesn't flow from the builtin code loading functionality. Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun. [1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
  - jhgg 4y ago
    This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable. We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.

slekker 4y ago

This is so cool! I often wondered how possible would it be to keep a TCP connection alive during a server restart. How is the connection severed and the client not being aware of it works?

pyinstallwoes 4y ago
Probably a layer of indirection. One layer acting as the mailbox for sessions which then communicates with whatever services handles the connection to the user client. Effectively builds up a buffer until the service tree is restarted. Pub/sub abstractions basically. I could be wrong.
ecdavis 4y ago
Depends what you mean by server. Some old game server processes would support a "copyover" function. They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket. Unsophisticated, but it works. It's completely transparent to the client because the connection is never actually closed. The worst the client might experience is some latency as their input gets buffered by the OS while the server process restarts. Of course the connection would not persist over a physical server restart (or something like a pod being killed or whatever). I imagine it's possible to move a connection between nodes, though, with a fairly similar process.
- rramadass 4y ago
  >They would write the file descriptors of connected sockets to a file, restart the process, then read the list of file descriptors back into memory and resume normal use of the socket Nope. When a Process exits for whatever reason, the OS closes and releases all resources connected with all File/Socket descriptors for that Process. You will have to architect your System to explicitly accommodate this scenario. Some techniques here : https://stackoverflow.com/questions/55006657/can-i-allow-my-server-process-to-restart-without-killing-existing-connections https://stackoverflow.com/questions/55006657/can-i-allow-my-...

rramadass 4y ago

>For external services, we built the concept of "session resumption" which allows us to sever the TCP connection, have it accepted by a new/different host, and have it continue as if the connection wasn't ever terminated. Is this based on Alex Snoeren's TCP Connection Migration draft ? - https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migrate-00 https://datatracker.ietf.org/doc/html/draft-snoeren-tcp-migr...

jhgg 4y ago
No, it's entirely in user-space. Our websocket protocol can just negotiate a resumption. https://discord.com/developers/docs/topics/gateway#resuming https://discord.com/developers/docs/topics/gateway#resuming

theptip 4y ago

Great context, thanks. Do you have a ballpark feel for the cost of just terminating connections? Is it mainly a tail latency thing where you might try to push a message down a socket that is currently reconnecting, and have to wait way longer than normal? Or a CPU/network overhead thing due to how many open connections you have, with a full restart causing a thundering herd of activity?