3 ms·

Damn, that's cool. Does Discord have any blog posts talking about this?

by bmitc 4y ago

abrookewood 4y ago
Just seconding this - would love to hear more about Discord's use of Elixir
- worthless-trash 4y ago
  And thirding, I'd like to subscribe to this newsletter.
masklinn 4y ago
I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code. Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.
- toast0 4y ago
  The bit about migrating to other nodes doesn't flow from the builtin code loading functionality. Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun. [1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
  - jhgg 4y ago
    This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable. We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.

Damn, that's cool. Does Discord have any blog posts talking about this?

by bmitc 4y ago

Damn, that's cool. Does Discord have any blog posts talking about this?

abrookewood 4y ago

Just seconding this - would love to hear more about Discord's use of Elixir

worthless-trash 4y ago
And thirding, I'd like to subscribe to this newsletter.

masklinn 4y ago

I don’t know if discord uses that (guess not) but it’s partially built into beam: beam can have multiple versions of the same code loaded, and old code will only call new code when using “external” calls (module:function()), as long as you keep calling bare function() you live in the old code. Now obviously when calling utilities that can be an issue, but internally to an actor if you’re careful it can let you process something the recurse into the next version of the actor.

toast0 4y ago
The bit about migrating to other nodes doesn't flow from the builtin code loading functionality. Depending on how things are connected, there's lots of ways to do that, you've basically got to start the new process on the new node, send the state, forward messages, update senders, and finally kill the old process. Depending on your ordering requirements, forwarding messages and updating senders can get tricky. When sending messages from process A -> B, OTP dist guarantees the messages that arrive will arrive in the order sent, but if you send one message from A-> C and one from A -> B -> C, C could get those messages in either order. There's a lot of potential fun here; especially if you get into the real weeds --- I've had dist connections that got bandwidth limited and developed hours of latency[1]; if you needed to guarantee ordering through process migration and that's in the path, that's not going to be fun. [1] You might think the dist tick timers would prevent that, but it's not a ping timer, it's a tick timer. The default? is 30 second intervals on ticks, and if you don't receive a tick in 4 intervals, the connection is marked dead. As long as you're getting some throughput, and you don't queue up too much data between sending ticks, the other side will still get ticks frequently enough to stay connected. There's no check that the tick was generated anytime recently; and I'm not sure if there's even a timestamp in there, and anyway the systems need not have a synchronized clock. Thankfully, most of this happened in a context where high latency dist connections was annoying and weird, but not fatal; eventually the network bottleneck was fixed and the backlog cleared.
- jhgg 4y ago
  This is the simplified version of how we do process migration. Of course the devil's in the details, but yes, the way the BEAM process model works makes it super amenable. We have some bits to negotiate the state handoff between different versions of the code, and it's not as straightforward as copying over state. Certain data structures need to be rebuilt (for example, data structures owned by our Rust NIFs), and additionally, monitors and references across the system need to be updated and re-established.