4 ms·

When I worked with Erlang outside of Ericsson, I’ve heard the following from our tech lead: OTP built-in distributed process and failover mechanisms assume much

by sesm 3y ago

When I worked with Erlang outside of Ericsson, I’ve heard the following from our tech lead: OTP built-in distributed process and failover mechanisms assume much lower latencies than usually exist between datacenter VMs and the underlying protocols consume a lot of bandwidth. I didn’t dig deep enough to verify this, but I can confirm that typical Ericsson hardware is several boards connected with high bandwidth ‘backplane’ and OTP failover was designed for this environment. VMs communicating through LAN is a different environment, and VMs living in different data centers are very far from this. I wonder if anyone else tried using OTP outside of Ericsson and what is their experience.

throwawaymaths 3y ago
You should probably reach for something like ra for anything but trivial stuff. If latency is not a concern, global module is probably fine but double check the quorum membership.
- rhodin 3y ago
  ra for those interested: https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra ("A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.") Used by RabbitMQ for quorum queues + Khepri (the new metadata store)
toast0 3y ago
Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we saw with pg2. It was obviously not ideal, or normal, but sometimes our WAN connections would get bandwidth limited due to congestion or other issues, and things would generally keep going without unexpected problems: replication got behind when the change stream was more than available bandwidth and there's no separation of dist traffic, so if replication has minutes or hours of delay, so does any other traffic in that direction, so interactive traffic will timeout, net_adm ping will take minutes or hours, etc. I guess technically the connection ticks must not quite suffer the same queuing, because the connections would not be severed by tick timeouts. The other issue related to WAN that we'd see is when you start a mnesia node, it loads all of the tables from peers if peers have copies of the tables, but out of the box it doesn't have a concept of LAN vs WAN peers, so it may load your half a terrabyte or whatever worth of data from WAN instead of LAN. Loading all that data from LAN might be undesirable too, but that's another story; you could probably maybe come up with a way to store and later replicate changelogs for offline mnesia peers, but that doesn't come out of the box, and WA didn't build it either. The community at large often reported maximum dist cluster sizes that were way less than what we ran in production, and comments about WAN distribution that didn't match our experience either. shrug It does help to have a very solid private network between nodes though. Dist in general, but mnesia specifically, don't like for nodes to disappear. pg2 has some quirks too, but pg resolves them, afaik, as I mentioned earlier. Some of pg2's issues were rooted in global locks, you've got to be careful about those if node lists are rapidly changing and there's contention on the locks.
- josevalim 3y ago
  I believe the reason why we have seen so many different numbers given as limits for distribution is because they are often measuring different things. As you know, the limits will vary depending if it is just distribution or if global/mnesia are used. I have seen distribution by itself going north of 300 nodes (and heard from others going quite beyond that) without issues and heard reports of global usage struggling past 40-60 nodes (especially global transactions). At least in the Elixir ecosystem, global is rarely used and I personally suggest looking for alternatives if you are going past 20 nodes. Unfortunately I have no practical experience with mnesia.
  - toast0 3y ago
    I think there's ways to get global to work, but it gets really hairy if there's much lock contention, if the node lists are changing frequently / there's a partition, or some nodes are processing slowly. At least in the versions we were running, there's likely been some improvements. IIRC, we had issues with pg2 which does global locks when we started many nodes in large clusters that all wanted to join processes to the same pg2 group. For reasons, pg2 wanted the process list per pg2 group to be consistently ordered on all nodes, although I'm not convinced that really worked 100%... Anyway, it would try to do a global lock, but if you've got about 1000 processes across as many nodes all trying to do a global lock and they also haven't all found each other quite yet.. it just doesn't make progress. Especially if some of the nodes are half a continent away. As I understand it, the new pg avoids global locks and group membership does not have a consistent ordering --- instead the local pg process just broadcasts joins and leaves from local processes and sets monitors to adjust when group members are killed or become unreachable. Global has some techniques to make progress, but they didn't work for our case, I think because the node lists didn't converge in a reasonable amount of time, or maybe pg2 was using locking similar to global locking, but not exactly. Anyway, we did adjust the locks to make sure they would progess first, but later avoided them, which is better. Mnesia with a large number of nodes sharing a schema would probably have issues too, we tended to run data services sharded over groups of 4 servers sharing a schema... the whole group of storage servers would be in a single dist cluster, but there's no need for all the mnesia servers in a system to share the same schema; and we'd send application level reads and writes to gen_servers on the mnesia nodes to do the actual mnesia reads and writes. Lots of benefits from making a clear boundary between uses of the data and the storage system.
hosh 3y ago
Dist Erlang worked fine in today’s environments for the application we were working on — nothing like real time audio. (Although people are using Elixir for real-time audio and video processing and muxing). I saw an example of something OTP is also good at — coordination across unreliable nodes and networks. IoT is obvious … but so are mobile point-of-sale and payment systems on food trucks.
keep_reading 3y ago
They epmd protocol is designed to be replaceable and that's what Partisan is for https://partisan.dev/ https://partisan.dev/

by sesm 3y ago

throwawaymaths 3y ago

You should probably reach for something like ra for anything but trivial stuff. If latency is not a concern, global module is probably fine but double check the quorum membership.

rhodin 3y ago
ra for those interested: https://github.com/rabbitmq/ra https://github.com/rabbitmq/ra ("A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.") Used by RabbitMQ for quorum queues + Khepri (the new metadata store)

toast0 3y ago

Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we saw with pg2. It was obviously not ideal, or normal, but sometimes our WAN connections would get bandwidth limited due to congestion or other issues, and things would generally keep going without unexpected problems: replication got behind when the change stream was more than available bandwidth and there's no separation of dist traffic, so if replication has minutes or hours of delay, so does any other traffic in that direction, so interactive traffic will timeout, net_adm ping will take minutes or hours, etc. I guess technically the connection ticks must not quite suffer the same queuing, because the connections would not be severed by tick timeouts. The other issue related to WAN that we'd see is when you start a mnesia node, it loads all of the tables from peers if peers have copies of the tables, but out of the box it doesn't have a concept of LAN vs WAN peers, so it may load your half a terrabyte or whatever worth of data from WAN instead of LAN. Loading all that data from LAN might be undesirable too, but that's another story; you could probably maybe come up with a way to store and later replicate changelogs for offline mnesia peers, but that doesn't come out of the box, and WA didn't build it either. The community at large often reported maximum dist cluster sizes that were way less than what we ran in production, and comments about WAN distribution that didn't match our experience either. shrug It does help to have a very solid private network between nodes though. Dist in general, but mnesia specifically, don't like for nodes to disappear. pg2 has some quirks too, but pg resolves them, afaik, as I mentioned earlier. Some of pg2's issues were rooted in global locks, you've got to be careful about those if node lists are rapidly changing and there's contention on the locks.

josevalim 3y ago
I believe the reason why we have seen so many different numbers given as limits for distribution is because they are often measuring different things. As you know, the limits will vary depending if it is just distribution or if global/mnesia are used. I have seen distribution by itself going north of 300 nodes (and heard from others going quite beyond that) without issues and heard reports of global usage struggling past 40-60 nodes (especially global transactions). At least in the Elixir ecosystem, global is rarely used and I personally suggest looking for alternatives if you are going past 20 nodes. Unfortunately I have no practical experience with mnesia.
- toast0 3y ago
  I think there's ways to get global to work, but it gets really hairy if there's much lock contention, if the node lists are changing frequently / there's a partition, or some nodes are processing slowly. At least in the versions we were running, there's likely been some improvements. IIRC, we had issues with pg2 which does global locks when we started many nodes in large clusters that all wanted to join processes to the same pg2 group. For reasons, pg2 wanted the process list per pg2 group to be consistently ordered on all nodes, although I'm not convinced that really worked 100%... Anyway, it would try to do a global lock, but if you've got about 1000 processes across as many nodes all trying to do a global lock and they also haven't all found each other quite yet.. it just doesn't make progress. Especially if some of the nodes are half a continent away. As I understand it, the new pg avoids global locks and group membership does not have a consistent ordering --- instead the local pg process just broadcasts joins and leaves from local processes and sets monitors to adjust when group members are killed or become unreachable. Global has some techniques to make progress, but they didn't work for our case, I think because the node lists didn't converge in a reasonable amount of time, or maybe pg2 was using locking similar to global locking, but not exactly. Anyway, we did adjust the locks to make sure they would progess first, but later avoided them, which is better. Mnesia with a large number of nodes sharing a schema would probably have issues too, we tended to run data services sharded over groups of 4 servers sharing a schema... the whole group of storage servers would be in a single dist cluster, but there's no need for all the mnesia servers in a system to share the same schema; and we'd send application level reads and writes to gen_servers on the mnesia nodes to do the actual mnesia reads and writes. Lots of benefits from making a clear boundary between uses of the data and the storage system.

hosh 3y ago

Dist Erlang worked fine in today’s environments for the application we were working on — nothing like real time audio. (Although people are using Elixir for real-time audio and video processing and muxing). I saw an example of something OTP is also good at — coordination across unreliable nodes and networks. IoT is obvious … but so are mobile point-of-sale and payment systems on food trucks.

keep_reading 3y ago

They epmd protocol is designed to be replaceable and that's what Partisan is for https://partisan.dev/ https://partisan.dev/