4 ms·
Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we
by toast0 3y ago
Welp, my browser ate my longer response. But in short, when I was at WhatsApp, dist worked fine over WANs, I think the WA contributed pg addresses the issues we saw with pg2. It was obviously not ideal, or normal, but sometimes our WAN connections would get bandwidth limited due to congestion or other issues, and things would generally keep going without unexpected problems: replication got behind when the change stream was more than available bandwidth and there's no separation of dist traffic, so if replication has minutes or hours of delay, so does any other traffic in that direction, so interactive traffic will timeout, net_adm ping will take minutes or hours, etc. I guess technically the connection ticks must not quite suffer the same queuing, because the connections would not be severed by tick timeouts.
The other issue related to WAN that we'd see is when you start a mnesia node, it loads all of the tables from peers if peers have copies of the tables, but out of the box it doesn't have a concept of LAN vs WAN peers, so it may load your half a terrabyte or whatever worth of data from WAN instead of LAN. Loading all that data from LAN might be undesirable too, but that's another story; you could probably maybe come up with a way to store and later replicate changelogs for offline mnesia peers, but that doesn't come out of the box, and WA didn't build it either.
The community at large often reported maximum dist cluster sizes that were way less than what we ran in production, and comments about WAN distribution that didn't match our experience either. shrug It does help to have a very solid private network between nodes though. Dist in general, but mnesia specifically, don't like for nodes to disappear. pg2 has some quirks too, but pg resolves them, afaik, as I mentioned earlier. Some of pg2's issues were rooted in global locks, you've got to be careful about those if node lists are rapidly changing and there's contention on the locks.
- josevalim 3y agoI believe the reason why we have seen so many different numbers given as limits for distribution is because they are often measuring different things. As you know, the limits will vary depending if it is just distribution or if global/mnesia are used. I have seen distribution by itself going north of 300 nodes (and heard from others going quite beyond that) without issues and heard reports of global usage struggling past 40-60 nodes (especially global transactions). At least in the Elixir ecosystem, global is rarely used and I personally suggest looking for alternatives if you are going past 20 nodes. Unfortunately I have no practical experience with mnesia.
- toast0 3y agoI think there's ways to get global to work, but it gets really hairy if there's much lock contention, if the node lists are changing frequently / there's a partition, or some nodes are processing slowly. At least in the versions we were running, there's likely been some improvements. IIRC, we had issues with pg2 which does global locks when we started many nodes in large clusters that all wanted to join processes to the same pg2 group. For reasons, pg2 wanted the process list per pg2 group to be consistently ordered on all nodes, although I'm not convinced that really worked 100%... Anyway, it would try to do a global lock, but if you've got about 1000 processes across as many nodes all trying to do a global lock and they also haven't all found each other quite yet.. it just doesn't make progress. Especially if some of the nodes are half a continent away. As I understand it, the new pg avoids global locks and group membership does not have a consistent ordering --- instead the local pg process just broadcasts joins and leaves from local processes and sets monitors to adjust when group members are killed or become unreachable. Global has some techniques to make progress, but they didn't work for our case, I think because the node lists didn't converge in a reasonable amount of time, or maybe pg2 was using locking similar to global locking, but not exactly. Anyway, we did adjust the locks to make sure they would progess first, but later avoided them, which is better. Mnesia with a large number of nodes sharing a schema would probably have issues too, we tended to run data services sharded over groups of 4 servers sharing a schema... the whole group of storage servers would be in a single dist cluster, but there's no need for all the mnesia servers in a system to share the same schema; and we'd send application level reads and writes to gen_servers on the mnesia nodes to do the actual mnesia reads and writes. Lots of benefits from making a clear boundary between uses of the data and the storage system.