4 ms·
> This can lead to scalability issues in large clusters, as the number of connections that each node needs to maintain grows quadratically with the number of no
by toast0 2y ago
> This can lead to scalability issues in large clusters, as the number of connections that each node needs to maintain grows quadratically with the number of nodes in the cluster.
No, the total number of dist connections grows quadratically with the number of nodes, but the number of dist connections each node makes grows linearally.
> Not only that, in order to keep the cluster connected, each node periodically sends heartbeat messages to every other node in the cluster.
IIRC, heat beats are once every 30 seconds by default.
> This can lead to a lot of network traffic in large clusters, which can put a strain on the network.
Lets say I'm right about 30 seconds between heart beats, and you've got 1000 nodes. Every 30 seconds each node sends out 999 heartbeats (which almost certainly fit in a single tcp packet each, maybe less if they're piggybacking on real data exchanges). That's 999,000 packets every second, or 33k pps across your whole cluster. For reference, GigE line rate with full 1500 mtu packets is 80k pps. If you actually have 1000 nodes worth of work, the heartbeats are not at all a big deal.
> Historically, a "large" cluster in Erlang was considered to be around 50-100 nodes. This may have changed in recent years, but it's still something to be aware of when designing distributed Erlang systems.
I don't have recent numbers, but Rick Reed's presentation at Erlang Factory in 2014 shows a dist cluster with 400 nodes. I'm pretty sure I saw 1000+ node clusters too. I left WhatsApp in 2019, and any public presentations from WA are less about raw scale, because it's passe.
Really, 1000 dist connections is nothing when you're managing 500k client connections. Dist connections weren't even a big deal when we went to smaller nodes in FB.
It's good to have a solid backend network, and to try to bias towards fewer larger nodes, rather than more smaller nodes. If you want to play with large scale dist, so you spin up 1000 low cpu, low memory VMs, you might have some trouble. It makes sense to start with small nodes and whatever number makes you comfortable for availability, and then when you run into limits, reach for bigger nodes until you get to the point where adding nodes is more cost effective: WA ran dual xeon 2690 servers before the move to FB infra; facebook had better economics with smaller single Xeon D nodes; I dunno what makes sense today, maybe a single socket Epyc?
- sausagefeet 2y ago> That's 999,000 packets every second, or 33k pps across your whole cluster. For reference, GigE line rate with full 1500 mtu packets is 80k pps. If you actually have 1000 nodes worth of work, the heartbeats are not at all a big deal. Using up almost half of your pps every 30 seconds for cluster maintenance certainly seems like it's more than "not a big deal", no?
- fwip 2y agoWouldn't you have a lot more than one gigabit connection? I'm struggling to imagine the 1000-node cluster layout that sends all intra-cluster traffic through a single cable.
- deleted 2y ago[deleted]
- desdenova 2y agoIf you're at the point where you decided the 1000 nodes are required, you should probably have already considered the sensible alternatives first, and concluded this was somehow better. The network saturation is just a necessary cost of running such a massive cluster. I really have no idea what kind of system would require 1000 nodes, that couldn't be replaced by 100, 10x larger, nodes instead. And at that point, you should probably be thinking of ways to scale the network itself as well.
- sausagefeet 2y agoI don't really understand what this comment is responding to. The comment I responded to hand waved away consuming almost 50% of your pps on heartbeats every 30 seconds as "no big deal".
- ricketycricket 2y ago> The comment I responded to hand waved away consuming almost 50% of your pps on heartbeats every 30 seconds as "no big deal". > The network saturation is just a necessary cost of running such a massive cluster. I think this actually answers it perfectly. 1. If you are running 1K distributed nodes, you have to understand that means you have some overhead for running such a large cluster. No one is hand waving this away, it's just being acknowledged that this level of complexity has a cost. 2. If heartbeats are almost 50% of your pps, you are trying to use 1Gbe to run a 1K-node cluster. No one would do this in production and no one is claiming you should. 3. If your system can tolerate it, change the heartbeat interval to whatever you want. 4. Don't use distributed Erlang if you don't have to. Erlang/Elixir/Gleam work perfectly fine for non-distributed workloads as do most languages that can't distribute in the first place. But if you do need a distributed system, you are unlikely to find a better way to do it than the BEAM. Basically, it seems you are taking issue with something that 1) is that way because that's how things work, and 2) is not how anyone would actually use it.
- vereis 2y agoahoy!! thanks for spotting the issue I wrote the post in a stream of consciousness after a long day! I'll make that edit and call it out! the statement about what historically constitutes a large erlang cluster was an anecdote told to by Francesco Cesarini during lunch a few years ago — I'm not actually sure of the time frame (or my memory!) likewise I'll update the post to reflect that! thanks ( ◜ ‿ ◝ ) ♡
- deleted 2y ago[deleted]
- znpy 2y ago> WA ran dual xeon 2690 servers before the move to FB infra; facebook had better economics with smaller single Xeon D nodes; I dunno what makes sense today, maybe a single socket Epyc? that cpu is 8c/16t [1] ... having two of them in a system would mean 16c/32t. AMD recently launched a 192c/384t monster cpu [2], and having two of them would mean having 384c/768t. So most likely a 100 nodes-clusters today, powered by those amd cpus would give you the same core count of a 1200-node cluster at the time. references: [1]: https://www.intel.com/content/www/us/en/products/sku/64596/intel-xeon-processor-e52690-20m-cache-2-90-ghz-8-00-gts-intel-qpi/specifications.html https://www.intel.com/content/www/us/en/products/sku/64596/i... [2]: https://www.youtube.com/watch?v=S-NbCPEgP1A https://www.youtube.com/watch?v=S-NbCPEgP1A