4 ms·
FWIW, the soft limit for 70 units is about using the global module, which provides consistent state in the cluster (i.e. when you do a change, it tries to chang
by josevalim 6y ago
FWIW, the soft limit for 70 units is about using the global module, which provides consistent state in the cluster (i.e. when you do a change, it tries to change all nodes at once).
The default Erlang Distribution was shown to scale up to ~300 nodes. After that, using sub-clusters is the way to go and relatively easy to setup: it is a matter of setting the "-connect_all false" flag and calling Node.connect/2 based on the data being fed by a service discovery tool (etcd, k8s, aws config, etc).
PS: this data came from a paper. I am struggling to find it right now but I will edit this once/if I do.
- strmpnk 6y agoI can also mirror this general guideline. I've run 250+ node erlang clusters just fine in the past. There were some caveats to how some built-in OTP libraries behaved but they were easy to replace or workaround in the past. That was many years ago as well. The distributed erlang story has improved with more recent releases (better performance on remote monitors for example) which might push the number a little higher than 300 if you are careful. Keep in mind the default style of clustering is fully connected so there is some danger in managing that many network connections (quadratically scaling for each node added) during network partitions which can be a problem if you're not tuning things like TCP's TIME_WAIT for local networking conditions. Even better, these days there are great libraries like partisan (https://github.com/lasp-lang/partisan https://github.com/lasp-lang/partisan) which can scale to much larger cluster sizes and can be dropped in for most distributed erlang uses cases w/o much effort.