3 ms·

> Let's skip over "low-latency", because virtually all programming languages and VMs are "low-latency". Perhaps under minimal load. However, the BEAM VM's styl

> Let's skip over "low-latency", because virtually all programming languages and VMs are "low-latency". Perhaps under minimal load. However, the BEAM VM's style of micro preemption allows it to retain really low latency under load, much better than traditional VM's on average. > "distributed" and "fault-tolerant" are what's most often touted as "special". The distributed piece is that theres not much difference between communicating to a local actor or a remote one. Also you get a lot of built in tools so you don't need a message bus, a cache, and a task manager, etc. Its convenient if you can stay below ~100 nodes. Fault tolerance is just how the supervisors encourage structuring the application. Java and C++ land tends to blur together things and it's difficult to disentangle the errors. Also, errors recovery tends to focus less on just "restarting" and more on trying to recover the current state.

toast0 3y ago
> Its convenient if you can stay below ~100 nodes. There's not much of a node limit. When I was at WhatsApp (through 2019) we ran thousands of nodes in some of our dist clusters. Unmodified pg2 was challenging at that scale (and modified pg2 was still sometimes challenging), but the new pg module doesn't rely on global:set_lock (or any form of global locking), and eliminates that scaling hurdle. Mnesia would likely get unmanagable with many nodes in a shared schema, but I'm not really sure why one would want to have that many nodes in a shared schema (with persistent nodes, we'd usually run 4 nodes in a mnesia schema cluster, with ocassional additional nodes added to facilitate resplitting data across more schema clusters. with ephemeral nodes, I believe it was typically 6 nodes). I was always confused when other groups suggested a node limit like 100 nodes. I just don't know where it comes from, and it doesn't match my experience. Hard limits come from the atom table and port limits, both of which are pretty easy to expand. Certainly if we could manage on the order of millions of connections to clients (that specific number dropped over time as the server did more work per connection and then when switching to the smaller node types that were the path of least resistance in FB hosting), a couple thousand for dist wasn't going to impinge on port limits.
weatherlight 3y ago
https://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf https://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf

toast0 3y ago

> Its convenient if you can stay below ~100 nodes. There's not much of a node limit. When I was at WhatsApp (through 2019) we ran thousands of nodes in some of our dist clusters. Unmodified pg2 was challenging at that scale (and modified pg2 was still sometimes challenging), but the new pg module doesn't rely on global:set_lock (or any form of global locking), and eliminates that scaling hurdle. Mnesia would likely get unmanagable with many nodes in a shared schema, but I'm not really sure why one would want to have that many nodes in a shared schema (with persistent nodes, we'd usually run 4 nodes in a mnesia schema cluster, with ocassional additional nodes added to facilitate resplitting data across more schema clusters. with ephemeral nodes, I believe it was typically 6 nodes). I was always confused when other groups suggested a node limit like 100 nodes. I just don't know where it comes from, and it doesn't match my experience. Hard limits come from the atom table and port limits, both of which are pretty easy to expand. Certainly if we could manage on the order of millions of connections to clients (that specific number dropped over time as the server did more work per connection and then when switching to the smaller node types that were the path of least resistance in FB hosting), a couple thousand for dist wasn't going to impinge on port limits.

weatherlight 3y ago

https://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf https://www.dcs.gla.ac.uk/~amirg/publications/DE-Bench.pdf