4 ms·

As an Erlang fan, who hasn't used it in production services, I'm wondering if you can let us know some specific scaling issues you encountered.

ilovecaching 8y ago
I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.
- toast0 8y ago
  Bursting is an interesting thought. I think, if you planned it out, it could be done -- subject to some constraints. It would be hard to burst stateful (mnesia) nodes --- schema operations require a lock across all the nodes in the schema, and that lock requires that the nodes not be in the middle of the 'log dumping' process (where the global transaction log gets divided into per table logs and such), which means long delays in high volume situations, and even longer delays if doing multiple schema operations. This could probably be patched around, but... In my team's experience, our mnesia nodes were generally ok under higher than normal load, expansion was driven by data size. Expansion could be a lot nicer, but I haven't heard of many database systems that handle expansion off the shelf. So that leaves stateless nodes. I don't see why you couldn't burst those, especially if using standard dist. Bring up the host, push your software, connect to one dist node, and get meshed automatically, once you see all the pg2 groups you need to operate, enable traffic. That said, we never did too much of that, we're in bare metal hosting so we don't have an incentive to run different server counts at different times of day, and provisioning isn't fast enough to handle incidental spikes -- we have a pretty good model of what spikes to expect, and provision to handle that load being mindful of the possibility of a load spike during a network or datacenter availability incident.
  - ilovecaching 8y ago
    Our nodes are stateful. We also wrote our own dist (I didn't write it, it could be a lot better). We also have good metrics on our spikes, we just have black friday where we need to burst like crazy.
    - toast0 8y ago
      Oh yeah, black Friday is crazy. For us, we've always been seeing that our annual big spike load ends up being our sustained daily peak in a few months, so we will need those nodes anyway.
RobertKerans 8y ago
Out-of-the-box, it's only really designed to scale to a certain point. It's all generally nice, predictable, and with with low latency up until that point. But because the nodes are fully meshed, the TCP heartbeats alone kill performance once you go past that point. So for ex. 100 nodes gives you 5050 TCP connections (100+99+98+97...etc). As parent says, there are methods to deal with this (Riak Core would be an example). But they're non-trivial. + you possibly need to bear in mind the design goals: that Erlang is designed to run as a highly reliable, self contained system in _a single geographic location_, with that system possibly left to run on its own for long periods of time (years)
- toast0 8y ago
  So, at one level, 5000 tcp connections is a lot, but at another level, some teams (including mine) are running hundreds of thousands of tcp connections to our clients from our front end Erlang nodes. I've never thought about the dist heartbeats as a scalaing problem. If you have thousands of dist nodes, and your nodes have small memory, dist buffers for each connection to add up -- I think the default is 8mb, you can tune it, but it's a scaling concern. Especially, if you have nodes far apart from each other. Really, the root design of Erlang was for two nodes colocated in a single chassis. That said, it turns out the design scales pretty well to much larger numbers of nodes, and nodes farther apart, but you have to be careful with some things. pg2:join and leave operate under a global lock, which will be slow if you have contention on the lock, or if one of your nodes has some problem where it's still up but very slow. Mnesia doesn't do well with queuing without a lot of help, schema operations under queuing is definitely a bad idea as well. If you want to run Erlang at larger scales, you will need to be ready to poke around in OTP, and ocassionaly in BEAM as well. If you're running big systems, IMHO it makes the most sense for your Erlang nodes to fill your physical nodes, so I don't see much need for containers, but if you do use containers, you need to figure out how to get their names consistent for Erlang, or it's going to be confused. (OTP has a concept of a 'diskless' node which would seem to be a good fit for an ephemeral systems environment, but I must admit I haven't played with that)
  - ilovecaching 8y ago

ilovecaching 8y ago

I can't really go into it in detail, but at a high level Erlang default methods of distribution and security don't scale very well. There are people working on better mechanisms for this, and I know of several companies that have custom solutions for clustering nodes. One big issue is you cannot easily burst Erlang nodes to handle peak traffic. The number of nodes is usually relatively static in a deployment.

toast0 8y ago
Bursting is an interesting thought. I think, if you planned it out, it could be done -- subject to some constraints. It would be hard to burst stateful (mnesia) nodes --- schema operations require a lock across all the nodes in the schema, and that lock requires that the nodes not be in the middle of the 'log dumping' process (where the global transaction log gets divided into per table logs and such), which means long delays in high volume situations, and even longer delays if doing multiple schema operations. This could probably be patched around, but... In my team's experience, our mnesia nodes were generally ok under higher than normal load, expansion was driven by data size. Expansion could be a lot nicer, but I haven't heard of many database systems that handle expansion off the shelf. So that leaves stateless nodes. I don't see why you couldn't burst those, especially if using standard dist. Bring up the host, push your software, connect to one dist node, and get meshed automatically, once you see all the pg2 groups you need to operate, enable traffic. That said, we never did too much of that, we're in bare metal hosting so we don't have an incentive to run different server counts at different times of day, and provisioning isn't fast enough to handle incidental spikes -- we have a pretty good model of what spikes to expect, and provision to handle that load being mindful of the possibility of a load spike during a network or datacenter availability incident.
- ilovecaching 8y ago
  Our nodes are stateful. We also wrote our own dist (I didn't write it, it could be a lot better). We also have good metrics on our spikes, we just have black friday where we need to burst like crazy.
  - toast0 8y ago
    Oh yeah, black Friday is crazy. For us, we've always been seeing that our annual big spike load ends up being our sustained daily peak in a few months, so we will need those nodes anyway.

RobertKerans 8y ago

Out-of-the-box, it's only really designed to scale to a certain point. It's all generally nice, predictable, and with with low latency up until that point. But because the nodes are fully meshed, the TCP heartbeats alone kill performance once you go past that point. So for ex. 100 nodes gives you 5050 TCP connections (100+99+98+97...etc). As parent says, there are methods to deal with this (Riak Core would be an example). But they're non-trivial. + you possibly need to bear in mind the design goals: that Erlang is designed to run as a highly reliable, self contained system in _a single geographic location_, with that system possibly left to run on its own for long periods of time (years)

toast0 8y ago
So, at one level, 5000 tcp connections is a lot, but at another level, some teams (including mine) are running hundreds of thousands of tcp connections to our clients from our front end Erlang nodes. I've never thought about the dist heartbeats as a scalaing problem. If you have thousands of dist nodes, and your nodes have small memory, dist buffers for each connection to add up -- I think the default is 8mb, you can tune it, but it's a scaling concern. Especially, if you have nodes far apart from each other. Really, the root design of Erlang was for two nodes colocated in a single chassis. That said, it turns out the design scales pretty well to much larger numbers of nodes, and nodes farther apart, but you have to be careful with some things. pg2:join and leave operate under a global lock, which will be slow if you have contention on the lock, or if one of your nodes has some problem where it's still up but very slow. Mnesia doesn't do well with queuing without a lot of help, schema operations under queuing is definitely a bad idea as well. If you want to run Erlang at larger scales, you will need to be ready to poke around in OTP, and ocassionaly in BEAM as well. If you're running big systems, IMHO it makes the most sense for your Erlang nodes to fill your physical nodes, so I don't see much need for containers, but if you do use containers, you need to figure out how to get their names consistent for Erlang, or it's going to be confused. (OTP has a concept of a 'diskless' node which would seem to be a good fit for an ephemeral systems environment, but I must admit I haven't played with that)
- ilovecaching 8y ago