3 ms·

Senior dev but Elixir newbie. I'll never have projects the scale of Discord, but listening to talks and podcasts I have some scalability questions that I think

by ralphc 3y ago

Senior dev but Elixir newbie. I'll never have projects the scale of Discord, but listening to talks and podcasts I have some scalability questions that I think you could answer. Some podcasters have mentioned their experiences with slowdowns and out-of-memory issues due to copying large pieces of data from one process to another to work on it. Their solution was to keep the large chunk of data in one process and instead pass in the functions needed to work on the data. Have you had a situation where you did something like that, for example with a list of members, or did you always handle that type of situation with ETS? Some conventional wisdom I've seen is that you can get up to a cluster of about 40 nodes or so, then you start to have problems. Have you seen any issues with horizontal scaling at that scale or have you been able to handle everything by scaling up on your nodes? Can you think of any other languages and VM's that could handle your particular scale, or is the BEAM the only option?

yuliyp 3y ago
Yeah copying large amounts of data from one process to another (especially across nodes) is slow and expensive. We have to take care to not do that where possible. We've had issues from accidentally passing too much data to worker processes, but those are thankfully pretty rare because that is a pattern we try to avoid (having a single process per actor in the system is nicer for a variety of reasons). Regarding cluster size: We haven't noticed too many problems with it, but that's because our communication is usually not a full mesh, but rather more like bipartite graphs (guilds talk to sessions, but not to other guilds, for instance). Definitely other languages like C++, Rust, Go, or Java could solve similar problems, but we haven't had a reason to rewrite it all. Migrating only some types of actors in the systems to another language would mean not being able to use message passing / monitors and result in a lot of complexity because of it.

by ralphc 3y ago

yuliyp 3y ago

Yeah copying large amounts of data from one process to another (especially across nodes) is slow and expensive. We have to take care to not do that where possible. We've had issues from accidentally passing too much data to worker processes, but those are thankfully pretty rare because that is a pattern we try to avoid (having a single process per actor in the system is nicer for a variety of reasons). Regarding cluster size: We haven't noticed too many problems with it, but that's because our communication is usually not a full mesh, but rather more like bipartite graphs (guilds talk to sessions, but not to other guilds, for instance). Definitely other languages like C++, Rust, Go, or Java could solve similar problems, but we haven't had a reason to rewrite it all. Migrating only some types of actors in the systems to another language would mean not being able to use message passing / monitors and result in a lot of complexity because of it.