3 ms·

"Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a

by jerf 7y ago

"Would it have been better if they went with Elixir?" No. It would have been unshippably bad. BEAM is generally fairly slow. It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. As a language, it is much slower than Rust. Plus, if you tried to implement a gigantic shared cache map in Erlang/Elixir, you'd have two major problems: One is that you'd need huge chunks of the map in single (BEAM) processes, and you'd get hit by the fact BEAM is not set up to GC well in that case. It wants lots of little processes, not a small number of processes holding tons of data. Second is that you'd be trading what in Rust is "accept some bytes, do some hashing, look some stuff up in memory" with generally efficient, low-copy operations, with "copy the network traffic into an Erlang binary, do some hashing, compute the PID that actually has the data, send a message to that PID with the request, wait for the reply message, and then send out the answer", with a whole lot of layers that expect to have time to make copies of lots of things. Adding this sort of coordination into these nominally fast lookups is going to slow this to a crawl. It's like when people try to benchmark Erlang/Elixir/Go's threading by creating processes/goroutines to receive two numbers and add them together "in parallel"; the IPC completely overshadows the tiny amount of work being done. (They mention tokio, but that's still going to add a lot less coordination overhead than Erlang messages.) Go is a significantly better language for this use case than Elixir/Erlang/BEAM is, let alone Rust. (This is not a "criticism" of Erlang/Elixir/BEAM. It's an engineering analysis. Erlang/Elixir/BEAM are still suitable for many tasks, just as people still use Python for many things despite the fact it would be a catastrophically bad choice for this particular task. This just isn't one of the tasks it would be suitable for.)

sergiotapia 7y ago
>It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. Such as?
- jerf 7y ago
  Go, Rust is apparently getting there since it just smoked Go on a pretty core multitasking-heavy task here, and a lot of the old "stodgy" languages like Java while none of the cool kids were looking have gotten pretty good at large numbers of threads too. I'm also expecting some sort of sensible solution to this sort of concurrency challenge to simply be baseline expected functionality for the next generation of languages. Anyone sitting down in 2020 to write The Next Big Language who ignores the fact that by the time they're done, their CPUs are going to be 64-core and the GPUs will be several-hundred-thousand core is really going to be missing the boat. I actually encourage Erlang partisans to consider this a win in the general sense. Quite serious. If you consider languages as a multi-decade conversation, almost everything Erlang "said" in the late 1990s and early 2000s is in fact proving out to be true. However, while Erlang was a trailblazer, it isn't going to be that Next Big Language, nor anything like it. It got a lot of things right, but it's got too much wrong to be the NBL, and even if you did the minimal fixes, the result wouldn't be backwards compatible with Erlang anymore so it'd be a new language. A lot of Erlang partisans are making the error of thinking the next language needs to be just like Erlang, except perhaps more so. But that's not how progress gets made. The good ideas are ripped out at a much more granular level and recombined with all sorts of other ideas and the end result may be quite different than what you expect. Go, for instance, can very much be seen as a direct sequel to Erlang among other things, even though it may not seem to have "OTP" or "built-in multinode concurrency" or whatever other apparent bullet-list features Erlang has, because a lot of those bullet-list features are really just epiphenomena of the real underlying features. Rust has its own nods to Erlang too; the whole "ownership" thing comes from the experiences with both mutation and immutability in a threading environment. But rather than making "Erlang, but a bit moreso", you get something that takes the lessons, ponders on them for another 15 years, and produces something different. It isn't Erlang, because at the time that Erlang was being written (Joe Armstrong RIP), nobody had the experience to think of Rust and imagine it could make a practical language. (Nor am I entirely sure the computers of the time could have compiled it; even if we sat through the clock time I'm not sure Rust could be stuffed into a 1997 desktop computer's RAM.)
blackrock 7y ago
Good points. > It wants lots of little processes, not a small number of processes holding tons of data Elixir/Erlang is good for handling a lot of little processes with a small amount of data. And not for a small number of processes, handling a large amount of data. The little processes holds smaller data, and it just gets dropped, after the function is done, instead of getting reclaimed by a garbage collector. This is probably what makes Elixir/Erlang good for telecom equipment, like packet switching hardware, but not good for more complex software applications that may need to fetch and manipulate a lot of structured data in multiple stages. In this case, does anyone know of Elixir’s maximum throughput?
hopia 7y ago
Not to disagree with your analysis of the performance implications, but I don't think having all that data under a single or a few processes would be the right architectural pattern to handle this in Elixir. The article says that the data is basically "per-user", indicating that the active client connection process could be used to store the data. It already hosts other data related to the client (connection) anyway. I think updating and querying it globally would be the trouble in that case. Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB. Probably better, but still not ideal to solve this. Anyway, you're right in that no matter how you'd try to solve this problem on pure Elixir you'd still be seeing some bottlenecks because BEAM just isn't very well suitable for this kind of problems, hence Rust. But can you elaborate on what you mean by other platforms catching up with Elixir's inherent concurrency advantages? Which modern platforms give similar features?
- jerf 7y ago
  "The article says that the data is basically "per-user"," Given that this is a table of who is "online", I don't think that's per-user in the sense that you are inferring. I infer that it's not a whole bunch of little local data that doesn't interact, it's a big global table of who is online and not online, constantly being heavily read from and written to in real time. Consider from the perspective of Bob's Erlang process that he wants to go offline and notify all of his currently-online friends that is is going offline. Bob's Erlang process doesn't have that data. Bob's Erlang process is going to get it from the Big Table of Who's Online. That table is the problem; it can't be stored in Bob's Erlang process. I was at least imagining that the table could be partitioned into pieces pretty trivially (first X bits of the hash), but with Erlang's design, that implies an IPC just to ask some server process to give me the PID of the chunk I need to talk to, which itself is going to bottleneck. (In practice we'd probably cheat and use a NIF to do that, but that amounts to an admission that Erlang can't do this, so....) At smaller scales you could try to live update Bob's local information as it changes, but this breaks down in all sorts of ways at scales far smaller than Discord, scales much closer to "a single mid-sized company". "Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB." I have used mnesia for loads literally a ten-thousandth as small as this, if that (I could probably tack two more zeros on there), and it breaks down. It is an absolutely ludicrous idea that mnesia could handle what Discord is doing here. Last I knew the official Erlang community consensus was basically that mnesia really shouldn't be used for anything serious; my experience backed that up. I think a non-trivial part of the reason why Erlang hasn't taken off is that its community still seems to exist in 2003, where it's a really incredible unique language that solves huge problems that nobody else does. In 2003, it rather has a point. But a lot of things have learned from Erlang, and incorporated its lessons into newer designs, and moved on. See my other comment for what other runtimes have Erlang's advantages, but I'd invite you just to consider what we seem to basically agree on here; Erlang would be wildly slower and require a lot more hardware than Rust, the Rust code probably wasn't that hard to write, ... and the Rust code is way more likely to be correct than the Erlang code, too. I mean, what more "catching up to Elixir's inherent concurrency advantages" in this context than "did a job Elixir couldn't possibly do" do you want?

by jerf 7y ago

sergiotapia 7y ago

>It was fast at multitasking for a while, but that advantage has been claimed by several other runtimes in 2020. Such as?

jerf 7y ago
Go, Rust is apparently getting there since it just smoked Go on a pretty core multitasking-heavy task here, and a lot of the old "stodgy" languages like Java while none of the cool kids were looking have gotten pretty good at large numbers of threads too. I'm also expecting some sort of sensible solution to this sort of concurrency challenge to simply be baseline expected functionality for the next generation of languages. Anyone sitting down in 2020 to write The Next Big Language who ignores the fact that by the time they're done, their CPUs are going to be 64-core and the GPUs will be several-hundred-thousand core is really going to be missing the boat. I actually encourage Erlang partisans to consider this a win in the general sense. Quite serious. If you consider languages as a multi-decade conversation, almost everything Erlang "said" in the late 1990s and early 2000s is in fact proving out to be true. However, while Erlang was a trailblazer, it isn't going to be that Next Big Language, nor anything like it. It got a lot of things right, but it's got too much wrong to be the NBL, and even if you did the minimal fixes, the result wouldn't be backwards compatible with Erlang anymore so it'd be a new language. A lot of Erlang partisans are making the error of thinking the next language needs to be just like Erlang, except perhaps more so. But that's not how progress gets made. The good ideas are ripped out at a much more granular level and recombined with all sorts of other ideas and the end result may be quite different than what you expect. Go, for instance, can very much be seen as a direct sequel to Erlang among other things, even though it may not seem to have "OTP" or "built-in multinode concurrency" or whatever other apparent bullet-list features Erlang has, because a lot of those bullet-list features are really just epiphenomena of the real underlying features. Rust has its own nods to Erlang too; the whole "ownership" thing comes from the experiences with both mutation and immutability in a threading environment. But rather than making "Erlang, but a bit moreso", you get something that takes the lessons, ponders on them for another 15 years, and produces something different. It isn't Erlang, because at the time that Erlang was being written (Joe Armstrong RIP), nobody had the experience to think of Rust and imagine it could make a practical language. (Nor am I entirely sure the computers of the time could have compiled it; even if we sat through the clock time I'm not sure Rust could be stuffed into a 1997 desktop computer's RAM.)

blackrock 7y ago

Good points. > It wants lots of little processes, not a small number of processes holding tons of data Elixir/Erlang is good for handling a lot of little processes with a small amount of data. And not for a small number of processes, handling a large amount of data. The little processes holds smaller data, and it just gets dropped, after the function is done, instead of getting reclaimed by a garbage collector. This is probably what makes Elixir/Erlang good for telecom equipment, like packet switching hardware, but not good for more complex software applications that may need to fetch and manipulate a lot of structured data in multiple stages. In this case, does anyone know of Elixir’s maximum throughput?

hopia 7y ago

Not to disagree with your analysis of the performance implications, but I don't think having all that data under a single or a few processes would be the right architectural pattern to handle this in Elixir. The article says that the data is basically "per-user", indicating that the active client connection process could be used to store the data. It already hosts other data related to the client (connection) anyway. I think updating and querying it globally would be the trouble in that case. Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB. Probably better, but still not ideal to solve this. Anyway, you're right in that no matter how you'd try to solve this problem on pure Elixir you'd still be seeing some bottlenecks because BEAM just isn't very well suitable for this kind of problems, hence Rust. But can you elaborate on what you mean by other platforms catching up with Elixir's inherent concurrency advantages? Which modern platforms give similar features?

jerf 7y ago
"The article says that the data is basically "per-user"," Given that this is a table of who is "online", I don't think that's per-user in the sense that you are inferring. I infer that it's not a whole bunch of little local data that doesn't interact, it's a big global table of who is online and not online, constantly being heavily read from and written to in real time. Consider from the perspective of Bob's Erlang process that he wants to go offline and notify all of his currently-online friends that is is going offline. Bob's Erlang process doesn't have that data. Bob's Erlang process is going to get it from the Big Table of Who's Online. That table is the problem; it can't be stored in Bob's Erlang process. I was at least imagining that the table could be partitioned into pieces pretty trivially (first X bits of the hash), but with Erlang's design, that implies an IPC just to ask some server process to give me the PID of the chunk I need to talk to, which itself is going to bottleneck. (In practice we'd probably cheat and use a NIF to do that, but that amounts to an admission that Erlang can't do this, so....) At smaller scales you could try to live update Bob's local information as it changes, but this breaks down in all sorts of ways at scales far smaller than Discord, scales much closer to "a single mid-sized company". "Another could be storing the data in mnesia, BEAM's internal mutable in-memory DB." I have used mnesia for loads literally a ten-thousandth as small as this, if that (I could probably tack two more zeros on there), and it breaks down. It is an absolutely ludicrous idea that mnesia could handle what Discord is doing here. Last I knew the official Erlang community consensus was basically that mnesia really shouldn't be used for anything serious; my experience backed that up. I think a non-trivial part of the reason why Erlang hasn't taken off is that its community still seems to exist in 2003, where it's a really incredible unique language that solves huge problems that nobody else does. In 2003, it rather has a point. But a lot of things have learned from Erlang, and incorporated its lessons into newer designs, and moved on. See my other comment for what other runtimes have Erlang's advantages, but I'd invite you just to consider what we seem to basically agree on here; Erlang would be wildly slower and require a lot more hardware than Rust, the Rust code probably wasn't that hard to write, ... and the Rust code is way more likely to be correct than the Erlang code, too. I mean, what more "catching up to Elixir's inherent concurrency advantages" in this context than "did a job Elixir couldn't possibly do" do you want?