4 ms·

So many people talk Erlang / Elixir insane concurrency … but I’m struggling to reconcile the talk with what I find in stress testing and benchmarks results. Ca

So many people talk Erlang / Elixir insane concurrency … but I’m struggling to reconcile the talk with what I find in stress testing and benchmarks results. Can you comment on why so many benchmarks show Erlang / Elixir performing significantly worse than other languages like Go. https://stressgrid.com/blog/webserver_benchmark/ https://stressgrid.com/blog/webserver_benchmark/ https://stressgrid.com/blog/cowboy_performance/ https://stressgrid.com/blog/cowboy_performance/ https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ EDIT: why the downvotes? I’d rather you post a comment to me so we can have a health dialogue.

eric4smith 5y ago
Elixir does not a fantastic computational story. That's why it has NIF's to bring in things like C or Rust to deal with the math stuff. Languages like Go, C, Rust will always beat Elixir/Erlang in the computationally intensive benchmarks. You would chose Elixir/Phoenix/Erlang for the concurrency and networking story.
- tiffanyh 5y ago
  But note, all of the benchmarks I posted in my parent post(especially the first two) are concurrency workloads - not numeric. And Erlang still performed noticeably worse than other languages.
  - toolz 5y ago
    > Erlang still performed noticeably worse than other languages I think you need to define worse here...unpredictable spikes in latency will give you plenty of headaches when trying to guess how much hardware you should throw at a service. Erlang's consistent latency here is what I would choose above everything that benchmark shows for almost every problem I've ever solved. Going fast at all costs is not a desirable trait for my software and I suspect it isn't for most peoples software. I want predictable behavior that operates gracefully under extreme circumstances.
  - jerf 5y ago
    In the 200Xs, the BEAM VM had a clear performance advantage in dealing with high-concurrency network loads. It has never had a raw performance advantage in terms of the bytecode that it implemented, that is, the Erlang/Elixir layer (it was generally faster than Python/Ruby, but that wasn't saying much, especially back then, but it had clear performance disadvantages vs. C/C++/Java), but it had a superior internal runtime that could make up for that in performance benchmarks, as long as you didn't try to run too much BEAM bytecode. Much like how NumPy is very fast, as long as you don't try to run too much pure Python with it. However, since BEAM doesn't have access to unique CPU instructions that nobody else has or anything else, and since a lot of focus across a lot of languages has been put on that problem, that particular advantage has waned, and Erlang to my eyes has indeed been outright passed on this front by multiple languages. In the 200Xs, I did not see people talking much about NIFs as a solution for performance; that talk has started as an effort to keep up with things like Go and other languages that have taken advantage of BEAM's lessons and explorations of the space. Personally, while I think a lot of the hoopla surrounding Erlang/Elixir isn't wrong per se, I do think a lot of it is outdated. They'll say "We do X and nobody else does!" but while that may have been true 10-15 years ago, it isn't anymore. There's no performance reason to pick Erlang/Elixir over Go, for instance, and if you take the models of memory access back to Go there isn't a huge organizational reason either. What Erlang/Elixir force you to do, you can voluntarily do in other languages too. And I think that's become true across a lot of other of the putative "advantages"; it isn't that Erlang/Elixir aren't nice in some ways, but I do wonder how much of the recent push is stemming from people who are experiencing some of these capabilities for the first time and trusting the Erlang/Elixir storyline that they're unique, when in fact they are increasingly just table stakes for a new language nowadays rather than special characteristics unique to the BEAM family of languages.
    - lostcolony 5y ago
      'What Erlang/Elixir force you to do, you can voluntarily do in other languages too'. Like immutability! :P Jokes aside, I agree for basic concurrency you can get pretty far with other modern languages. I think it's what brings people's attention to Erlang/Elixir, but I don't think it's the most important differentiator. It also isn't the one that Erlang's community (I can't speak to Elixir) really touts, except as one that is easily understood by those outside of it. The real benefit is fault tolerance. Everything about Erlang, the concurrency and distribution stories included, is built around fault tolerance. You need concurrency and distribution to be fault tolerant (can't have one bad process choking out others; can't have one bad machine taking the service down, etc). The immutability, the supervision tree, those also are about fault tolerance. I've written production systems in Go. It scaled better with way less tweaking than the JVM based stuff we'd written previously required. But it wasn't nearly as resilient to failure, or as predictable, as the Erlang stuff I've run in prod was.
  - sa1 5y ago
    I'm not disagreeing with your results, but you should be using https://github.com/giltene/wrk2 https://github.com/giltene/wrk2 based benchmarks to avoid coordinated omission errors in measuring latency.
- rkangel 5y ago
  The new JIT does improve the computational story a little, but yes it's a little like Python where you use it for orchestration and then offload the work. That said new systems like NX do make the 'offloading' part significantly cleaner for some applications.
dwohnitmok 5y ago
The Erlang ecosystem is centered around creating tools for dealing with concurrency. Concurrent programming concerns itself with trying to represent a certain set of semantics in which a program must deal with multiple requests that overlap in time. A concurrent program could ultimately be executed on a single logical processor via some form of time sharing. It just has to be able to deal with multiple requests which can overlap in time rather than neatly coming one after the other. derefr's comment that is a sibling to yours is a good example of pointing out Erlang's strengths when it comes to expressing concurrent semantics (notice nowhere in derefr's response is there any talk of performance). However, Erlang, as you have correctly pointed out, is not very good if you only care about parallelism. Parallelism is not concerned with semantics, but performance; it concerns itself with trying to make a given computation faster by using multiple physical hardware resources but maintaining the same semantics as a hypothetical non-parallel version. Your argument is a cogent one for why Erlang is not great if you only care about parallelism. Being able to scale a program to use 20 cores is no good if the same program written in another language could beat the pants off that program using just one core. However, that is an independent concern from concurrency.
- tiffanyh 5y ago
  Would you say then that Erlang has low latency and low standard deviation of how fast it will respond to any request under any amount of load. It might not have the highest transactions per second, but for those transactions it completes - it’ll do it fast and with the same low latency. There’s other languages that can perform more transaction per second but as load increases, their latency and standard deviations grows exponentially. (This is seen in the stressgrid benchmarks).
  - dwohnitmok 5y ago
    I'm not the best person to ask about Erlang's performance characteristics. I know them in broad strokes from light reading about the BEAM VM but I've never written any Erlang that has made it out to production, only for the smallest of hobby projects. So I don't know, e.g. what the P99 latency of a given Erlang program might be. That being said, my point was that concurrency is an independent topic from runtime performance (including latency), which is the domain of parallelism. Erlang's big selling point is it makes it possible to write concurrent programs whose functionality (not performance) would require oodles and oodles of code and discipline in some other languages. EDIT: On reflection I do think if what you're talking is catastrophic latency overruns caused by cascading queuing failures then Erlang can indeed help make it easier to avoid those pitfalls.
brightball 5y ago
Here’s a good one to get into deeper comparison with Python, Go and Elixir. It’s one of the few that I’ve seen that does a good job of showing more than just straight line speed. https://medium.com/@marcelo_lebre/a-tale-of-three-kings-e0be17a16e2b https://medium.com/@marcelo_lebre/a-tale-of-three-kings-e0be...
lostcolony 5y ago
Someone else alluded to this, but Erlang as a runtime prioritizes -latency-. Most languages prioritize -throughput-. These are fundamentally at odds with each other (oversimplifed, but - to maximize throughput you want things queuing up; to maximize latency you don't). This also isn't to say different versions of libraries and things don't muddy it some too, but look at the latency tail. From those techempower benchmarks, look at elixir-plug-ecto. 2.8 ms max latency, with an average of 1.0ms latency. That average latency is very respectable (but not the lowest), but that max? That max IS the lowest. What do you want from a web server? Something super fast for 99% of requests, and then takes orders of magnitude longer for the worst case, or something that is somewhat slower on average, but stays predictable? And as someone still else alluded to, those have to do with the runtime characteristics. How easy is it to write reasonably performant, correct, concurrent code? Erlang (and Elixir) makes it very easy; I would argue easier than any other language.
di4na 5y ago
Just woke up so multiple things. First of all some context. The BEAM community does not engage heavily with these benchmarks compared to a lot of others communities. That means that this code is definitely not optimised. On top of that, benchmarking makes hard to do what the BEAM community love above all else, which is to put the thingy in prod and then tweak the performance based on what you discover there. The reason to do it so much is that the BEAM come equipped for that. Second point is that as pointed down there, the BEAM optimise heavily for latency and graceful degradation. One important point is that if you look at the std deviation on latency and the latency itself here. They are super low. What this mean is that the BEAM system will still be highly reactive under load. And interactive. But also that we are nowhere near the max load they can deal in this benchmarks! If you wanted to compare you would need to overload the machine so that this latency climb. A latency so low mean that the machine is not overloaded. It may be at 100% CPU but it is totally coping ok. The BEAM is basically exploiting the machine to its limits better. The reasons are multiple but basically you should consider that these benchmark for the BEAM are not sweating it. And that means that this load is just a normal day for a BEAM setup, not one that would make you auto scale. In that light, it is a far different picture, no? Happy to discuss
- di4na 5y ago
  This comment lower point to this too in a slightly different way that i deeply agree with https://news.ycombinator.com/item?id=27683999 https://news.ycombinator.com/item?id=27683999
dnautics 5y ago
the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting. That optimization is really important for some lower-performing platforms of yesteryear (which the erlang VM does need to continue supporting) but basically irrelevant for modern platforms, and actually bad, if you're getting charged CPU credits. However, it's a command line switch away. I don't know if the default has been changed in more recent BEAM versions (I feel like they did, around the time they were changing defaults to favor cloud deploys, like setting up concurrency to detect cgroup vcores instead of machine vcores) The stressgrid folks did a RCA and posted the explanation in a subsequent, but most people rushing to shoot down erlang/elixir don't get around to reading that article. Extracted comments from the article: > we discovered that Elixir had much higher CPU usage than Go, and yet its responsiveness remained excellent. Some of our readers suggested that busy waiting may be responsible for this behavior. > c5.9xlarge and c5.4xlarge instances show similar results in responsiveness, and no meaningful difference with respect to busy wait settings. https://stressgrid.com/blog/beam_cpu_usage/ https://stressgrid.com/blog/beam_cpu_usage/
- tiffanyh 5y ago
  What is the "spin your cpus" setting exactly you're referring too? Would be good to share for other to know. Also, note that your link shared from stressgrid was published BEFORE the benchmarks links I shared from stressgrid. As such, wouldn't those benchmarks take into account your explanation?
  - dnautics 5y ago
    busy wait spins your cpus. I don't know for sure, but the benchmarks you linked do some more RCA, which involves looking into why cowboy2 is not great (tl;dr HTTP2 support is hard) also, here is how stressgrid can POC (possibly too dangerous for prod?) to 100k cps (which is on par with what Go can do in the article you linked), you could recompile kernel and configure ranch differently, would probably need to be proven out more: https://stressgrid.com/blog/100k_cps_with_elixir/ https://stressgrid.com/blog/100k_cps_with_elixir/
    - 5y ago
rkangel 5y ago
Note that Erlang OTP 24 (the latest release) includes a JIT for the first time. It only runs on x64 but should significantly improve performance on that platform. For some workloads people are reporting as much as a 40% improvement. I would expect to see some improvement in those benchmarks as a result. OTP 25 will also include JIT support for ARM64
archsurface 5y ago
Concurrency vs performance, not concurrency is performance.

eric4smith 5y ago

Elixir does not a fantastic computational story. That's why it has NIF's to bring in things like C or Rust to deal with the math stuff. Languages like Go, C, Rust will always beat Elixir/Erlang in the computationally intensive benchmarks. You would chose Elixir/Phoenix/Erlang for the concurrency and networking story.

tiffanyh 5y ago
But note, all of the benchmarks I posted in my parent post(especially the first two) are concurrency workloads - not numeric. And Erlang still performed noticeably worse than other languages.
- toolz 5y ago
  > Erlang still performed noticeably worse than other languages I think you need to define worse here...unpredictable spikes in latency will give you plenty of headaches when trying to guess how much hardware you should throw at a service. Erlang's consistent latency here is what I would choose above everything that benchmark shows for almost every problem I've ever solved. Going fast at all costs is not a desirable trait for my software and I suspect it isn't for most peoples software. I want predictable behavior that operates gracefully under extreme circumstances.
- jerf 5y ago
  In the 200Xs, the BEAM VM had a clear performance advantage in dealing with high-concurrency network loads. It has never had a raw performance advantage in terms of the bytecode that it implemented, that is, the Erlang/Elixir layer (it was generally faster than Python/Ruby, but that wasn't saying much, especially back then, but it had clear performance disadvantages vs. C/C++/Java), but it had a superior internal runtime that could make up for that in performance benchmarks, as long as you didn't try to run too much BEAM bytecode. Much like how NumPy is very fast, as long as you don't try to run too much pure Python with it. However, since BEAM doesn't have access to unique CPU instructions that nobody else has or anything else, and since a lot of focus across a lot of languages has been put on that problem, that particular advantage has waned, and Erlang to my eyes has indeed been outright passed on this front by multiple languages. In the 200Xs, I did not see people talking much about NIFs as a solution for performance; that talk has started as an effort to keep up with things like Go and other languages that have taken advantage of BEAM's lessons and explorations of the space. Personally, while I think a lot of the hoopla surrounding Erlang/Elixir isn't wrong per se, I do think a lot of it is outdated. They'll say "We do X and nobody else does!" but while that may have been true 10-15 years ago, it isn't anymore. There's no performance reason to pick Erlang/Elixir over Go, for instance, and if you take the models of memory access back to Go there isn't a huge organizational reason either. What Erlang/Elixir force you to do, you can voluntarily do in other languages too. And I think that's become true across a lot of other of the putative "advantages"; it isn't that Erlang/Elixir aren't nice in some ways, but I do wonder how much of the recent push is stemming from people who are experiencing some of these capabilities for the first time and trusting the Erlang/Elixir storyline that they're unique, when in fact they are increasingly just table stakes for a new language nowadays rather than special characteristics unique to the BEAM family of languages.
  - lostcolony 5y ago
    'What Erlang/Elixir force you to do, you can voluntarily do in other languages too'. Like immutability! :P Jokes aside, I agree for basic concurrency you can get pretty far with other modern languages. I think it's what brings people's attention to Erlang/Elixir, but I don't think it's the most important differentiator. It also isn't the one that Erlang's community (I can't speak to Elixir) really touts, except as one that is easily understood by those outside of it. The real benefit is fault tolerance. Everything about Erlang, the concurrency and distribution stories included, is built around fault tolerance. You need concurrency and distribution to be fault tolerant (can't have one bad process choking out others; can't have one bad machine taking the service down, etc). The immutability, the supervision tree, those also are about fault tolerance. I've written production systems in Go. It scaled better with way less tweaking than the JVM based stuff we'd written previously required. But it wasn't nearly as resilient to failure, or as predictable, as the Erlang stuff I've run in prod was.
- sa1 5y ago
  I'm not disagreeing with your results, but you should be using https://github.com/giltene/wrk2 https://github.com/giltene/wrk2 based benchmarks to avoid coordinated omission errors in measuring latency.
rkangel 5y ago
The new JIT does improve the computational story a little, but yes it's a little like Python where you use it for orchestration and then offload the work. That said new systems like NX do make the 'offloading' part significantly cleaner for some applications.

dwohnitmok 5y ago

The Erlang ecosystem is centered around creating tools for dealing with concurrency. Concurrent programming concerns itself with trying to represent a certain set of semantics in which a program must deal with multiple requests that overlap in time. A concurrent program could ultimately be executed on a single logical processor via some form of time sharing. It just has to be able to deal with multiple requests which can overlap in time rather than neatly coming one after the other. derefr's comment that is a sibling to yours is a good example of pointing out Erlang's strengths when it comes to expressing concurrent semantics (notice nowhere in derefr's response is there any talk of performance). However, Erlang, as you have correctly pointed out, is not very good if you only care about parallelism. Parallelism is not concerned with semantics, but performance; it concerns itself with trying to make a given computation faster by using multiple physical hardware resources but maintaining the same semantics as a hypothetical non-parallel version. Your argument is a cogent one for why Erlang is not great if you only care about parallelism. Being able to scale a program to use 20 cores is no good if the same program written in another language could beat the pants off that program using just one core. However, that is an independent concern from concurrency.

tiffanyh 5y ago
Would you say then that Erlang has low latency and low standard deviation of how fast it will respond to any request under any amount of load. It might not have the highest transactions per second, but for those transactions it completes - it’ll do it fast and with the same low latency. There’s other languages that can perform more transaction per second but as load increases, their latency and standard deviations grows exponentially. (This is seen in the stressgrid benchmarks).
- dwohnitmok 5y ago
  I'm not the best person to ask about Erlang's performance characteristics. I know them in broad strokes from light reading about the BEAM VM but I've never written any Erlang that has made it out to production, only for the smallest of hobby projects. So I don't know, e.g. what the P99 latency of a given Erlang program might be. That being said, my point was that concurrency is an independent topic from runtime performance (including latency), which is the domain of parallelism. Erlang's big selling point is it makes it possible to write concurrent programs whose functionality (not performance) would require oodles and oodles of code and discipline in some other languages. EDIT: On reflection I do think if what you're talking is catastrophic latency overruns caused by cascading queuing failures then Erlang can indeed help make it easier to avoid those pitfalls.

brightball 5y ago

Here’s a good one to get into deeper comparison with Python, Go and Elixir. It’s one of the few that I’ve seen that does a good job of showing more than just straight line speed. https://medium.com/@marcelo_lebre/a-tale-of-three-kings-e0be17a16e2b https://medium.com/@marcelo_lebre/a-tale-of-three-kings-e0be...

lostcolony 5y ago

Someone else alluded to this, but Erlang as a runtime prioritizes -latency-. Most languages prioritize -throughput-. These are fundamentally at odds with each other (oversimplifed, but - to maximize throughput you want things queuing up; to maximize latency you don't). This also isn't to say different versions of libraries and things don't muddy it some too, but look at the latency tail. From those techempower benchmarks, look at elixir-plug-ecto. 2.8 ms max latency, with an average of 1.0ms latency. That average latency is very respectable (but not the lowest), but that max? That max IS the lowest. What do you want from a web server? Something super fast for 99% of requests, and then takes orders of magnitude longer for the worst case, or something that is somewhat slower on average, but stays predictable? And as someone still else alluded to, those have to do with the runtime characteristics. How easy is it to write reasonably performant, correct, concurrent code? Erlang (and Elixir) makes it very easy; I would argue easier than any other language.

di4na 5y ago

Just woke up so multiple things. First of all some context. The BEAM community does not engage heavily with these benchmarks compared to a lot of others communities. That means that this code is definitely not optimised. On top of that, benchmarking makes hard to do what the BEAM community love above all else, which is to put the thingy in prod and then tweak the performance based on what you discover there. The reason to do it so much is that the BEAM come equipped for that. Second point is that as pointed down there, the BEAM optimise heavily for latency and graceful degradation. One important point is that if you look at the std deviation on latency and the latency itself here. They are super low. What this mean is that the BEAM system will still be highly reactive under load. And interactive. But also that we are nowhere near the max load they can deal in this benchmarks! If you wanted to compare you would need to overload the machine so that this latency climb. A latency so low mean that the machine is not overloaded. It may be at 100% CPU but it is totally coping ok. The BEAM is basically exploiting the machine to its limits better. The reasons are multiple but basically you should consider that these benchmark for the BEAM are not sweating it. And that means that this load is just a normal day for a BEAM setup, not one that would make you auto scale. In that light, it is a far different picture, no? Happy to discuss

di4na 5y ago
This comment lower point to this too in a slightly different way that i deeply agree with https://news.ycombinator.com/item?id=27683999 https://news.ycombinator.com/item?id=27683999

dnautics 5y ago

the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting. That optimization is really important for some lower-performing platforms of yesteryear (which the erlang VM does need to continue supporting) but basically irrelevant for modern platforms, and actually bad, if you're getting charged CPU credits. However, it's a command line switch away. I don't know if the default has been changed in more recent BEAM versions (I feel like they did, around the time they were changing defaults to favor cloud deploys, like setting up concurrency to detect cgroup vcores instead of machine vcores) The stressgrid folks did a RCA and posted the explanation in a subsequent, but most people rushing to shoot down erlang/elixir don't get around to reading that article. Extracted comments from the article: > we discovered that Elixir had much higher CPU usage than Go, and yet its responsiveness remained excellent. Some of our readers suggested that busy waiting may be responsible for this behavior. > c5.9xlarge and c5.4xlarge instances show similar results in responsiveness, and no meaningful difference with respect to busy wait settings. https://stressgrid.com/blog/beam_cpu_usage/ https://stressgrid.com/blog/beam_cpu_usage/

tiffanyh 5y ago
What is the "spin your cpus" setting exactly you're referring too? Would be good to share for other to know. Also, note that your link shared from stressgrid was published BEFORE the benchmarks links I shared from stressgrid. As such, wouldn't those benchmarks take into account your explanation?
- dnautics 5y ago
  busy wait spins your cpus. I don't know for sure, but the benchmarks you linked do some more RCA, which involves looking into why cowboy2 is not great (tl;dr HTTP2 support is hard) also, here is how stressgrid can POC (possibly too dangerous for prod?) to 100k cps (which is on par with what Go can do in the article you linked), you could recompile kernel and configure ranch differently, would probably need to be proven out more: https://stressgrid.com/blog/100k_cps_with_elixir/ https://stressgrid.com/blog/100k_cps_with_elixir/
  - 5y ago

rkangel 5y ago

Note that Erlang OTP 24 (the latest release) includes a JIT for the first time. It only runs on x64 but should significantly improve performance on that platform. For some workloads people are reporting as much as a 40% improvement. I would expect to see some improvement in those benchmarks as a result. OTP 25 will also include JIT support for ARM64

archsurface 5y ago

Concurrency vs performance, not concurrency is performance.