4 ms·

Is Erlang's BEAM VM really slow compared to other virtual machines, or is it Phoenix? It seems to be a bit of a laggard on benchmarks [1] and there's been this

Is Erlang's BEAM VM really slow compared to other virtual machines, or is it Phoenix? It seems to be a bit of a laggard on benchmarks [1] and there's been this thread going on for years [2] which hasn't reached a conclusion. [1] https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=plaintext&l=zg24n1-f https://www.techempower.com/benchmarks/#section=data-r18&hw=... [2] https://elixirforum.com/t/techempower-benchmarks/171 https://elixirforum.com/t/techempower-benchmarks/171 EDIT: Thanks everyone for the responses.

TylerE 7y ago
It's much more complicated than a simple slow <--> fast continuum. Elixir is pretty slow for simple singlethreaded CPU bound tasks. It's incredibly good at multiprocessing, which is really useful in the websphere since your workload is much more likely to resemble thousands of simultaneous communications that need to not block each other rather than calculating a 13042345235223452 digit prime number.
- taspeotis 7y ago
  > It's incredibly good at multiprocessing, which is really useful in the websphere Right so the benchmark I linked to was TechEmpower Web Framework Benchmarks, and Phoenix managed 186,774 requests/sec on a 14C/28T Xeon to do basically nothing (echo Hello World) and ASP.NET Core managed ... seven million.
  - TylerE 7y ago
    Have you considered that's revealing more about the webserver used than the software running behind it? Plus, 186k/sec is still a hell of a lot.
    - taspeotis 7y ago
      That seems to be what the forum thread is about. It couldn't possibly be that Phoenix is slow, it must be ...
  - ghayes 7y ago
    I'd throw this hat into the arena, which is the creator of Phoenix trying to push the framework to its limits (from a few years ago) and getting 2MM active websocket connections from one server: https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://www.phoenixframework.org/blog/the-road-to-2-million-...
    - leeoniya 7y ago
      or 1MM on an old laptop with uWebSockets.js: https://medium.com/@alexhultman/millions-of-active-websockets-with-node-js-7dc575746a01 https://medium.com/@alexhultman/millions-of-active-websocket...
    - taspeotis 7y ago
      I think that's a good example of how Phoenix can scale but not necessarily how fast it is. Scale is one part of performance but I should have made it more clear: I'm interested in how fast it can be. Looking at those charts it takes them 7.5 minutes to reach 2 million subscribers. Without knowing how much work is involved for each subscription it's hard to take away information about its speed.
      - dnautics 7y ago
        Why do you care so much about pure speed out of context?
      - kilburn 7y ago
        > I should have made it more clear: I'm interested in how fast it can be. The answer you are looking for is: much slower than .NET core. The "being good at multiprocessing" of the BEAM VM people are bringing up is not about raw speed at it: it is about giving you the tools to actually being able to do multiprocessing in a sensible, stable manner. For instance, what happens if a bug in your program blocks threads for a long time in .NET core? What if one thread crashes? Do you have any guarantees that one thread cannot affect others in any way? Etc. Some developers think that giving up raw performance for guarantees in these areas makes sense. Other developers think they can deal with those issues without nice guarantees from the platform and thus prefer the speed. There's no definitely right answer...
- nathan_long 7y ago
  I like to say that the BEAM's preemptive scheduling provides non-blocking IO AND non-blocking computation. To expand on your workload description: things like waiting for a database query or an API result are smoothly handled in this paradigm; the process goes to sleep for a bit whle another request is processed. If your workflow is like a delivery truck which makes a lot of stops, "faster trucks" doesn't help much with throughput. "More trucks" is better.
derefr 7y ago
BEAM is by default tuned for throughput over latency, but also for soft-realtime responsiveness over throughput. The right benchmark for BEAM performance isn’t how fast it can do a single-threaded or even multi-threaded CPU-bound task; but rather the 99th percentile packet-to-pcket latency + makespan + memory-requirement volatility of a C10M workload. In other words: if the VM were hosting a million VoIP calls, how many frames would it deliver “on time” such that the client wouldn’t discard them? (This is not to say you can’t tune BEAM to the needs of other workloads; those are just the defaults. Though, because that is the default assumed workload, it’s also the one most optimization effort goes into.) BEAM is also pure-functional in a way that a lot of VMs hosting pure-functional languages aren’t; things like mutable atomic memory handles were only introduced recently, and there’s no plan to rewrite core operations in terms of them, because these primitives are less predictable in their time costs than equivalent functional primitives. You can go fast under BEAM, but the people who want to do so, often decide that it’s too hard to go fast while also retaining soft-real-time guarantees they began using a BEAM language to attain; so they instead use one of BEAM’s many IPC bridges to hook up a fast native “port program” written in another language to the BEAM node, ensuring that any hiccups or crashes in the native code are isolated to its own address space and execution threads. This last consideration means that very few people are really driving BEAM’s performance forward, because so many instead take the “escape hatch” of low-overhead IPC. Taken as whole systems, though, you’ll find BEAM in a lot of services that go very fast indeed—you just won’t usually find BEAM itself handling the performance-critical part!
- taspeotis 7y ago
  Thank you for the explanation of BEAM + its objectives and internals. > It's incredibly good at multiprocessing, which is really useful in the websphere So on that benchmark I linked, if you switched to the latency tab Phoenix clocks in at 14.6ms average latency with a standard deviation of 23ms and a max of 460.7ms. The best looking ASP.NET Core result from a standard deviation and max point of view is 1.5ms average latency, 1.4ms standard deviation and 49ms max. So again it's hard for me to tell whether it's BEAM or Phoenix that's responsible for the relative performance hit.
  - 7y ago
josevalim 7y ago
I have commented this in other places but the main explanation for the benchmark you linked is that Phoenix (and its underlying Cowboy webserver) are not optimized for this particular workflow. For a plain-text benchmark, you are ultimately measuring the ability of sitting on top of a socket and reading/writing to that socket as fast as you can. However, Phoenix/Cowboy, whenever there is an HTTP connection, spawns a VM light-weight process to own that connection, and then each individual request runs in its own light-weight process too. This comes with its own guarantees in terms of state isolation, reasoning about failures, and you get both I/O and CPU concurrency for free - if you were to do any meaningful work on those requests. Even though the VM processes are cheap to spawn and lightweight, it is overhead compared to something that is just directly reading and writing to the socket. I actually wrote a proof of concept where we just sit on top of the socket without spawning processes and it performs quite well - although I don't think it has any practical purpose. There is also an interesting article from [StressGrid](https://stressgrid.com/blog/cowboy_performance_part_2/ https://stressgrid.com/blog/cowboy_performance_part_2/) that shows how removing the per-request process speeds it up by ~60%. But once you are going through long requests (i.e. 1ms because you need to talk to a database or API, encode JSON, etc), these differences tend to matter less and the concurrency model starts to give you an upper hand.
- taspeotis 7y ago
  Thanks. So a 60% improvement would be approximately 300K requests/sec vs. ASP.NET Core's seven million. Which still looks concerning. > But once you are going through long requests (i.e. 1ms because you need to talk to a database or API, encode JSON, etc), these differences tend to matter less and the concurrency model starts to give you an upper hand. If you look at the other tabs in the benchmark there are "heavier" loads that include database access and the most favourable one I can find is "Fortunes" which puts Phoenix at 17.8% of the relative performance of the best ASP.NET Core one.
  - josevalim 7y ago
    For the fortunes case, you are still comparing a web framework with a platform. When comparing web frameworks, some ASP.NET frameworks performs worse, others perform better, but at most 2.4x faster. On the plain text one, the difference between frameworks is at most 6x. You would have to yank Phoenix and sit directly on top of a socket, as per my previous comment, if you want to compare both VMs.
gregors 7y ago
I think the repo with these benchmarks have some really interesting things in them in general. It's really worth taking a peek at some of the implementations. The aspcore platform benchmark does some cool trickery based on whether the test even requires a database using compiler #if directives. Though it seems to be flagged as "realistic approach" Dunno if I agree with that. I think I'm looking at the right code. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/PlatformBenchmarks/BenchmarkApplication.cs#L58 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
zinclozenge 7y ago
I'm typing this all from memory so I might be wrong, but the short answer is that gen_tcp:recv/2 is a blocking call. There is a private API called prim_inet:async_recv which is async, and ironically gen_tcp builds on it. There have been attempts writing experimental HTTP servers using prim_inet:async_recv which have much better performance,

TylerE 7y ago

It's much more complicated than a simple slow <--> fast continuum. Elixir is pretty slow for simple singlethreaded CPU bound tasks. It's incredibly good at multiprocessing, which is really useful in the websphere since your workload is much more likely to resemble thousands of simultaneous communications that need to not block each other rather than calculating a 13042345235223452 digit prime number.

taspeotis 7y ago
> It's incredibly good at multiprocessing, which is really useful in the websphere Right so the benchmark I linked to was TechEmpower Web Framework Benchmarks, and Phoenix managed 186,774 requests/sec on a 14C/28T Xeon to do basically nothing (echo Hello World) and ASP.NET Core managed ... seven million.
- TylerE 7y ago
  Have you considered that's revealing more about the webserver used than the software running behind it? Plus, 186k/sec is still a hell of a lot.
  - taspeotis 7y ago
    That seems to be what the forum thread is about. It couldn't possibly be that Phoenix is slow, it must be ...
- ghayes 7y ago
  I'd throw this hat into the arena, which is the creator of Phoenix trying to push the framework to its limits (from a few years ago) and getting 2MM active websocket connections from one server: https://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://www.phoenixframework.org/blog/the-road-to-2-million-...
  - leeoniya 7y ago
    or 1MM on an old laptop with uWebSockets.js: https://medium.com/@alexhultman/millions-of-active-websockets-with-node-js-7dc575746a01 https://medium.com/@alexhultman/millions-of-active-websocket...
  - taspeotis 7y ago
    I think that's a good example of how Phoenix can scale but not necessarily how fast it is. Scale is one part of performance but I should have made it more clear: I'm interested in how fast it can be. Looking at those charts it takes them 7.5 minutes to reach 2 million subscribers. Without knowing how much work is involved for each subscription it's hard to take away information about its speed.
    - dnautics 7y ago
      Why do you care so much about pure speed out of context?
    - kilburn 7y ago
      > I should have made it more clear: I'm interested in how fast it can be. The answer you are looking for is: much slower than .NET core. The "being good at multiprocessing" of the BEAM VM people are bringing up is not about raw speed at it: it is about giving you the tools to actually being able to do multiprocessing in a sensible, stable manner. For instance, what happens if a bug in your program blocks threads for a long time in .NET core? What if one thread crashes? Do you have any guarantees that one thread cannot affect others in any way? Etc. Some developers think that giving up raw performance for guarantees in these areas makes sense. Other developers think they can deal with those issues without nice guarantees from the platform and thus prefer the speed. There's no definitely right answer...
nathan_long 7y ago
I like to say that the BEAM's preemptive scheduling provides non-blocking IO AND non-blocking computation. To expand on your workload description: things like waiting for a database query or an API result are smoothly handled in this paradigm; the process goes to sleep for a bit whle another request is processed. If your workflow is like a delivery truck which makes a lot of stops, "faster trucks" doesn't help much with throughput. "More trucks" is better.

derefr 7y ago

BEAM is by default tuned for throughput over latency, but also for soft-realtime responsiveness over throughput. The right benchmark for BEAM performance isn’t how fast it can do a single-threaded or even multi-threaded CPU-bound task; but rather the 99th percentile packet-to-pcket latency + makespan + memory-requirement volatility of a C10M workload. In other words: if the VM were hosting a million VoIP calls, how many frames would it deliver “on time” such that the client wouldn’t discard them? (This is not to say you can’t tune BEAM to the needs of other workloads; those are just the defaults. Though, because that is the default assumed workload, it’s also the one most optimization effort goes into.) BEAM is also pure-functional in a way that a lot of VMs hosting pure-functional languages aren’t; things like mutable atomic memory handles were only introduced recently, and there’s no plan to rewrite core operations in terms of them, because these primitives are less predictable in their time costs than equivalent functional primitives. You can go fast under BEAM, but the people who want to do so, often decide that it’s too hard to go fast while also retaining soft-real-time guarantees they began using a BEAM language to attain; so they instead use one of BEAM’s many IPC bridges to hook up a fast native “port program” written in another language to the BEAM node, ensuring that any hiccups or crashes in the native code are isolated to its own address space and execution threads. This last consideration means that very few people are really driving BEAM’s performance forward, because so many instead take the “escape hatch” of low-overhead IPC. Taken as whole systems, though, you’ll find BEAM in a lot of services that go very fast indeed—you just won’t usually find BEAM itself handling the performance-critical part!

taspeotis 7y ago
Thank you for the explanation of BEAM + its objectives and internals. > It's incredibly good at multiprocessing, which is really useful in the websphere So on that benchmark I linked, if you switched to the latency tab Phoenix clocks in at 14.6ms average latency with a standard deviation of 23ms and a max of 460.7ms. The best looking ASP.NET Core result from a standard deviation and max point of view is 1.5ms average latency, 1.4ms standard deviation and 49ms max. So again it's hard for me to tell whether it's BEAM or Phoenix that's responsible for the relative performance hit.
- 7y ago

josevalim 7y ago

I have commented this in other places but the main explanation for the benchmark you linked is that Phoenix (and its underlying Cowboy webserver) are not optimized for this particular workflow. For a plain-text benchmark, you are ultimately measuring the ability of sitting on top of a socket and reading/writing to that socket as fast as you can. However, Phoenix/Cowboy, whenever there is an HTTP connection, spawns a VM light-weight process to own that connection, and then each individual request runs in its own light-weight process too. This comes with its own guarantees in terms of state isolation, reasoning about failures, and you get both I/O and CPU concurrency for free - if you were to do any meaningful work on those requests. Even though the VM processes are cheap to spawn and lightweight, it is overhead compared to something that is just directly reading and writing to the socket. I actually wrote a proof of concept where we just sit on top of the socket without spawning processes and it performs quite well - although I don't think it has any practical purpose. There is also an interesting article from [StressGrid](https://stressgrid.com/blog/cowboy_performance_part_2/ https://stressgrid.com/blog/cowboy_performance_part_2/) that shows how removing the per-request process speeds it up by ~60%. But once you are going through long requests (i.e. 1ms because you need to talk to a database or API, encode JSON, etc), these differences tend to matter less and the concurrency model starts to give you an upper hand.

taspeotis 7y ago
Thanks. So a 60% improvement would be approximately 300K requests/sec vs. ASP.NET Core's seven million. Which still looks concerning. > But once you are going through long requests (i.e. 1ms because you need to talk to a database or API, encode JSON, etc), these differences tend to matter less and the concurrency model starts to give you an upper hand. If you look at the other tabs in the benchmark there are "heavier" loads that include database access and the most favourable one I can find is "Fortunes" which puts Phoenix at 17.8% of the relative performance of the best ASP.NET Core one.
- josevalim 7y ago
  For the fortunes case, you are still comparing a web framework with a platform. When comparing web frameworks, some ASP.NET frameworks performs worse, others perform better, but at most 2.4x faster. On the plain text one, the difference between frameworks is at most 6x. You would have to yank Phoenix and sit directly on top of a socket, as per my previous comment, if you want to compare both VMs.

gregors 7y ago

I think the repo with these benchmarks have some really interesting things in them in general. It's really worth taking a peek at some of the implementations. The aspcore platform benchmark does some cool trickery based on whether the test even requires a database using compiler #if directives. Though it seems to be flagged as "realistic approach" Dunno if I agree with that. I think I'm looking at the right code. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/CSharp/aspnetcore/PlatformBenchmarks/BenchmarkApplication.cs#L58 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

zinclozenge 7y ago

I'm typing this all from memory so I might be wrong, but the short answer is that gen_tcp:recv/2 is a blocking call. There is a private API called prim_inet:async_recv which is async, and ironically gen_tcp builds on it. There have been attempts writing experimental HTTP servers using prim_inet:async_recv which have much better performance,