4 ms·

the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting. That optimization is really important for some lower-performin

the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting. That optimization is really important for some lower-performing platforms of yesteryear (which the erlang VM does need to continue supporting) but basically irrelevant for modern platforms, and actually bad, if you're getting charged CPU credits. However, it's a command line switch away. I don't know if the default has been changed in more recent BEAM versions (I feel like they did, around the time they were changing defaults to favor cloud deploys, like setting up concurrency to detect cgroup vcores instead of machine vcores) The stressgrid folks did a RCA and posted the explanation in a subsequent, but most people rushing to shoot down erlang/elixir don't get around to reading that article. Extracted comments from the article: > we discovered that Elixir had much higher CPU usage than Go, and yet its responsiveness remained excellent. Some of our readers suggested that busy waiting may be responsible for this behavior. > c5.9xlarge and c5.4xlarge instances show similar results in responsiveness, and no meaningful difference with respect to busy wait settings. https://stressgrid.com/blog/beam_cpu_usage/ https://stressgrid.com/blog/beam_cpu_usage/

tiffanyh 5y ago
What is the "spin your cpus" setting exactly you're referring too? Would be good to share for other to know. Also, note that your link shared from stressgrid was published BEFORE the benchmarks links I shared from stressgrid. As such, wouldn't those benchmarks take into account your explanation?
- dnautics 5y ago
  busy wait spins your cpus. I don't know for sure, but the benchmarks you linked do some more RCA, which involves looking into why cowboy2 is not great (tl;dr HTTP2 support is hard) also, here is how stressgrid can POC (possibly too dangerous for prod?) to 100k cps (which is on par with what Go can do in the article you linked), you could recompile kernel and configure ranch differently, would probably need to be proven out more: https://stressgrid.com/blog/100k_cps_with_elixir/ https://stressgrid.com/blog/100k_cps_with_elixir/
  - tiffanyh 5y ago
    Your comment: > "the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting" stressgrid blog post: > "When running HTTP workloads with Cowboy on dedicated hardware, it would make sense to leave the default—busy waiting enabled—in place. When running BEAM on an OS kernel shared with other software, it makes sense to turn off busy waiting, to avoid stealing time from non-BEAM processes." stressgrid seems to recommend the opposite of what you mention, taken from the blog post you shared [0]. Am I misunderstanding? [0] https://stressgrid.com/blog/beam_cpu_usage/ https://stressgrid.com/blog/beam_cpu_usage/
    - dnautics 5y ago
      You are. I'm saying turn it off unless you're on legacy hardware that is known to need it (most people are in the cloud). Unless it's already off by default (I don't know off the top of my head) The perf looks bad because they are also looking at CPU usage. Getting to 100k rps is unrelated to busy wait.
      - tiffanyh 5y ago
        I feel like you're making an orthogonal statement related to cpu usage and the wait/spin settings for "cloud hardware". CPU usage has no baring on transactions per second. So why are you bringing it up. Spin/wait settings: how is "cloud hardware" different than "legacy hardware"? Both are servers located in a data center. Furthermore, the blog post goes to say that if you aren't running any other non-erlang services (you box is dedicated to Erlang) that you should KEEP the default setting of spin/wait since it won't cannobalize other services running on that box ... which is exactly how these benchmarks were tested. The fact of the matter is, Erlang had lower transactions per second than most of the languages benchmarked. My insight from this HN discussion is that where Erlang shines is having consistently low latency/response time. It's not the fastest, but it's the most consistent even under load.
        ellyagg 5y ago
        That's still really understating erlang's advantages though. Erlang is a lot easier to write working, custom concurrent and distributed systems in because of the design of its primitives and runtime. Erlang has a bunch of different advantes all related to controlling disparate pieces of software under one roof, so it's not easy to summarize why I love working with it in one comment. In general, I think blog posts like this do a pretty bad job of explaining it. At some point I'll publish my take and hopefully it'll reach the front page here.

tiffanyh 5y ago

What is the "spin your cpus" setting exactly you're referring too? Would be good to share for other to know. Also, note that your link shared from stressgrid was published BEFORE the benchmarks links I shared from stressgrid. As such, wouldn't those benchmarks take into account your explanation?

dnautics 5y ago
busy wait spins your cpus. I don't know for sure, but the benchmarks you linked do some more RCA, which involves looking into why cowboy2 is not great (tl;dr HTTP2 support is hard) also, here is how stressgrid can POC (possibly too dangerous for prod?) to 100k cps (which is on par with what Go can do in the article you linked), you could recompile kernel and configure ranch differently, would probably need to be proven out more: https://stressgrid.com/blog/100k_cps_with_elixir/ https://stressgrid.com/blog/100k_cps_with_elixir/
- tiffanyh 5y ago
  Your comment: > "the stressgrid benchmarks look bad because they forgot to turn off the "spin your cpus" setting" stressgrid blog post: > "When running HTTP workloads with Cowboy on dedicated hardware, it would make sense to leave the default—busy waiting enabled—in place. When running BEAM on an OS kernel shared with other software, it makes sense to turn off busy waiting, to avoid stealing time from non-BEAM processes." stressgrid seems to recommend the opposite of what you mention, taken from the blog post you shared [0]. Am I misunderstanding? [0] https://stressgrid.com/blog/beam_cpu_usage/ https://stressgrid.com/blog/beam_cpu_usage/
  - dnautics 5y ago
    You are. I'm saying turn it off unless you're on legacy hardware that is known to need it (most people are in the cloud). Unless it's already off by default (I don't know off the top of my head) The perf looks bad because they are also looking at CPU usage. Getting to 100k rps is unrelated to busy wait.
    - tiffanyh 5y ago
      I feel like you're making an orthogonal statement related to cpu usage and the wait/spin settings for "cloud hardware". CPU usage has no baring on transactions per second. So why are you bringing it up. Spin/wait settings: how is "cloud hardware" different than "legacy hardware"? Both are servers located in a data center. Furthermore, the blog post goes to say that if you aren't running any other non-erlang services (you box is dedicated to Erlang) that you should KEEP the default setting of spin/wait since it won't cannobalize other services running on that box ... which is exactly how these benchmarks were tested. The fact of the matter is, Erlang had lower transactions per second than most of the languages benchmarked. My insight from this HN discussion is that where Erlang shines is having consistently low latency/response time. It's not the fastest, but it's the most consistent even under load.
      - ellyagg 5y ago
        That's still really understating erlang's advantages though. Erlang is a lot easier to write working, custom concurrent and distributed systems in because of the design of its primitives and runtime. Erlang has a bunch of different advantes all related to controlling disparate pieces of software under one roof, so it's not easy to summarize why I love working with it in one comment. In general, I think blog posts like this do a pretty bad job of explaining it. At some point I'll publish my take and hopefully it'll reach the front page here.