7 ms·

> Not to mention it's much faster than Ruby or Python or PHP... In some cases it's faster than Javascript although JS and Erlang/Elixir are, on average, tied on

> Not to mention it's much faster than Ruby or Python or PHP... In some cases it's faster than Javascript although JS and Erlang/Elixir are, on average, tied on performance. By what metric? Phoenix on Elixir doesn't perform any better than Ruby and a similarly lightweight framework like Roda: https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=db&l=zg21a7-f&f=zik0z3-zik0zj-zik0zj-zik0zj-zhxjwf-zik0zj-zijbpb-zik0zj-zik0zj-e7 https://www.techempower.com/benchmarks/#section=data-r18&hw=... NodeJS is also much faster than Elixir in other circumstances thanks to the V8 JIT. Elixir's and Erlang's strengths over other dynamic languages aren't really performance. It's mostly just that Phoenix performs much better than Rails.

di4na 7y ago
By every real production load we know. Techempower have not been representative of anything. The only reason elixir and phoenix is not higher is because their code is buggy and they discouraged the community to try to fix it.
- jashmatthews 7y ago
  Can you give us an example of a performance improvement which was rejected which didn't break the rules? There were performance improvements merged just days ago with input from Jose Valim.
  - pdimitar 7y ago
    Q: Do you guys allow DB connection pools be changed? I know for a fact from my production experience that a small enough number for a DB connection pool in a Phoenix project can cripple its performance due to a lot of requests waiting for the pool.
    - jashmatthews 7y ago
      I don't work for TE but this is all covered in their documentation, which is great! https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview#general-test-requirements https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Proj... DB pool size is very subjective much dependent on how long a connection is held without making a query. A recent PR for Phoenix significantly improved performance by _decreasing_ the pool size: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153
      - pdimitar 7y ago
        Oh well, 256 was too much anyway! I meant more like, people were using pools of 10 on very beefy servers and were surprised at bad performance. I agree with the PR author that the DB pool should be anywhere from one to double the CPU cores. Thanks for the pointer. Currently reading the docs. Wonder if I can contribute to another language (not just Elixir).
  - di4na 7y ago
    i did not say it was rejected. I said the community lost the momentum to work on it. Mostly because it felt nearly impossible to get return on perf. When people had momentum to do it last time, we were never able to get perf numbers and to merge fixes before the official run. This basically killed most of the interest by long time members of the community to engage with the benchmark. I am ofc not saying it is a good thing from the elixir community. Just that it is the sad state of the momentum to change it right now.
thibaut_barrere 7y ago
I can only speak from my own experience but here is a data point (note that I do not use Elixir only, but also other technologies, in production). For instance, I implemented a HTTP "business proxy" last year, which needs to be able to keep a large number of concurrent connections & proxy the call to a slow responding remote server. I made multiple implementations (including Go, Node, Ruby with Roda, JRuby, Elixir, Crystal). In this specific case I had, Elixir throughput & stability was by far the best. The gap (under load) between min & max was also the best (all this without tuning), and so was the CPU use. Of course, YMMV, make your own experiences!
- pdimitar 7y ago
  I'd be extremely interested if you open sourced all those repositories. I thought of doing something very similar and your shared experience would be invaluable.
  - thibaut_barrere 7y ago
    I know (you are not the first to ask!) but honestly, sorry, I won't and I will let anyone truly interested to do their own experiments :-) This type of topic is too prone to endless debates on the validity of this and that, and the fact that anyone proficient in whichever stack I tested could definitely improve it (including me on the subpart of those stacks which I'm most proficient with!). I don't want to lose any energy with this, and I have made my decision for that business case (in this case, a very happy one).
    - swat535 7y ago
      > I know (you are not the first to ask!) but honestly, sorry, I won't and I will let anyone truly interested to do their own experiments You made an extraordinary claim, yet refuse to provide any evidence. Either backup your claims with data or don't make blanket statement perhaps? As to your points regarding scrutiny.. that's the whole point. How else would be able to measure the metrics?
      - 7y ago
pdimitar 7y ago
By the metric of the average Phoenix request taking between 0.3ms and 10ms as opposed to 100% the same app in Rails never responding for less than 230ms -- an app that I rewrote to Elixir/Phoenix from scratch over the course of months. You are correct in your specification though; I should have said Phoenix in particular and web development in general, not Elixir itself. The BEAM languages aren't excellent in raw performance, that much is true. Also I am yet to be impressed by JS web apps. They are quick when you spin them off on your local machine or on a world-unknown website that only you and 3 buddies visit. But I've seen NodeJS apps crumble at 10 reqs/sec routinely. I'll grant you that this was 2 years ago though. --- In short, I am not claiming to have made a scientific and thorough research in ideal conditions. I did work with PHP and Ruby for a long time though (and have been partially exposed to Python), and the difference in performance and reliability between them and Elixir is night and day. I've seen an Elixir app with a rudimentary 20-tables data schema endure 1000 reqs/sec on a RPi 3B. --- Final note: I'd like to see TechEmpower's benchmarks based off of a more real-world app like https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld as well though.
udfalkso 7y ago
I'll pile on here with my experience. I took a fairly complex Python/Django app and rewrote it into Elixir/Phoenix (and ripped out nearly all the complex caching along the way). The Elixir/Phoenix app rewrite was a little more than 10x faster, much more consistent in response times, and needed far fewer resources to run.
- jashmatthews 7y ago
  There's no secret sauce in Elixir which isn't in Python to make it 10x faster: https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac6ee0e0d https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac... Elixir and BEAM are clearly enormously better at some tasks but general performance is actually very similar between Elixir, Ruby, and Python if you use similarly performant frameworks.
  - udfalkso 7y ago
    Ok, but why did my app get 10 times faster? Why do so many other people report similar results? Perhaps factorial benchmarks don't translate well to the comparison of complex real world systems. I think, when it comes to web apps specifically, there are a few advantages that make a practical difference. These seem to be the major ones: - Compiled templates and more efficient string interpolation - Improved parallelism and lighter weight processes. This leads to better handling of many requests in parallel (esp. while most are just waiting on IO), better db connection pooling, less blocking, etc. A few other ideas are mentioned here by someone who seems to know more about this than I do: https://news.ycombinator.com/item?id=12014243 https://news.ycombinator.com/item?id=12014243
    - zbentley 7y ago
      > Ok, but why did my app get 10 times faster? You rewrote it. The language you rewrote it in probably has less to do with it than rebuilding all at once what initially was built over time/organically.
      - 7y ago
keithnoizu 7y ago
Beam thread handling is vastly different than node, I'd be deeply shocked to see node handle as well at load on the same hardware as a similar configured elixir/erlang stack.

di4na 7y ago

By every real production load we know. Techempower have not been representative of anything. The only reason elixir and phoenix is not higher is because their code is buggy and they discouraged the community to try to fix it.

jashmatthews 7y ago
Can you give us an example of a performance improvement which was rejected which didn't break the rules? There were performance improvements merged just days ago with input from Jose Valim.
- pdimitar 7y ago
  Q: Do you guys allow DB connection pools be changed? I know for a fact from my production experience that a small enough number for a DB connection pool in a Phoenix project can cripple its performance due to a lot of requests waiting for the pool.
  - jashmatthews 7y ago
    I don't work for TE but this is all covered in their documentation, which is great! https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview#general-test-requirements https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Proj... DB pool size is very subjective much dependent on how long a connection is held without making a query. A recent PR for Phoenix significantly improved performance by _decreasing_ the pool size: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5153
    - pdimitar 7y ago
      Oh well, 256 was too much anyway! I meant more like, people were using pools of 10 on very beefy servers and were surprised at bad performance. I agree with the PR author that the DB pool should be anywhere from one to double the CPU cores. Thanks for the pointer. Currently reading the docs. Wonder if I can contribute to another language (not just Elixir).
- di4na 7y ago
  i did not say it was rejected. I said the community lost the momentum to work on it. Mostly because it felt nearly impossible to get return on perf. When people had momentum to do it last time, we were never able to get perf numbers and to merge fixes before the official run. This basically killed most of the interest by long time members of the community to engage with the benchmark. I am ofc not saying it is a good thing from the elixir community. Just that it is the sad state of the momentum to change it right now.

thibaut_barrere 7y ago

I can only speak from my own experience but here is a data point (note that I do not use Elixir only, but also other technologies, in production). For instance, I implemented a HTTP "business proxy" last year, which needs to be able to keep a large number of concurrent connections & proxy the call to a slow responding remote server. I made multiple implementations (including Go, Node, Ruby with Roda, JRuby, Elixir, Crystal). In this specific case I had, Elixir throughput & stability was by far the best. The gap (under load) between min & max was also the best (all this without tuning), and so was the CPU use. Of course, YMMV, make your own experiences!

pdimitar 7y ago
I'd be extremely interested if you open sourced all those repositories. I thought of doing something very similar and your shared experience would be invaluable.
- thibaut_barrere 7y ago
  I know (you are not the first to ask!) but honestly, sorry, I won't and I will let anyone truly interested to do their own experiments :-) This type of topic is too prone to endless debates on the validity of this and that, and the fact that anyone proficient in whichever stack I tested could definitely improve it (including me on the subpart of those stacks which I'm most proficient with!). I don't want to lose any energy with this, and I have made my decision for that business case (in this case, a very happy one).
  - swat535 7y ago
    > I know (you are not the first to ask!) but honestly, sorry, I won't and I will let anyone truly interested to do their own experiments You made an extraordinary claim, yet refuse to provide any evidence. Either backup your claims with data or don't make blanket statement perhaps? As to your points regarding scrutiny.. that's the whole point. How else would be able to measure the metrics?
    - 7y ago

pdimitar 7y ago

By the metric of the average Phoenix request taking between 0.3ms and 10ms as opposed to 100% the same app in Rails never responding for less than 230ms -- an app that I rewrote to Elixir/Phoenix from scratch over the course of months. You are correct in your specification though; I should have said Phoenix in particular and web development in general, not Elixir itself. The BEAM languages aren't excellent in raw performance, that much is true. Also I am yet to be impressed by JS web apps. They are quick when you spin them off on your local machine or on a world-unknown website that only you and 3 buddies visit. But I've seen NodeJS apps crumble at 10 reqs/sec routinely. I'll grant you that this was 2 years ago though. --- In short, I am not claiming to have made a scientific and thorough research in ideal conditions. I did work with PHP and Ruby for a long time though (and have been partially exposed to Python), and the difference in performance and reliability between them and Elixir is night and day. I've seen an Elixir app with a rudimentary 20-tables data schema endure 1000 reqs/sec on a RPi 3B. --- Final note: I'd like to see TechEmpower's benchmarks based off of a more real-world app like https://github.com/gothinkster/realworld https://github.com/gothinkster/realworld as well though.

udfalkso 7y ago

I'll pile on here with my experience. I took a fairly complex Python/Django app and rewrote it into Elixir/Phoenix (and ripped out nearly all the complex caching along the way). The Elixir/Phoenix app rewrite was a little more than 10x faster, much more consistent in response times, and needed far fewer resources to run.

jashmatthews 7y ago
There's no secret sauce in Elixir which isn't in Python to make it 10x faster: https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac6ee0e0d https://gist.github.com/jamatthews/7b41b5ce5058f3f4ed674b6ac... Elixir and BEAM are clearly enormously better at some tasks but general performance is actually very similar between Elixir, Ruby, and Python if you use similarly performant frameworks.
- udfalkso 7y ago
  Ok, but why did my app get 10 times faster? Why do so many other people report similar results? Perhaps factorial benchmarks don't translate well to the comparison of complex real world systems. I think, when it comes to web apps specifically, there are a few advantages that make a practical difference. These seem to be the major ones: - Compiled templates and more efficient string interpolation - Improved parallelism and lighter weight processes. This leads to better handling of many requests in parallel (esp. while most are just waiting on IO), better db connection pooling, less blocking, etc. A few other ideas are mentioned here by someone who seems to know more about this than I do: https://news.ycombinator.com/item?id=12014243 https://news.ycombinator.com/item?id=12014243
  - zbentley 7y ago
    > Ok, but why did my app get 10 times faster? You rewrote it. The language you rewrote it in probably has less to do with it than rebuilding all at once what initially was built over time/organically.
    - 7y ago

keithnoizu 7y ago

Beam thread handling is vastly different than node, I'd be deeply shocked to see node handle as well at load on the same hardware as a similar configured elixir/erlang stack.