3 ms·

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix. I actually think Elixir really doesn't have great performan

This fits my experience too. I've been able to crush elixir apps pretty easily, cowboy and phoenix. I actually think Elixir really doesn't have great performance. TechEmpower, which is IMO the most real world standardized tests out there shows that Phoenix doesn't even complete. And Elixir+Plug+Ecto performs worse than Rails, which is an entire framework. Everyone in Elixir land tells me "Oh those benchmarks don't matter". Yet they are heavily talked about, and referred to here and other places. They only don't matter if you perform terribly on them I suppose. And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. He's written extensively about how this type of test doesn't really fit elixir, etc, but ultimately it's just doing DB queries, why does this not work? I really think Elixir is mostly propaganda at this point. It's a huge mental paradigm shift and I have seen myself that it wasn't performant, and as you said you keep thinking "Oh I must be doing it wrong". I just cannot fathom why anyone in a decent sized company would use with all the negatives it has going for it. YMMV Proof below https://www.techempower.com/benchmarks/#section=data-r22&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r22&hw=...

josevalim 2y ago
> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2]. My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/application.rb#L33-L53 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... But almost nobody runs a Rails application like that in production. And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests. I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 [2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.
- DoesntMatter22 2y ago
  Sounds like you are agreeing with me. You tried to work on it years ago and didn't fix it. Even when it was completing last year it's performance was half of what Rails was, which isn't known to be fast [1]. The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot. If these people can't get good results in tuning Elixir, who can? Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/database.yml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r21&hw=...
  - josevalim 2y ago
    > The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less. Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land. > If these people can't get good results in tuning Elixir, who can? You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=176ba510-3607-4faa-996e-74f0778b88d4&hw=ph&test=query https://www.techempower.com/benchmarks/#section=test&runid=1... If you want to compare with Rails in particular, we beat Rails in every benchmark except two: 1. Single query: which will be tackled by better sizing our pools 2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too) And it does not include any of my proposed changes yet. My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it. [1]: https://ark.intel.com/content/www/us/en/ark/products/120474/intel-xeon-gold-5120-processor-19-25m-cache-2-20-ghz.html https://ark.intel.com/content/www/us/en/ark/products/120474/... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/auto_tune.rb https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

josevalim 2y ago

> And they say "Oh we didn't care to put much effort into it", Yet Jose Valim himself tried to work on it and didn't fix it. This is inaccurate. I have started looking again into solutions only last week [1]. My suspicion was always the database pool size was too small but, when I tried to contribute 4+ years ago, fine tuning was hard because it took too long, so I didn't pursue it further [2]. My discontent with the benchmarks is that they are not measuring what people effectively run in production. Since you mentioned Rails, here is how a Rails application looks like: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/application.rb#L33-L53 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... But almost nobody runs a Rails application like that in production. And then you look at the configuration of most stacks and they have been explicitly fine tuned to match exactly the concurrency traffic and patterns that the benchmark sends through. But, for most web applications, your web traffic is not homogeneous and you don't have a predetermined number of concurrent requests. I still believe those benchmarks are not indicative of what you will actually see in production. Most companies who have gone from Rails to Phoenix, for example, report a 10-20x reduction in operation size and costs. But it is clear at this point people put way too much stock on these benchmarks. The irony of it all is that, if someone copies these setups into their actual applications, they will most likely perform worse. Oh well. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 https://github.com/TechEmpower/FrameworkBenchmarks/pull/9302 [2]: https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 https://github.com/TechEmpower/FrameworkBenchmarks/pull/5432 - here you can see me increasing the database pool to 40... but most benchmarks today run with 512-1024 connections (which, once again, is most likely not what you would do in prod). In any case, we need to bump our numbers accordingly.

DoesntMatter22 2y ago
Sounds like you are agreeing with me. You tried to work on it years ago and didn't fix it. Even when it was completing last year it's performance was half of what Rails was, which isn't known to be fast [1]. The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Even if, somehow, Elixir is actually faster, which seemingly lacks evidence at this point other than some anecdotes by people who have a vested interest in people adopting the language (primarily Elixir dev shops and framework/language authors), the fact that the community of Elixir devs cannot get decent results on these tests says a lot. If these people can't get good results in tuning Elixir, who can? Not a surprise to see people ITT also noting that they ran into the same types of problems that the tests reveal. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/database.yml https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://www.techempower.com/benchmarks/#section=data-r21&hw=ph&test=update https://www.techempower.com/benchmarks/#section=data-r21&hw=...
- josevalim 2y ago
  > The database pool needs to be at 512 to 1024? Yet Rails seems to get away with far less than that and still have better performance.[2] Rails concurrency model is per process. So you will have `n_workers` pools and the amount of connections will be `n_workers * 3` based on the code you shared. The production machine has 28 total threads [1], which means Rails is starting 35 workers, according to their auto tuning code [2]. Overall, Rails uses 35x more pools than the Phoenix application and has twice the number of connections (105). It is absurd to imply that Rails is actually running with far less. Second of all, the database pool needs to be 512 or 1024 if your app server can push traffic quickly enough to the database. If you are increasing the Rails pool size and it is not getting faster, then Rails is your bottleneck. Based on the runs I did last week, it seems the pool is the bottleneck in Elixir's case. We will know for sure in a couple weeks once the PRs land. > If these people can't get good results in tuning Elixir, who can? You can look at the results from the latest community efforts in main: https://www.techempower.com/benchmarks/#section=test&runid=176ba510-3607-4faa-996e-74f0778b88d4&hw=ph&test=query https://www.techempower.com/benchmarks/#section=test&runid=1... If you want to compare with Rails in particular, we beat Rails in every benchmark except two: 1. Single query: which will be tackled by better sizing our pools 2. Plain text: where we are currently sending the wrong payload (but when we sent the proper payload, as in the last published round, we beat Rails too) And it does not include any of my proposed changes yet. My point is exactly that the feedback cycle is long, so it takes time, and I was not able to invest time before. But given how intent some folks are to misinterpret benchmarks (and assign malice to the whole community), it is probably better to continue looking into it. [1]: https://ark.intel.com/content/www/us/en/ark/products/120474/intel-xeon-gold-5120-processor-19-25m-cache-2-20-ghz.html https://ark.intel.com/content/www/us/en/ark/products/120474/... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Ruby/rails/config/auto_tune.rb https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...