4 ms·

Can anyone doing Phoenix in production share their typical HTTP response times? I built a small Phoenix project about a year ago, and compared to Rails it wasn

Can anyone doing Phoenix in production share their typical HTTP response times? I built a small Phoenix project about a year ago, and compared to Rails it wasn't quite as fluent but still pretty nice. But ever since then, I've been trying to decide if the extra development time is worth it for the performance gains. I get that Elixir is more scalable, and I love how the RAM requirements compared to Rails are tiny, but what I still haven't been able to really answer is the latency improvement. According to these benchmarks it is about as fast as Django: https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=query&b=2&s=1&l=sg&d=1n https://www.techempower.com/benchmarks/#section=data-r14&hw=... Is that really the best I can hope for?

blatyo 9y ago
Here's a screenshot of mean times for one of my company's app. http://take.ms/qGVOA http://take.ms/qGVOA The one at the top is proxying the request to a java app which happens to be a little slow.
chrismccord 9y ago
Phoenix creator here. The techmeme results are not representative of the framework. Several months ago they added Phoenix in a preview, but it was a very poor implementation. They were testing JSON benchmarks through the :browser pipeline, complete with crsf token generation. They had a dev DB pool size of 10, where other frameworks were given of pool size of 100. And they also had heavy IO logging, where other frameworks did no logging. We sent a PR to address these issues, and I was hoping to see true results in the subsequent runs, but no preview was provided this time and we weren't able to work with them on the error rate issues they ran into. After these experiences we've decided the core-team's time is better spent on the framework than chasing techmeme issues. If you go by real-world success stories from companies switching from Django, Rails, etc, you can certainly expect much better.
- alberth 9y ago
  Awesome response. Maybe to put the naysayers to rest you guys can update http://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections http://www.phoenixframework.org/blog/the-road-to-2-million-w...
- pjungwir 9y ago
  Okay, thanks for the reply. I asked about this benchmark once before (almost a year ago I expect), and was told roughly the same, so I was hoping things had improved by now. I understand it's just a benchmark and you have other things to do, but it does matter to me and I expect others. None of us want to invest a lot of time becoming experts in a new world and then not gain what we hoped. Personally these benchmarks make me think if I'm going to trade dev time for performance I should just use Rocket. That's probably not a reasonable position though. :-) Anyway, thank you for Phoenix, and I will look for opportunities to give it another spin.
- awj 9y ago
  > The techmeme results are not representative of the framework. I've seen several instances of this, now. Some Rails benchmarks jumped up pretty high as well when they managed to get someone to competently configure Puma. Practically the same story, just changing out names. Take techempower benchmarks with a boatload of salt. Not only are they not particularly representative of your real world use cases, they apparently are often not representative of competent use of the tools they're benchmarking. Hopefully this is an issue the techempower people figure out some way to address. It would be nice if this was a more trustworthy resource.
deleted 9y ago
[deleted]
udfalkso 9y ago
I recently rewrote an (admittedly old and stale) django app from the ground up with Elixir/Phoenix. The website gets quite a bit of traffic. I went from an elaborate multi-tier caching setup, with varnish and memcache, to a ZERO caching setup. The amount of complexity reduced by doing this is huge. The Elixir app hits the postgres db for nearly every request and I'm getting average response times around 45ms. Quite a bit of that is database wait time. It's super stable and efficient. And I'm running it on a dirt cheap, tiny node in the cloud, where before I needed two small EC2 instances to keep up with peak loads. Also, now that I've gotten the hang of it I think it's actually more efficient with regards to development than either Django or Rails. Here are the google crawler response times directly before/after the switch to Phoenix: http://imgur.com/a/FASyJ http://imgur.com/a/FASyJ
- pjungwir 9y ago
  You and blatyo are convincing me. :-) I really appreciate your openness in sharing these stats. I don't need to serve 2 million concurrent websockets. But I need to render HTML/JSON based on some database queries, under moderate traffic, and I'd like to get it under 200ms without trying so hard. I mostly agree with DHH that response time is dominated by database queries and Rails is "fast enough". Most problems you can fix by improving your query patterns or doing some SQL tuning. And yet . . . Rails is still awfully slow! Most companies don't have time for Russian-doll caching, let alone making the designers consider that at design time. I would love to use something as effortless as Rails that still gave me snappy performance well into the app's maturity. I'm fairly confident Elixir can do that, but it's hard to look at these benchmarks and come on HN and hear the Elixir folks saying, "Just trust me." There is a strong temptation to walk away thinking it just doesn't live up to the hype. So having some hard numbers is a great reassurance. Thank you!
  - blatyo 9y ago
    One note on database time. If you're loading multiple related records, Ecto is able to do some of those requests concurrently. For example, if you get a user and their posts and comments, the fetch of user happens, and then posts and comments can be fetched concurrently. Also, when I used rails, quite a bit of time was spent wrapping the data into active record objects, whereas Ecto returns simple data structures, which seems to be much faster.
Scarbutt 9y ago
For single a web api server Elixir is going to be far behind, performance wise, against nodejs, the JVM and Go. Their big feature is reliability/clustering but in the context of web requests, scaling them is real easy today, just use a load balancer. For me Erlang/Elixir have very specific niche use cases(at which it excels at) but its fans confuse beginners promoting it as a "general purpose language" that they should use it for everything.
- OvermindDL1 9y ago
  > For single a web api server Elixir is going to be far behind, performance wise, against nodejs, the JVM and Go. To some Go web api frameworks perhaps. To the JVM, not really. The frameworks I used with it in the past (spring/play/coupleOtherSmallOnes) are throttled to the number of threads for concurrent connections. The BEAM (Erlang/Elixir) has no such limitation and it blew away my old Java servers handily just due to its concurrency alone. And not to nodejs at all. Running a single BEAM vs a single node there is no comparison, the BEAM runs circles around it. With nodejs you can shard it out to multiple running instances to get concurrency better but you run into OS limits far before the workload that the BEAM can handle.
  - always_good 9y ago
    I have a hard time believing it's faster than a JVM solution, especially since you think Go is categorically faster than the JVM. For example, Jetty will beat every HTTP hello-world you can write in Go. I know because I've tried to beat Jetty in Go.
    - blatyo 9y ago
      I believe what Overmind meant was that a single request in another language can probably beat Elixir in performance, but Elixirs ability to handle a large amount of requests concurrently means that it'll have better throughput. For example, if a web request in framework A takes 100ms and framework B takes 200ms, but framework B can handle 4,000 requests concurrently and framework A can handle 1,000 requests concurrently, then framework B will beat framework A in handling 20,000 requests. framework A - (20,000r / 1,000r) * 100ms = 2,000ms framework B - (20,000r / 4,000r) * 200ms = 1,000ms

blatyo 9y ago

Here's a screenshot of mean times for one of my company's app. http://take.ms/qGVOA http://take.ms/qGVOA The one at the top is proxying the request to a java app which happens to be a little slow.

chrismccord 9y ago

Phoenix creator here. The techmeme results are not representative of the framework. Several months ago they added Phoenix in a preview, but it was a very poor implementation. They were testing JSON benchmarks through the :browser pipeline, complete with crsf token generation. They had a dev DB pool size of 10, where other frameworks were given of pool size of 100. And they also had heavy IO logging, where other frameworks did no logging. We sent a PR to address these issues, and I was hoping to see true results in the subsequent runs, but no preview was provided this time and we weren't able to work with them on the error rate issues they ran into. After these experiences we've decided the core-team's time is better spent on the framework than chasing techmeme issues. If you go by real-world success stories from companies switching from Django, Rails, etc, you can certainly expect much better.

alberth 9y ago
Awesome response. Maybe to put the naysayers to rest you guys can update http://www.phoenixframework.org/blog/the-road-to-2-million-websocket-connections http://www.phoenixframework.org/blog/the-road-to-2-million-w...
pjungwir 9y ago
Okay, thanks for the reply. I asked about this benchmark once before (almost a year ago I expect), and was told roughly the same, so I was hoping things had improved by now. I understand it's just a benchmark and you have other things to do, but it does matter to me and I expect others. None of us want to invest a lot of time becoming experts in a new world and then not gain what we hoped. Personally these benchmarks make me think if I'm going to trade dev time for performance I should just use Rocket. That's probably not a reasonable position though. :-) Anyway, thank you for Phoenix, and I will look for opportunities to give it another spin.
awj 9y ago
> The techmeme results are not representative of the framework. I've seen several instances of this, now. Some Rails benchmarks jumped up pretty high as well when they managed to get someone to competently configure Puma. Practically the same story, just changing out names. Take techempower benchmarks with a boatload of salt. Not only are they not particularly representative of your real world use cases, they apparently are often not representative of competent use of the tools they're benchmarking. Hopefully this is an issue the techempower people figure out some way to address. It would be nice if this was a more trustworthy resource.

deleted 9y ago

[deleted]

udfalkso 9y ago

I recently rewrote an (admittedly old and stale) django app from the ground up with Elixir/Phoenix. The website gets quite a bit of traffic. I went from an elaborate multi-tier caching setup, with varnish and memcache, to a ZERO caching setup. The amount of complexity reduced by doing this is huge. The Elixir app hits the postgres db for nearly every request and I'm getting average response times around 45ms. Quite a bit of that is database wait time. It's super stable and efficient. And I'm running it on a dirt cheap, tiny node in the cloud, where before I needed two small EC2 instances to keep up with peak loads. Also, now that I've gotten the hang of it I think it's actually more efficient with regards to development than either Django or Rails. Here are the google crawler response times directly before/after the switch to Phoenix: http://imgur.com/a/FASyJ http://imgur.com/a/FASyJ

pjungwir 9y ago
You and blatyo are convincing me. :-) I really appreciate your openness in sharing these stats. I don't need to serve 2 million concurrent websockets. But I need to render HTML/JSON based on some database queries, under moderate traffic, and I'd like to get it under 200ms without trying so hard. I mostly agree with DHH that response time is dominated by database queries and Rails is "fast enough". Most problems you can fix by improving your query patterns or doing some SQL tuning. And yet . . . Rails is still awfully slow! Most companies don't have time for Russian-doll caching, let alone making the designers consider that at design time. I would love to use something as effortless as Rails that still gave me snappy performance well into the app's maturity. I'm fairly confident Elixir can do that, but it's hard to look at these benchmarks and come on HN and hear the Elixir folks saying, "Just trust me." There is a strong temptation to walk away thinking it just doesn't live up to the hype. So having some hard numbers is a great reassurance. Thank you!
- blatyo 9y ago
  One note on database time. If you're loading multiple related records, Ecto is able to do some of those requests concurrently. For example, if you get a user and their posts and comments, the fetch of user happens, and then posts and comments can be fetched concurrently. Also, when I used rails, quite a bit of time was spent wrapping the data into active record objects, whereas Ecto returns simple data structures, which seems to be much faster.

Scarbutt 9y ago

For single a web api server Elixir is going to be far behind, performance wise, against nodejs, the JVM and Go. Their big feature is reliability/clustering but in the context of web requests, scaling them is real easy today, just use a load balancer. For me Erlang/Elixir have very specific niche use cases(at which it excels at) but its fans confuse beginners promoting it as a "general purpose language" that they should use it for everything.

OvermindDL1 9y ago
> For single a web api server Elixir is going to be far behind, performance wise, against nodejs, the JVM and Go. To some Go web api frameworks perhaps. To the JVM, not really. The frameworks I used with it in the past (spring/play/coupleOtherSmallOnes) are throttled to the number of threads for concurrent connections. The BEAM (Erlang/Elixir) has no such limitation and it blew away my old Java servers handily just due to its concurrency alone. And not to nodejs at all. Running a single BEAM vs a single node there is no comparison, the BEAM runs circles around it. With nodejs you can shard it out to multiple running instances to get concurrency better but you run into OS limits far before the workload that the BEAM can handle.
- always_good 9y ago
  I have a hard time believing it's faster than a JVM solution, especially since you think Go is categorically faster than the JVM. For example, Jetty will beat every HTTP hello-world you can write in Go. I know because I've tried to beat Jetty in Go.
  - blatyo 9y ago
    I believe what Overmind meant was that a single request in another language can probably beat Elixir in performance, but Elixirs ability to handle a large amount of requests concurrently means that it'll have better throughput. For example, if a web request in framework A takes 100ms and framework B takes 200ms, but framework B can handle 4,000 requests concurrently and framework A can handle 1,000 requests concurrently, then framework B will beat framework A in handling 20,000 requests. framework A - (20,000r / 1,000r) * 100ms = 2,000ms framework B - (20,000r / 4,000r) * 200ms = 1,000ms