5 ms·

Benchmarking Phoenix vs. Rails vs. Sinatra vs. Express vs. Martini

79 points by rebelidealist 12y ago

ryanong 12y ago
I would love to see openresty in there too.
alinenache 12y ago
http://waltherpragerandphilosophy1.blogspot.ro/2013/02/technical-illusion.html http://waltherpragerandphilosophy1.blogspot.ro/2013/02/techn...
vruiz 12y ago
For the ones surprised by Go's performance, remember that Martini in known to be slow as it uses heavily reflexion.
patrickaljord 12y ago
Martini is slow, try again with Negroni which is the new lib by the author of Martini.
- SwellJoe 12y ago
  Aren't they quite different creatures, though? It's clear that language is not the only determining factor, since Sinatra is so much faster than Rails, and the Node example is so much faster in a cluster configuration. If the experience of programming for Sinatra, Phoenix, and Martini are very similar, it's fair to compare them (though maybe benchmarks may not be super useful, given that the database layer is likely to be the bottleneck in most large site deployments rather than simple routing and other stuff being compared here). If Negroni is very different (I don't know that it is, but a brief perusal of the site for Martini and the github for Negroni indicates very different experiences) then it's possibly no more useful a comparison than Rails to Phoenix.
  - nkozyra 12y ago
    Plus isn't Negroni made to specifically be more about middleware than a fully-featured framework? If you say use Negroni, you might as well say use net/http (which is fair enough, since it's a low-level framework in itself).
- Rapzid 12y ago
  So, profiled the Martini bench and discovered it was doing a LOT of file operations. I looked in the code and found that it does that in dev mode. The benchmark sets the mode to "production". D'oh! Ok, so I profiled it again and guess what? Martini is STILL doing a lot of file stuff. It's in net/http.Dir.Open and I believe it has to do with how the middleware is setup. You can see a TON of time spent in runtime.cgocall which I believe is it waiting for a syscall to come back. Don't have time to look into it further today but I'd imagine clearing that up could possibly double the performance.
jamieomatthews 12y ago
Martini is not a good reflection on Go's performance (no pun intended). If you want a Go benchmark of a minimalistic framework, use Gorilla mux or even Beego
- anonfunction 12y ago
  Also check out Gin which is very lightweight and performant with a great API. https://github.com/gin-gonic/gin https://github.com/gin-gonic/gin
HashanP 12y ago
I wonder how koa would fare.
- babby 12y ago
  Probably the same if it uses EJS. That's really what is being benchmarked here, which is confusing, as EJS is one of the slower templating languages.
- jtblin 12y ago
  Probably a bit slower than the express example as ES6 generators cannot be optimised the same way than callbacks can.
deleted 12y ago
[deleted]
imanaccount247 12y ago
Is elixir just a different syntax for erlang, or are the semantic differences? Their website doesn't seem to make it very clear.
- erikj54 12y ago
  Elixir is to Erlang what Scala is to Java. They leverage the Erlang VM in the same way that Scala leverages the JVM.
  - jeremyjh 12y ago
    Actually it is not a good comparison. scalac compiles directly to byte code that runs on the JVM. Scala has pretty different semantics from Java. Whereas Elixir is compiled to the Erlang AST - a pretty high-level data structure - and then the Erlang compiler is invoked on this structure to compile byte code for BEAM. So whereas Scala can do things Java can't do (such as closures on Java 6), there really is nothing Elixir can do that Erlang can't. The fact that Elixir has the same semantics is really a good thing, because it makes it much easier to learn both languages at the same time and that is still pretty much essential in my opinion, if you want to become productive in Elixir.
- angersock 12y ago
  Elixir makes a lot of the syntax of Erlang (derived from Prolog) simpler and more familiar to people working from, say, a Ruby background. It adds better support for UTF8, better metaprogramming tools for declaring custom constructs, a standard library for things like collections and streams, proper default setup for some use of OTP patterns, and various other things. It adds a few sane default conventions for things like documentation, testing, and private/public visibility in modules. It in turn transpiles to Erlang, so has all the same performance characteristics.
  - disbelief 12y ago
    Is it therefore possible to use Erlang libraries in an Elixir application, or to "drop to Erlang" in places where it could be advantageous?
nmerouze 12y ago
Gin, a Go web framework similar to Martini, would probably performs 10 times better. Since the only purpose of using Martini is because it has a similar style than Phoenix, I would suggest changing the framework to Gin or another Go framework since Martini is not the only framework available.
pmontra 12y ago
Elixir is Erlang with some Ruby-like syntactical sugar that vanishes when you look very close at it but makes the transition very easy. It's not like moving to Erlang or Haskell or OCaml coming from procedural languages. I have experience with Sinatra and Rails and I did a small project with Elixir and Phoenix in September. My bet before looking at the benchmarks was Phoenix >> Sinatra > Ruby, and that's the way it turned out to be. I was curious about how Martini and Node would compare. That said, I enjoyed developing with Phoenix and it's really similar to Rails in its structure. The differences are about having views+templates (decorator pattern) instead of Rails' plain views (which would be Phoenix's templates) and having first class handlers for websockets. The developers of Phoenix answered quickly to all my questions and to the issues I opened on GitHub (it's still 0.x software in heavy development). Anyway I still can't use it to replace Rails in the projects I do for my customers. There many reasons for that, some about Phoenix and some about me. 1) There aren't enough developers and it won't be wise for a customer to become too dependent from me (or any other developer) 2) There isn't an ecosystem comparable to the Rails one yet. Example: it can take forever to develop a replacement for all the basic functionality a Rails developer gets for free from a gem like devise (everything that revolves around authentication). 3) There might be many useful tools from the Erlang world but it takes time to learn them and become decently productive with Erlang and Elixir. How to make decent money while going back so many years productivity-wise? Obviously this applies to any transition to any ecosystem. 4) Performances usually don't matter at all: most servers have to sustain only low loads and Rails is good enough for that. In 20 years I probably had to work with only two web applications were speed really mattered. One was done with Java, the other one was PHP+Java and it was constrained more by the db than by the frontend. However I would consider Elixir right now for any heavily concurrent server, let's say a game server with thousands of players.
jbhatab 12y ago
I feel like these benchmarks always put rails in a tough spot. Let's choose bare bones frameworks and compare it to a fully loaded framework. Oh wow, the bare bones ones are faster?! I'd be concerned for them if they weren't.
- jtblin 12y ago
  Elixir's goal is _not_ to be a micro framework. Perhaps it does less than Rails now but its philosophy seems to be more similar to Rails than Sinatra, see talk from the creator http://www.confreaks.com/videos/4132-elixirconf2014-rise-of-the-phoenix-building-an-elixir-web-framework http://www.confreaks.com/videos/4132-elixirconf2014-rise-of-....
  - angersock 12y ago
    And we're already seeing people mixing up Elixir with Phoenix...just like Ruby and Rails. Good sign. :)
deleted 12y ago
[deleted]
rdtsc 12y ago
> Latency (ms): 4.54 Consistency (σ ms): 1.75 That is not to be underestimated. Many frameworks will provide better throughput but consistent low latency is harder to do and Erlang's BEAM VM does it. It is really a marvel of engineering and has been battle tested for decades.
- chrismccord 12y ago
  Yep. This is a very important metric to keep in mind. It gives you a sense of what you might expect under heavy load.
touristtam 12y ago
reminds me of http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/

79 points by rebelidealist 12y ago

I would love to see openresty in there too.

http://waltherpragerandphilosophy1.blogspot.ro/2013/02/technical-illusion.html http://waltherpragerandphilosophy1.blogspot.ro/2013/02/techn...

vruiz 12y ago

For the ones surprised by Go's performance, remember that Martini in known to be slow as it uses heavily reflexion.

patrickaljord 12y ago

Martini is slow, try again with Negroni which is the new lib by the author of Martini.

SwellJoe 12y ago
Aren't they quite different creatures, though? It's clear that language is not the only determining factor, since Sinatra is so much faster than Rails, and the Node example is so much faster in a cluster configuration. If the experience of programming for Sinatra, Phoenix, and Martini are very similar, it's fair to compare them (though maybe benchmarks may not be super useful, given that the database layer is likely to be the bottleneck in most large site deployments rather than simple routing and other stuff being compared here). If Negroni is very different (I don't know that it is, but a brief perusal of the site for Martini and the github for Negroni indicates very different experiences) then it's possibly no more useful a comparison than Rails to Phoenix.
- nkozyra 12y ago
  Plus isn't Negroni made to specifically be more about middleware than a fully-featured framework? If you say use Negroni, you might as well say use net/http (which is fair enough, since it's a low-level framework in itself).
Rapzid 12y ago
So, profiled the Martini bench and discovered it was doing a LOT of file operations. I looked in the code and found that it does that in dev mode. The benchmark sets the mode to "production". D'oh! Ok, so I profiled it again and guess what? Martini is STILL doing a lot of file stuff. It's in net/http.Dir.Open and I believe it has to do with how the middleware is setup. You can see a TON of time spent in runtime.cgocall which I believe is it waiting for a syscall to come back. Don't have time to look into it further today but I'd imagine clearing that up could possibly double the performance.

jamieomatthews 12y ago

Martini is not a good reflection on Go's performance (no pun intended). If you want a Go benchmark of a minimalistic framework, use Gorilla mux or even Beego

anonfunction 12y ago
Also check out Gin which is very lightweight and performant with a great API. https://github.com/gin-gonic/gin https://github.com/gin-gonic/gin

HashanP 12y ago

I wonder how koa would fare.

babby 12y ago
Probably the same if it uses EJS. That's really what is being benchmarked here, which is confusing, as EJS is one of the slower templating languages.
jtblin 12y ago
Probably a bit slower than the express example as ES6 generators cannot be optimised the same way than callbacks can.

deleted 12y ago

[deleted]

imanaccount247 12y ago

Is elixir just a different syntax for erlang, or are the semantic differences? Their website doesn't seem to make it very clear.

erikj54 12y ago
Elixir is to Erlang what Scala is to Java. They leverage the Erlang VM in the same way that Scala leverages the JVM.
- jeremyjh 12y ago
  Actually it is not a good comparison. scalac compiles directly to byte code that runs on the JVM. Scala has pretty different semantics from Java. Whereas Elixir is compiled to the Erlang AST - a pretty high-level data structure - and then the Erlang compiler is invoked on this structure to compile byte code for BEAM. So whereas Scala can do things Java can't do (such as closures on Java 6), there really is nothing Elixir can do that Erlang can't. The fact that Elixir has the same semantics is really a good thing, because it makes it much easier to learn both languages at the same time and that is still pretty much essential in my opinion, if you want to become productive in Elixir.
angersock 12y ago
Elixir makes a lot of the syntax of Erlang (derived from Prolog) simpler and more familiar to people working from, say, a Ruby background. It adds better support for UTF8, better metaprogramming tools for declaring custom constructs, a standard library for things like collections and streams, proper default setup for some use of OTP patterns, and various other things. It adds a few sane default conventions for things like documentation, testing, and private/public visibility in modules. It in turn transpiles to Erlang, so has all the same performance characteristics.
- disbelief 12y ago
  Is it therefore possible to use Erlang libraries in an Elixir application, or to "drop to Erlang" in places where it could be advantageous?

nmerouze 12y ago

Gin, a Go web framework similar to Martini, would probably performs 10 times better. Since the only purpose of using Martini is because it has a similar style than Phoenix, I would suggest changing the framework to Gin or another Go framework since Martini is not the only framework available.

pmontra 12y ago

Elixir is Erlang with some Ruby-like syntactical sugar that vanishes when you look very close at it but makes the transition very easy. It's not like moving to Erlang or Haskell or OCaml coming from procedural languages. I have experience with Sinatra and Rails and I did a small project with Elixir and Phoenix in September. My bet before looking at the benchmarks was Phoenix >> Sinatra > Ruby, and that's the way it turned out to be. I was curious about how Martini and Node would compare. That said, I enjoyed developing with Phoenix and it's really similar to Rails in its structure. The differences are about having views+templates (decorator pattern) instead of Rails' plain views (which would be Phoenix's templates) and having first class handlers for websockets. The developers of Phoenix answered quickly to all my questions and to the issues I opened on GitHub (it's still 0.x software in heavy development). Anyway I still can't use it to replace Rails in the projects I do for my customers. There many reasons for that, some about Phoenix and some about me. 1) There aren't enough developers and it won't be wise for a customer to become too dependent from me (or any other developer) 2) There isn't an ecosystem comparable to the Rails one yet. Example: it can take forever to develop a replacement for all the basic functionality a Rails developer gets for free from a gem like devise (everything that revolves around authentication). 3) There might be many useful tools from the Erlang world but it takes time to learn them and become decently productive with Erlang and Elixir. How to make decent money while going back so many years productivity-wise? Obviously this applies to any transition to any ecosystem. 4) Performances usually don't matter at all: most servers have to sustain only low loads and Rails is good enough for that. In 20 years I probably had to work with only two web applications were speed really mattered. One was done with Java, the other one was PHP+Java and it was constrained more by the db than by the frontend. However I would consider Elixir right now for any heavily concurrent server, let's say a game server with thousands of players.

jbhatab 12y ago

I feel like these benchmarks always put rails in a tough spot. Let's choose bare bones frameworks and compare it to a fully loaded framework. Oh wow, the bare bones ones are faster?! I'd be concerned for them if they weren't.

jtblin 12y ago
Elixir's goal is _not_ to be a micro framework. Perhaps it does less than Rails now but its philosophy seems to be more similar to Rails than Sinatra, see talk from the creator http://www.confreaks.com/videos/4132-elixirconf2014-rise-of-the-phoenix-building-an-elixir-web-framework http://www.confreaks.com/videos/4132-elixirconf2014-rise-of-....
- angersock 12y ago
  And we're already seeing people mixing up Elixir with Phoenix...just like Ruby and Rails. Good sign. :)

deleted 12y ago

[deleted]

rdtsc 12y ago

> Latency (ms): 4.54 Consistency (σ ms): 1.75 That is not to be underestimated. Many frameworks will provide better throughput but consistent low latency is harder to do and Erlang's BEAM VM does it. It is really a marvel of engineering and has been battle tested for decades.

chrismccord 12y ago
Yep. This is a very important metric to keep in mind. It gives you a sense of what you might expect under heavy load.

touristtam 12y ago

reminds me of http://www.techempower.com/benchmarks/ http://www.techempower.com/benchmarks/