5 ms·

A benchmark comparison of Node.js, Tornado, and 3 Erlang servers

95 points by EvanMiller 15y ago

megaman821 15y ago
I would be interested on how the Haskell frameworks like Warp or Snap stand up to Erlang.
- hendzen 15y ago
  Yesod/Warp seems to be incredibly fast according to a benchmark conducted by the Yesod devs. [1] I would also like to see a comparison with Erlang. [1] - http://www.yesodweb.com/blog/preliminary-warp-cross-language-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language...
reddittor 15y ago
Breaking news: Author of web framework writes benchmark to prove his web framework is the fastest.
- j2d2j2d2 15y ago
  All three Erlang frameworks beat the pants off node.js. Not just his.
  - j2d2j2d2 15y ago
    I should have clarified to say that the Erlang frameworks beat both Node and Tornado thoroughly.
EvanMiller 15y ago
The author emphasizes the first picture (which shows Misultin delivering 3X the throughput of a Node.js server), but I think the most interesting picture is the Response Times (picture #3). Look at the purple and yellow lines. When Node.js starts overloading at 3000 reqs/sec, the average response time drops to ~300 ms (just guessing, I'm not good at reading log graphs). But when Mochiweb starts overloading at 5000 reqs/sec, the average response time is around ~50ms. In other words, the slowest of the Erlang servers has twice the throughput of Node.js, and handles server overload much more gracefully (at least in this contrived example).
azakai 15y ago
I'm not sure this is surprising. People use Python and JS-based webservers not for their performance, but because they like the languages. And those languages were designed not for speed or concurrency but for other things, like expresiveness. Erlang was designed with performance and concurrency in mind, and if it didn't beat the pants off of Python and JS servers, I'd be shocked. Not that that means it's the right choice for all things. Performance isn't everything, being able to write code in Python is a good enough reason to use something like Django.
- cbf 15y ago
  Erlang was not designed with performance in mind. It's only gotten reasonably performant in it's later life.
  - dons 15y ago
    Indeed. I'd really like to see the Haskell servers, warp and snap, thrown in, since the infrastructure there was actually built for raw performance.
    - ozataman 15y ago
      I have put together the code to implement the benchmark using Snap and would be happy to share with anyone willing to boot up the instance and run. I might do so myself if I find the time tomorrow.
  - regularfry 15y ago
    That's not quite fair. Erlang was designed from the start to be very, very fast at process switching and message passing. True, that design took a while to bear fruit, but my understanding (from listening to Joe Armstrong speak about it) is that being able to make the core of the language highly efficient was a goal from the start.
    - forensic 15y ago
      Trying to make it as efficient as possible is different from the purpose of the language being efficiency.
cbf 15y ago
I found Steve Vinoski's comment on this exercise interesting: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/ http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-b...
ericflo 15y ago
The author claims that he doesn't want to test an overly-simple "hello world" benchmark, and then he does exactly that.
- rfrey 15y ago
  The author says what he means by an overly-simple "hello world" is a test of retrieval of static resources. His test is very simple but it does require a dynamic response from the server.
kingkilr 15y ago
Comment I left on reddit, with my benchmark numbers for the tornado server under PyPy: http://www.reddit.com/r/programming/comments/h7cs9/cowboy_misultin_mochiweb_tornado_and_nodejs/c1t6pl4 http://www.reddit.com/r/programming/comments/h7cs9/cowboy_mi...
fingerprinter 15y ago
This should just fall under the 'duh' category. Quite simply, Erlang is the gold standard for scalability (horizontal and vertical) and nearly everything else is far #2. However, choosing a tech is much more than just raw numbers. There are so many factors at play and it shouldn't be taken lightly. That being said, we did something similar about 2 years ago and Erlang/Riak came out so far ahead of the competition for our needs and it wasn't even close. I did go back to my notes just recently and looked again with Node in mind. Though I didn't do the same bake off we did back then, I did a thorough exercise with Node to make an evaluation and, while a cool tech, would not have made the cut. Erlang/OTP w/ Riak is not hard to program (anyone can learn), takes so much of the guess work out of scalability and is easy to debug. Node isn't quite mature enough to handle this yet and coupled with the true lack of multi-core, can't quite hang with the big boys just yet. Erlang/OTP is clearly in a league of its own.
- ozataman 15y ago
  Gold standard for scalability? There are certainly others. Check out numbers for Haskell: http://snapframework.com/blog/2010/11/17/snap-0.3-benchmarks http://snapframework.com/blog/2010/11/17/snap-0.3-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language... Would be interesting to repeat the author's experiment exactly, but in any case I would expect an order of magnitude upwards difference in performance. Haskell gives you non-blocking IO without the hassle of callbacks, multithreaded operation and more.
  - justincormack 15y ago
    I wouldnt expect an order of magnitude better on the same benchmark actually, especially as the benchmark in question is for a single core. There is clearly some scope to improve over these, but I doubt it is a factor of 10. Run the benchmarks rather than making claims with no backup.
c00p3r 15y ago
Good reality check for Node's fanboys. There are literally no advantages in Node, except it is cool and it is Javascript. It is like web-server written in PHP just because we can and php is so "widely used" language.
argus 15y ago
Is it me or is this not a really fair comparison since the erlang frameworks would be utilizing both cpu's while node and tornado wouldn't be? In real life a load balancer in front of tornado or node would be a far better solution than going down the erlang road in production.. support and maintenance wise + more programmers out there knowing the language would win hands down IMO...
- evgen 15y ago
  If you read the article closely you will notice that the Erlang VMs were started without SMP support enabled; everything was single-core. If you turned on Erlang SMP support then the comparison would be even more lop-sided than the example presented in the OP. In real life a hardware load balancer in front of a pool of Erlang servers would be even better :)
tomaszkubacki 15y ago
would be interesting hoe this compare to manos de mono https://github.com/jacksonh/manos https://github.com/jacksonh/manos
tomaszkubacki 15y ago
would be interesting how this compare to manos de mono https://github.com/jacksonh/manos https://github.com/jacksonh/manos
bergie 15y ago
Would be interesting to add AppServer-in-PHP: https://github.com/indeyets/appserver-in-php https://github.com/indeyets/appserver-in-php It performed quite equally to Node.js on my small-scale tests (http://bergie.iki.fi/blog/php_can_perform_better_than_node-js/ http://bergie.iki.fi/blog/php_can_perform_better_than_node-j...)
muyuu 15y ago
By using couch-DB you can have Erland I/O scalability and then use whatever you want for the rest of the server side logic. If you have such a massive success that you need the rest of it also ported, then you can make this investment later on. The rest of us simply don't have the time to learn whatever is the fastest each month.
- tmcneal 15y ago
  The fact that CouchDB is written in Erlang is only part of the equation. You're not considering that CouchDB views are written in Javascript or that CouchDB uses HTTP as its connection protocol, which has more overhead than the typical binary protocol of traditional DBMSes. Also, avg. response time in CouchDB is directly proportional to the number of view invalidations that occur. If your data is changing a lot your views will be invalidated frequently and for those requests avg response time will drop significantly. Throw in a reduce function and re-indexing can become an order of magnitude (or more) slower.
antihero 15y ago
What sort of datastore would be a good pairing for this? Seeing as it doesn't matter how fast a server is if the datastore can't keep up. Redis? Riak? CouchDB?
st3fan 15y ago
Two comments. These microbenchmarks don't really do it for me. It would be much more interesting to see a real test application being benchmarked. Something with users, objects, pages that have content. It would be more fair to look at the code, complexity and performance drops when you actually build something more real. Also, I know Erlang is awesome and super fast. But personally I would rather spend the money on a good horizontally scalable architecture and extra hardware if that means we can build apps in a more mainstream language like for example Python or Ruby.

95 points by EvanMiller 15y ago

megaman821 15y ago

I would be interested on how the Haskell frameworks like Warp or Snap stand up to Erlang.

hendzen 15y ago
Yesod/Warp seems to be incredibly fast according to a benchmark conducted by the Yesod devs. [1] I would also like to see a comparison with Erlang. [1] - http://www.yesodweb.com/blog/preliminary-warp-cross-language-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language...

reddittor 15y ago

Breaking news: Author of web framework writes benchmark to prove his web framework is the fastest.

j2d2j2d2 15y ago
All three Erlang frameworks beat the pants off node.js. Not just his.
- j2d2j2d2 15y ago
  I should have clarified to say that the Erlang frameworks beat both Node and Tornado thoroughly.

EvanMiller 15y ago

The author emphasizes the first picture (which shows Misultin delivering 3X the throughput of a Node.js server), but I think the most interesting picture is the Response Times (picture #3). Look at the purple and yellow lines. When Node.js starts overloading at 3000 reqs/sec, the average response time drops to ~300 ms (just guessing, I'm not good at reading log graphs). But when Mochiweb starts overloading at 5000 reqs/sec, the average response time is around ~50ms. In other words, the slowest of the Erlang servers has twice the throughput of Node.js, and handles server overload much more gracefully (at least in this contrived example).

azakai 15y ago

I'm not sure this is surprising. People use Python and JS-based webservers not for their performance, but because they like the languages. And those languages were designed not for speed or concurrency but for other things, like expresiveness. Erlang was designed with performance and concurrency in mind, and if it didn't beat the pants off of Python and JS servers, I'd be shocked. Not that that means it's the right choice for all things. Performance isn't everything, being able to write code in Python is a good enough reason to use something like Django.

cbf 15y ago
Erlang was not designed with performance in mind. It's only gotten reasonably performant in it's later life.
- dons 15y ago
  Indeed. I'd really like to see the Haskell servers, warp and snap, thrown in, since the infrastructure there was actually built for raw performance.
  - ozataman 15y ago
    I have put together the code to implement the benchmark using Snap and would be happy to share with anyone willing to boot up the instance and run. I might do so myself if I find the time tomorrow.
- regularfry 15y ago
  That's not quite fair. Erlang was designed from the start to be very, very fast at process switching and message passing. True, that design took a while to bear fruit, but my understanding (from listening to Joe Armstrong speak about it) is that being able to make the core of the language highly efficient was a goal from the start.
  - forensic 15y ago
    Trying to make it as efficient as possible is different from the purpose of the language being efficiency.

cbf 15y ago

I found Steve Vinoski's comment on this exercise interesting: http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-benchmarking/ http://steve.vinoski.net/blog/2011/05/09/erlang-web-server-b...

ericflo 15y ago

The author claims that he doesn't want to test an overly-simple "hello world" benchmark, and then he does exactly that.

rfrey 15y ago
The author says what he means by an overly-simple "hello world" is a test of retrieval of static resources. His test is very simple but it does require a dynamic response from the server.

kingkilr 15y ago

Comment I left on reddit, with my benchmark numbers for the tornado server under PyPy: http://www.reddit.com/r/programming/comments/h7cs9/cowboy_misultin_mochiweb_tornado_and_nodejs/c1t6pl4 http://www.reddit.com/r/programming/comments/h7cs9/cowboy_mi...

fingerprinter 15y ago

This should just fall under the 'duh' category. Quite simply, Erlang is the gold standard for scalability (horizontal and vertical) and nearly everything else is far #2. However, choosing a tech is much more than just raw numbers. There are so many factors at play and it shouldn't be taken lightly. That being said, we did something similar about 2 years ago and Erlang/Riak came out so far ahead of the competition for our needs and it wasn't even close. I did go back to my notes just recently and looked again with Node in mind. Though I didn't do the same bake off we did back then, I did a thorough exercise with Node to make an evaluation and, while a cool tech, would not have made the cut. Erlang/OTP w/ Riak is not hard to program (anyone can learn), takes so much of the guess work out of scalability and is easy to debug. Node isn't quite mature enough to handle this yet and coupled with the true lack of multi-core, can't quite hang with the big boys just yet. Erlang/OTP is clearly in a league of its own.

ozataman 15y ago
Gold standard for scalability? There are certainly others. Check out numbers for Haskell: http://snapframework.com/blog/2010/11/17/snap-0.3-benchmarks http://snapframework.com/blog/2010/11/17/snap-0.3-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language-benchmarks http://www.yesodweb.com/blog/preliminary-warp-cross-language... Would be interesting to repeat the author's experiment exactly, but in any case I would expect an order of magnitude upwards difference in performance. Haskell gives you non-blocking IO without the hassle of callbacks, multithreaded operation and more.
- justincormack 15y ago
  I wouldnt expect an order of magnitude better on the same benchmark actually, especially as the benchmark in question is for a single core. There is clearly some scope to improve over these, but I doubt it is a factor of 10. Run the benchmarks rather than making claims with no backup.

c00p3r 15y ago

Good reality check for Node's fanboys. There are literally no advantages in Node, except it is cool and it is Javascript. It is like web-server written in PHP just because we can and php is so "widely used" language.

argus 15y ago

Is it me or is this not a really fair comparison since the erlang frameworks would be utilizing both cpu's while node and tornado wouldn't be? In real life a load balancer in front of tornado or node would be a far better solution than going down the erlang road in production.. support and maintenance wise + more programmers out there knowing the language would win hands down IMO...

evgen 15y ago
If you read the article closely you will notice that the Erlang VMs were started without SMP support enabled; everything was single-core. If you turned on Erlang SMP support then the comparison would be even more lop-sided than the example presented in the OP. In real life a hardware load balancer in front of a pool of Erlang servers would be even better :)

tomaszkubacki 15y ago

would be interesting hoe this compare to manos de mono https://github.com/jacksonh/manos https://github.com/jacksonh/manos

tomaszkubacki 15y ago

would be interesting how this compare to manos de mono https://github.com/jacksonh/manos https://github.com/jacksonh/manos

bergie 15y ago

Would be interesting to add AppServer-in-PHP: https://github.com/indeyets/appserver-in-php https://github.com/indeyets/appserver-in-php It performed quite equally to Node.js on my small-scale tests (http://bergie.iki.fi/blog/php_can_perform_better_than_node-js/ http://bergie.iki.fi/blog/php_can_perform_better_than_node-j...)

muyuu 15y ago

By using couch-DB you can have Erland I/O scalability and then use whatever you want for the rest of the server side logic. If you have such a massive success that you need the rest of it also ported, then you can make this investment later on. The rest of us simply don't have the time to learn whatever is the fastest each month.

tmcneal 15y ago
The fact that CouchDB is written in Erlang is only part of the equation. You're not considering that CouchDB views are written in Javascript or that CouchDB uses HTTP as its connection protocol, which has more overhead than the typical binary protocol of traditional DBMSes. Also, avg. response time in CouchDB is directly proportional to the number of view invalidations that occur. If your data is changing a lot your views will be invalidated frequently and for those requests avg response time will drop significantly. Throw in a reduce function and re-indexing can become an order of magnitude (or more) slower.

antihero 15y ago

What sort of datastore would be a good pairing for this? Seeing as it doesn't matter how fast a server is if the datastore can't keep up. Redis? Riak? CouchDB?

st3fan 15y ago

Two comments. These microbenchmarks don't really do it for me. It would be much more interesting to see a real test application being benchmarked. Something with users, objects, pages that have content. It would be more fair to look at the code, complexity and performance drops when you actually build something more real. Also, I know Erlang is awesome and super fast. But personally I would rather spend the money on a good horizontally scalable architecture and extra hardware if that means we can build apps in a more mainstream language like for example Python or Ruby.