5 ms·

Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted vs. Unicorn

53 points by acangiano 17y ago

viraptor 17y ago
It would be interesting if Twisted people actually looked at the problem and tried to tune their framework to match the performance... Since someone from that team (afaik) already argued that Twisted can be just as fast, they should be inclined to prove it.
xal 17y ago
Testing against unicorn makes no sense whatsoever. Test against eventmachine-httpserver, that's similar in scope and featureset.
- acangiano 17y ago
  I agree that it could have been left out as it's a different beast. But I wanted to compare Tornado and the others with one of the fastest (and increasingly popular) web server from the Ruby world. PS: I'm about to update my post with the results from Thin and evma_httpserver. PS2: My post has been updated.
  - acangiano 17y ago
    I decided to keep asynchronous, non-blocking web servers only. So I removed both Unicorn and Thin.
adamt 17y ago
Be careful about reading to much into this - ApacheBench is a very limited (if convenient) benchmark that isn't very realistic of real world load.
- boucher 17y ago
  3-4x the number of simultaneous connections (ignoring the actual performance gains) strikes me as pretty significant, even under a limited test.
  - moe 17y ago
    The test is not just limited, it's completely meaningless because the portion of the stack he tested is not going to be your bottleneck in any real application. When will people learn that 'ab' is not a suitable benchmark for application servers...
    - deleted 17y ago
      [deleted]
erlanger 17y ago
People, please stop benchmarking with ab, it does not produce accurate results. All benchmarking of this sort should be done with Siege. It's easier to use too.
- acangiano 17y ago
  Would using Siege radically change the results?
  - erlanger 17y ago
    Yep, I'd expect it to halve the requests per second for every test. Paul Jones (created the Solar framework) periodically releases some pretty authoritative benchmarks for the major PHP frameworks, and here's what he has to say on the topic: http://paul-m-jones.com/?p=421 http://paul-m-jones.com/?p=421
    - acangiano 17y ago
      Good to know for future reference.
    - dsp 17y ago
      His assertions are not confidence inspiring: http://paul-m-jones.com/?p=413#comment-410123 http://paul-m-jones.com/?p=413#comment-410123
- tlrobinson 17y ago
  While we're on the subject of benchmarking, has anyone run in to a problem on OS X (Leopard) where you seem to run out of ports, and the benchmark stalls at almost exactly 16,384 (2^16) requests? The benchmark eventually resumes 10-30 seconds later. It seems OS X can't free up closed ports fast enough. I've tried this with numerous HTTP servers (including Apache) and both ApacheBench and Siege, so it seems like it's an OS problem.
deleted 17y ago
[deleted]
fizx 17y ago
Did not expect to see EM-httpserver kicking ass.
- paul 17y ago
  I think it's a c++ library (based on scanning the docs -- I could be wrong), so it's not too surprising to see it outperforming pure Python solutions.
  - alecco 17y ago
    Indeed. http://github.com/eventmachine/eventmachine/raw/b4765960db9136197767ca028039650d75f9307d/README http://github.com/eventmachine/eventmachine/raw/b4765960db91... EventMachine consists of an extension library written in C++ (which can be accessed from languages other than Ruby) So the post compares pure Python web frameworks with a C++ library. And not even a good library based on Boost/Asio or libevent.
    - n8agrin 17y ago
      Is eventmachine's c++ lib really not a "good library"? If not, why?
      - alecco 17y ago
        It reinvents the wheel. As a developer, you have to have really good reasons not to use it. Especially since Boost license is practically AS-IS, they could have taken the original source. It is possible they are better at this problem than others, but that is very unlikely, from experience. As a developer it isn't wise to do a reliability test on a new library by yourself. Making a portable async framework is quite error-prone because of the heterogenous situations it could face. You can witness this reading libevent's evolution over the years. And those developers are as good unix coders as you can possibly get with vast experience on the subject (since mid 90s.) I bet the Boost/Asio guys are very good C++ coders too, Boost is as close as you can get to STL. At a glance, http://rubyeventmachine.com/ http://rubyeventmachine.com/ doesn't state why a new C++ async library. They only mention the Reactor Pattern, and Boost/Asio supports it: http://www.boost.org/doc/libs/1_40_0/doc/html/boost_asio/overview/core/reactor.html http://www.boost.org/doc/libs/1_40_0/doc/html/boost_asio/ove...
paul 17y ago
It would also be fun to see how the raw Tornado httpserver (http://github.com/facebook/tornado/blob/master/tornado/httpserver.py http://github.com/facebook/tornado/blob/master/tornado/https...) performs without the Tornado web.py framework on top of it.
jraines 17y ago
Is it possible to take advantage of this with any existing Ruby framework?
acangiano 17y ago
I updated my post and introduced substantial changes.

53 points by acangiano 17y ago

It would be interesting if Twisted people actually looked at the problem and tried to tune their framework to match the performance... Since someone from that team (afaik) already argued that Twisted can be just as fast, they should be inclined to prove it.

xal 17y ago

Testing against unicorn makes no sense whatsoever. Test against eventmachine-httpserver, that's similar in scope and featureset.

acangiano 17y ago
I agree that it could have been left out as it's a different beast. But I wanted to compare Tornado and the others with one of the fastest (and increasingly popular) web server from the Ruby world. PS: I'm about to update my post with the results from Thin and evma_httpserver. PS2: My post has been updated.
- acangiano 17y ago
  I decided to keep asynchronous, non-blocking web servers only. So I removed both Unicorn and Thin.

adamt 17y ago

Be careful about reading to much into this - ApacheBench is a very limited (if convenient) benchmark that isn't very realistic of real world load.

boucher 17y ago
3-4x the number of simultaneous connections (ignoring the actual performance gains) strikes me as pretty significant, even under a limited test.
- moe 17y ago
  The test is not just limited, it's completely meaningless because the portion of the stack he tested is not going to be your bottleneck in any real application. When will people learn that 'ab' is not a suitable benchmark for application servers...
  - deleted 17y ago
    [deleted]

erlanger 17y ago

People, please stop benchmarking with ab, it does not produce accurate results. All benchmarking of this sort should be done with Siege. It's easier to use too.

acangiano 17y ago
Would using Siege radically change the results?
- erlanger 17y ago
  Yep, I'd expect it to halve the requests per second for every test. Paul Jones (created the Solar framework) periodically releases some pretty authoritative benchmarks for the major PHP frameworks, and here's what he has to say on the topic: http://paul-m-jones.com/?p=421 http://paul-m-jones.com/?p=421
  - acangiano 17y ago
    Good to know for future reference.
  - dsp 17y ago
    His assertions are not confidence inspiring: http://paul-m-jones.com/?p=413#comment-410123 http://paul-m-jones.com/?p=413#comment-410123
tlrobinson 17y ago
While we're on the subject of benchmarking, has anyone run in to a problem on OS X (Leopard) where you seem to run out of ports, and the benchmark stalls at almost exactly 16,384 (2^16) requests? The benchmark eventually resumes 10-30 seconds later. It seems OS X can't free up closed ports fast enough. I've tried this with numerous HTTP servers (including Apache) and both ApacheBench and Siege, so it seems like it's an OS problem.

deleted 17y ago

[deleted]

fizx 17y ago

Did not expect to see EM-httpserver kicking ass.

paul 17y ago
I think it's a c++ library (based on scanning the docs -- I could be wrong), so it's not too surprising to see it outperforming pure Python solutions.
- alecco 17y ago
  Indeed. http://github.com/eventmachine/eventmachine/raw/b4765960db9136197767ca028039650d75f9307d/README http://github.com/eventmachine/eventmachine/raw/b4765960db91... EventMachine consists of an extension library written in C++ (which can be accessed from languages other than Ruby) So the post compares pure Python web frameworks with a C++ library. And not even a good library based on Boost/Asio or libevent.
  - n8agrin 17y ago
    Is eventmachine's c++ lib really not a "good library"? If not, why?
    - alecco 17y ago
      It reinvents the wheel. As a developer, you have to have really good reasons not to use it. Especially since Boost license is practically AS-IS, they could have taken the original source. It is possible they are better at this problem than others, but that is very unlikely, from experience. As a developer it isn't wise to do a reliability test on a new library by yourself. Making a portable async framework is quite error-prone because of the heterogenous situations it could face. You can witness this reading libevent's evolution over the years. And those developers are as good unix coders as you can possibly get with vast experience on the subject (since mid 90s.) I bet the Boost/Asio guys are very good C++ coders too, Boost is as close as you can get to STL. At a glance, http://rubyeventmachine.com/ http://rubyeventmachine.com/ doesn't state why a new C++ async library. They only mention the Reactor Pattern, and Boost/Asio supports it: http://www.boost.org/doc/libs/1_40_0/doc/html/boost_asio/overview/core/reactor.html http://www.boost.org/doc/libs/1_40_0/doc/html/boost_asio/ove...

paul 17y ago

It would also be fun to see how the raw Tornado httpserver (http://github.com/facebook/tornado/blob/master/tornado/httpserver.py http://github.com/facebook/tornado/blob/master/tornado/https...) performs without the Tornado web.py framework on top of it.

jraines 17y ago

Is it possible to take advantage of this with any existing Ruby framework?

acangiano 17y ago

I updated my post and introduced substantial changes.