59 ms·

Web Framework Benchmarks

• bhanks 14y ago
  I guess it is a good thing we are finally migrating away from Cake :/
• bhauer 14y ago
  I worked with Pat (pfalls) on this effort. He pulled the benchmarks together and built the script to automate the tests. We aimed to deploy each framework/platform according to best-practices for a production environment and then stress test common operations: JSON serialization of objects and database connectivity. We were surprised by the wide spectrum of performance we observed and hope that this interesting to you as well. Four orders of magnitude in one of our tests! If you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.
  • goodwink 14y ago
    I'd be interested to see performance for Vert.x on its other hosts (this is the JVM version, I believe).
    • bhauer 14y ago
      Interesting point. You're correct, we've only tested Vert.x as a Java/JVM platform.
      • proland 14y ago
        I agree, seeing vert.x with it's other language options would be interesting.
      • kptnkrk 14y ago
        Rhino with RingoJS would be another good JVM-based test.
    • spartango 14y ago
      I think you may misunderstand Vert.x's polyglot features: While vert.x supports many programming languages, all of these are run on the JVM runtime. This means when you use the ruby vert.x API, you're using JRuby; likewise with Javascript run through Rhino, Python through Jython, and Groovy/Scala run through their own interpreter/compilers. That said, it would definitely be interesting to see the performance implications of using one of those languages and vert.x on the JVM.
  • davidw 14y ago
    Erlang would be nice to see, with, say, Chicago Boss.
    • bhauer 14y ago
      Agreed. As you can imagine, we had to stop adding additional frameworks somewhere or we'd never get this posted. :)
      • Licenser 14y ago
        I've send a pull request for the cowboy webserver (erlang based) it does pretty well in my local tests :) ranks between Go and gemini
  • ubershmekel 14y ago
    I'm no expert, but I think certain languages/frameworks are better suited to be behind certain servers when high concurrency is tested. E.g. from http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers it seems django would be better served behind gevent.
    • rozap 14y ago
      There are a lot of variables and tweaking that can be done, and it would be nearly impossible to optimize each. Similarly, I was wondering what sort of an effect connection pooling would have, as the out of the box django distribution doesn't do that. It really didn't perform too well in their tests.
      • knappador 14y ago
        At LEAST gevent with session write-through caching, psycopg2pool, Postgres SQL (hello, excellent South support?), no unnecessary middlewares or applications that rely on them (if it's a speed oriented use of Django, we're hosting on a specific API sub-domain, right?). At most, THEN you tune the settings to have an optimum number of Postgres threads staying alive and tweak some gunicorn/nginx max connections parameters for your site. If running all locally, use UNIX sockets. This article is trash when it comes to providing any useful data other than Django that's barely been configured beyond not using SQL-Lite, and who the hell uses that in production, so I don't buy their argument about "oh well we just wanted to see what It'd do out of the box" rhetoric. Might as well benchmark ./manage.py runserver. I wish they'd it right or don't publish, let alone publish to shill their company that doesn't provide what they advertise.
        • bhauer 14y ago
          Thanks for your pull requests, knappador. We will try to get a revised post out in roughly a week or so with as many of the tweaks we've received (as pulls and tips) as we can muster.
  • bascule 14y ago
    Since these tests seem to be all about JSON serialization, it would've been interesting to see the tests with rails-api instead of the standard Rails stack: https://github.com/rails-api/rails-api https://github.com/rails-api/rails-api What webserver were you using on JRuby? Was it Trinidad? Did you try Jetpack?
    • stephenhuey 14y ago
      I concur, and Rails 4 may not be officially released yet but it's stable enough to run these tests against.
      • mikeycgto 14y ago
        rails 4 with some concurrency would be interesting to see
• krg 14y ago
  There is a huge difference between raw PHP and CakePHP. I'd be curious to see other PHP frameworks (such as Zend, or Slim) in there-- is Cake just particularly slow, or is that simply what happens when you have a PHP framework?
  • insraq 14y ago
    Per my experience, CakePHP is probably one of the slowest PHP frameworks. It has a large overhead and lots of legacy code that slows down the whole system. Generally, frameworks targeted at PHP 5.3+ are faster. PHP 5.4 has further performance boost in production and PHP 5.5 has Zend Optimizer built-in, which suppose to further speed up but I have never try that in production.
    • bhauer 14y ago
      We have Zend in mind as the next PHP framework we'd like to include in the tests.
      • camus 14y ago
        use either symfony 2 or silex. There is really 0 reason to use Zend Framework II.
  • Schlaefer 14y ago
    CakePHP is always at the bottom end of these "framework performance" tests even when comparing PHP only. It's the everything but the kitchen sink of PHP frameworks with bells and whistles never used by these "hello world" performance tests. Just the wrong tool for the job if you want to spill out some js.
  • Joeri 14y ago
    I'm curious if they configured php with apc or zend optimizer. The huge difference is easy to explain as parse overhead for the framework's code, which happens on each request if you're not using a bytecode cache.
    • 14y ago
• goodwink 14y ago
  What was the parallelization like across these tests? Were they all running single thread/process mode or did you try to take advantage of threading/multiple-processes/etc. to optimize a production-like performance across all the different frameworks? If the latter, that's a really impressive amount of work!
  • pfalls 14y ago
    We attempted to take advantage of threading/multiple-processes as best we could (see the nodejs code for an example of using the cluster module). But we suspect there are additional areas of improvement here.
  • goodwink 14y ago
    After having a look at the code on github it looks like they did set up multi processes/threads to really test the scalability on these platforms. That's really impressive, way beyond the single thread experiment I would have anticipated something like this doing! Wow!
  • bhauer 14y ago
    The short answer is that we attempted to use the CPU as fully as possible across all frameworks. So for those that had tunable parallelism, we used the settings that seemed best given the number of cores available on each platform (2 for EC2 Large, 8 for i7). We posted the deployment approach for each framework to the Github page.
• Kiro 14y ago
  So this means I should stop using CakePHP?
  • anuraj 14y ago
    If you care about performance, the thinner the framework, the better.
    • dkroy 14y ago
      Doesn't matter if you compile it like Facebook had done with their php at one time when they needed to scale.
      • anuraj 14y ago
        To some extent yes, but bloated framework issue will remain even if you compile to C++ despite the compiler's best effort optimization.
  • kh_hk 14y ago
    Why would you? If you write clean maintainable code and are productive using it there's no problem, there are tons of ways of tuning Cake (as most frameworks). The default configuration is meant for 'bootstrapping'. However, if you are interested, you can always check out other PHP frameworks (Yii, slim), or if you like all the bell and whistles from Cake and are willing to dive into another language, you can always experiment with ruby (ror, sinatra), node (express) and python (flask).
  • asadkn 14y ago
    Not really. Take this benchmark with a grain of salt unless they benchmark it again with APC at minimum. Nobody serious about a web app would run a PHP app without APC. Similarly, relatively modern deployment standards (not really cutting-edge) like nginx and php-fpm OR apache 2.4 with worker mpm + php-fpm should be added to the mix.
    • bhauer 14y ago
• akmiller 14y ago
  I'd like to see how .Net MVC would compare. I realize you'd have to spin it up on a Windows EC2 instance and there would definitely be some variance in the performance of that box vs. the nix EC2 instances but I'd still be interested in seeing how it fares in comparison.
  • bhauer 14y ago
    We agree. We aim to provide a .NET MVC test soon. We did briefly test that on our i7 hardware and if memory serves me correctly, it clocked in at around the position as Spring. But don't quote me on that! :)
    • macca321 14y ago
      Yep, would also be interesting to see Web Api, hosted on Windows and on Mono.
    • pfalls 14y ago
      I think you're about right on that one. Of course we were running on mono.
    • bblackshear 14y ago
      I would also suggest ServiceStack
    • hhudolet 14y ago
      I would very much love to see: (win | mono) + (httphandler | asp.net mvc | webapi | servicestack | nancyfx) it would hopefully compare with java stacks!
    • adad95 14y ago
      And Synchronous and Asycnhrounous (Using C#5 async) version.
• anuraj 14y ago
  Interesting - C# should have also been added. Java still rules the roost. If you should absolutely have a scripting language - old php with warts is better. More magic in the framework - more abstraction and indirection and code inefficiency - that can be the spoil port when it comes to performance.
  • bhauer 14y ago
    We agree and want to get a C# test in there. It's among the top priorities for us.
• jondot 14y ago
  pfalls - amazingly, I spent the last 2 days of my holiday doing the same thing for a future open source project. I was just stumped when I saw you guys did the same (could have saved me a couple of days!) I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it. I tested: Go, Java (servlet, dropwizard), Scala (scalatra), Ruby, Node.js (connect). For me it was: * Scala * Java * Clojure (equal to Java - big surprise here) * Node.js * Go (almost equal to Node.js) * Ruby (far far down) Scala took the lead with amazing results. More over, a good metric was latency which Scala was the only one to take micro-second resolution. I'm not a fan of Scala because of its surrounding tools, which is why I'm still considering going for either Clojure or Node.js. I think the most surprising positively was Clojure, being that it is a dynamic language. And most surprising negatively was Go - by itself is impressive, but when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to see this correlates with your findings too; I'm assuming this is a symptom of library maturity..? I'd be happy to see how Scala fares on your tests. You've done an awesome job!
  • pfalls 14y ago
    Thanks for the comment! This started out as a small exercise, that quickly ballooned because we were curious about every framework and platform. Obviously we had to stop somewhere, but we're very interested in adding more tests in the future. In fact, we're hoping the community will help us out as well!
    • jondot 14y ago
      Yes, I know the feeling :) What started as a couple of hours of exercise for myself ended up as 2 days of hacking and barely sleeping, as surprises in my assumptions kept unfolding, and as I wrote and rewrote POCs just to validate that Clojure is as fast as the number say, and that Scala is faster than Java, etc.
  • zerni 14y ago
    Scala really would have been nice. Especially a Scala/Akka/Spray kombo :) I'm working with a setup like this and just love it!
• wldlyinaccurate 14y ago
  Cake was an interesting choice for a PHP framework to test. I wonder why they didn't choose Symfony, which is arguably the leading PHP framework.
  • bhauer 14y ago
    It was simply what we have seen most often in our experience. We've got Zend as a next target for PHP frameworks. We can also put Symfony on the list. We'd also love to have anyone who has production experience with those to contribute a test for them. The test should be fairly quick to write.
    • deleted 14y ago
      [deleted]
  • asadkn 14y ago
    Symfony and Zend might perform even worse. At least Zend's bootstrapping is even heavier. The sheer number of files that have to be loaded from disk on each hit is appalling. Hence I mentioned in another comment the opcode cache (ex. APC) requirement and how every serious PHP app utilizes it.
  • jasonhanley 14y ago
    Zend and Symfony are probably the leading OO MVC frameworks for PHP. I'd also like to see a "light" framework meant for building APIs, like Slim, for instance.
    • deleted 14y ago
      [deleted]
    • wldlyinaccurate 14y ago
      Yes, I'd definitely like to see Slim tested as a light framework. People keep mentioning Silex as a "light" framework, but I've found it to be almost as heavy as Symfony2 (which isn't much of a surprise considering it's just Symfony2 components strapped together).
• donebizkit 14y ago
  Very useful info. Thanks for sharing.
• goodwink 14y ago
  One of the most interesting things this comparison brings out to me is not so much the differences between the various frameworks (although the differences between options on the same platform is definitely very useful information), but also the issue that few of us seem to think about these days: the cost of any of the frameworks above the bare standard library of the platform its hosted on. Theres a consistant, considerable gap between their "raw" benchmarks (things like netty, node, plain php, etc.) and frameworks hosted on those same platforms. I think this is something we should keep in mind when we're tuning performance-sensitive portions of APIs and the like. We may actually need to revisit our framework choice and implement selected portions outside of it (just like ruby developers sometimes write performance-critical pieces of gems in C etc.) or optimize the framework further. I'd like to crunch these numbers further to get a "framework optimization index" which would be the percentage slowdown or ratio of performance between the host platform and the performance of the framework on top of it. I might do this later if I get a chance.
  • jondot 14y ago
    I think this is a much needed and excellent point to make. Just take a look at how Go dips down when using Webgo.
    • goodwink 14y ago
      I think Go is almost the ideal example here, you're right. Go provides a pretty rich "standard library" for writing web serving stuff so it's a good place where you could really imagine writing your performance critical stuff just on the base platform even if you use something like Webgo for the rest of your app. Some of the other platforms are much less amenable to that since the standard primitives the platform exposes are very primitive indeed (servlets api, rack api, etc.). Perhaps there's some value in looking at how your favorite framework stacks up against its raw platform and trying to contribute some optimizations to close the gap a bit.
    • voidlogic 14y ago
      Webgo... I'll stick with "net/http" and Gorilla thanks. Also, They used Go 1.0.3... I hope they update to 1.1 next month. Most everyone using Go for intensive production uses is using Go tip (which is the branch due to become 1.1 RC next month)
• atonse 14y ago
  I'm torn about this. On the one hand, while I've known my framework of choice (Rails) is slow, I didn't know how much slower it could be in the grand scheme of things. But on the other hand, I'm more shocked by the difference between EC2 and dedicated hardware (10x improvement with rails), and even 89 requests per second (20 query benchmark on EC2) is still a decent amount of traffic. (Plus this doesn't count any optimizations I would make anyway, like caching). Either way good architectures usually optimize the high traffic or high CPU areas anyway away from a scripted language. Thanks for the really informative post! Go seems to be a good balance as a high performance language without having to go back to my traumatic Java days.
  • bhauer 14y ago
    Thanks for the feedback, atonse! We had a great amount of fun putting this together, as you can imagine. I agree, a remarkable take-away for us was how dramatically our i7s excelled over the EC2 instances. Admittedly, those were EC2 Large and not Extra Large instances. A previous draft of this blog entry had a long-winded analysis of hosting costs--discussing the balance between ease and peace-of-mind provided by something like AWS versus the raw performance of owned hardware--but we elected to remove that since it wasn't really the point of the exercise.
    • goodwink 14y ago
      Were these the new 2nd-generation large instances or the original ones?
      • bhauer 14y ago
        They were m1.large.
  • jondot 14y ago
    Don't be too disappointed about Rails. As a rule I like to divide this world into "Featureless" and "Featurefull" products. When you use Rails, you're aiming to pile up features. You want to react to Product managers, to users, you want to work fast and satisfy the needs of customers - or else you won't have anyone to build to. In this reality, the fact that you're doing 20req/s is OK. In fact, I'm betting that even when you take Go or Node.js - pile up all of the infrastructure and features that exist in Rails, and pile up a ton of your code - buggy and not buggy - you'll get around the same kind of satisfaction index from users. This is because your product can be perceived as slow even though your servers are blazingly fast. On the other side of the spectrum there are "Featureless" products. These are infrastructural products. A logging service. An analytics service. A full-text search. A classification and recommendation engine. These you don't want to build in Rails. I'm sure you haven't even considered it. These you want to build with one of the top-notch libraries that this survey indicate.
• gruseom 14y ago
  It would be very interesting to see OpenResty (Nginx + Lua) in there, since it's so different from other approaches.
  • xt 14y ago
    I would love to see Openresty tested, for simple json and databases Openresty is a magnificent tool.
  • ajacksified 14y ago
    Agreed! Especially built with luajit.
• adad95 14y ago
  I Will love to see dotNet (C#) incluid in this test. Asp.Net WebAPI (Synchronous and Asycnhrounous) Asp.Net MVC (Synchronous and Asycnhrounous) Asp.Net HTTP Handlers (Synchronous and Asycnhrounous)
  • Throwadev 14y ago
    Also ServiceStack vs ASP.NET MVC vs NancyFX vs Fubu, Mono vs IIS/windows, default serializer vs JSON.NET vs ServiceStack's, etc. There are tons of variations that could be done.
    • platz 14y ago
      ServiceStack vs WCF would probably be a more apt comparison, but I suppose there are people using SerivceStack for web apps as well.
  • mythz 14y ago
    Here's a fairly recent end-to-end ServiceStack vs WebApi benchmark: https://twitter.com/anilmujagic/status/272544925478973440 https://twitter.com/anilmujagic/status/272544925478973440 ServiceStack 9615ms WebApi 30607ms GitHub project for benchmarks used: https://github.com/anilmujagic/ServiceBenchmark https://github.com/anilmujagic/ServiceBenchmark
• eCa 14y ago
  Would love to see how modern Perl frameworks (Mojolicious/Dancer) ranks in such a test.
  • bhauer 14y ago
    Interesting thought. Would you be able to put together a test for one of those? pfalls can give you the details for how we expect the test for a new framework to work.
  • deleted 14y ago
    [deleted]
    • bigpresh 14y ago
      No, they're not. They can run via CGI (so, yes, compiled for each request) but that's slow as ass. They run via Plack/PSGI - they can be deployed in a multitude of ways, including via FastCGI or running standalone using built-in webservers, or via Starman - the latter in particular is very fast indeed. With a simple "hello world" type app, > 6000 requests/sec can be easily handled. Obviously an app of more realistic complexity won't be quite that fast, but you'll still handle many requests every second with no trouble. http://stackoverflow.com/a/4770406/4040 http://stackoverflow.com/a/4770406/4040 contains some basic benchmarks. EDIT: also, mod_perl is generally best avoided these days; it's old, not very pleasant to work with, and ties you to Apache. Writing an app with Dancer / Mojolicious etc means you can deploy in various different ways with ease.
      • laumars 14y ago
        Funny enough that SO link was the same one I got my misinformation from as the top answer pointed to CGI benchmarks. Thanks for the correction though, but that still doesn't answer my original question: how does Plack compare with mod_perl?
        • draegtun 14y ago
          Plack/PSGI (Perl) == WSGI (Python) == Rack (Ruby) These are all abstraction layers (for each language) which then can be run seamlessly on top of CGI, SCGI, mod_(perl|python|ruby), etc. - http://plackperl.org http://plackperl.org - http://en.wikipedia.org/wiki/Plack_(software) http://en.wikipedia.org/wiki/Plack_(software) - http://en.wikipedia.org/wiki/PSGI http://en.wikipedia.org/wiki/PSGI - http://en.wikipedia.org/wiki/Wsgi http://en.wikipedia.org/wiki/Wsgi - http://en.wikipedia.org/wiki/Rack_(web_server_interface) http://en.wikipedia.org/wiki/Rack_(web_server_interface)
• ryanslade 14y ago
  Why are there no Go results for the DB access benchmarks?
  • laumars 14y ago
    I'd be interested to know this as well. I can understand time constraints in compiling these kind of comparisons, but as I'm about to embark on a web project in Go, I'd be interested to know if there was any technical constraints that prevented benchmarking Go.
    • bhauer 14y ago
      We recognize that deficiency and we do aim to add database tests for Go. If you could quickly draft that up and submit a pull request, we'd love to add it.
      • laumars 14y ago
        Sorry, but as much as I'd love to contribute I'm still learning how to write database apps in Go myself (that's actually part of the reason I wanted to see your tests). So I'd be the wrong person to to write code for that particular test.
• deleted 14y ago
  [deleted]
• neya 14y ago
  How about Lift? Btw, the play framework you tested is Java or Scala based? Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!! Maybe Node.js critics should start appreciating it after all..
  • kainsavage 14y ago
    It is probably worth noting that while we strive to make the tests as fair as possible, we followed the official tutorials for each framework when building out the tests but we fully expect there to be small instances where minor tweaks improve a given test. Given that I am no Play expert, it would be of great value to have one who is (and it sounds like you could lend a hand there) to check out the code on the github page. If we did anything wrong with the setup or in general, we would rerun the tests. Again, we followed all the official 'getting started' posts for each framework, so we believe we have best practices used. Disclaimer: I am a colleague of the author of the linked article.
    • jholla14 14y ago
      The problem is that the database queries were done in a blocking fashion. The test essentially blocks the main event loop which is of course going to kill performance.
    • mark242 14y ago
      There are a few problems with your Play code that are causing it to be unnecessarily slow. First-- what you're really testing here is the Jackson library. A majority of the cycles used in your application are being burned in that toJson call of an array of objects. This isn't a fair test compared to the servlet implementation because you're calling Jackson against a map in the Play example, versus against a simple String in the servlet example. Second-- you are running database calls serially, and those are blocking. Considering that you're using the more-or-less default Play/Akka configuration, there are only enough threads as you have available processor cores. I would start by increasing the parallelism-factor and parallelism-max to be higher, so you'll have more available threads. More importantly though, the database access should be wrapped in a Future, and you should be returning asynchronously. This should speed up the application by a huge amount.
• akg 14y ago
  Would love to see how these results compare to some of the web frameworks for concurrent functional languages like Erlang/Haskell: Nitrogen, Chicago Boss, Snap, Yesod, etc.
  • pilgrim689 14y ago
    ditto. I hear Warp (the server behind Yesod) is a beast.
    • bhauer 14y ago
      I'm not familiar with Warp. Would one of you guys be willing to help us put together a test for Yesod?
      • josh-j 14y ago
        #haskell, #yesod, #snapframework on freenode are very helpful.
  • lorddamien 14y ago
    Right, where is Zotonic??, which is actually focused on speed/performance
  • chrisdone 14y ago
    Please don't encourage them. Even if Haskell comes out on top, I would still be unsatisfied because the rest of the benchmarks are unfair. Lies by confusion are still lies.
  • chrisdone 14y ago
    Please don't encourage them. Even if Haskell comes out on top, I would still be unsatisfied because the rest of the benchmarks are unfair. Lies by confusion are still lies.
• nohorse 14y ago
  Nice work. I'd like to see comparison also against Microsoft stack.
• obsurvey 14y ago
  If I wanted to return a bunch of _ids from a mongo database i would do a _id:$in[array of ObjectIDs] and then stream the resulting cursor to res. Streams are core to node. You should use them.
• jtreminio 14y ago
  I would love to contribute a code for a specific PHP framework: Silex. Do you have requirements you'd like to hit?
  • bhauer 14y ago
    That would be fantastic. Get in touch with pfalls on Github and he can give you the information. It's fairly simple.
    • wyck 14y ago
      +1 this would be interesting given that there are a lot of PHP frameworks besides cake, namely. Silex/Symfony2 Yii Zend Kohana Fuel Laravel
      • johnx123-up 14y ago
        I'll be happy to see benchmarks with PHP frameworks written as a C extension: 1. Yaf PHP http://www.yafdev.com/ http://www.yafdev.com/ , 2. PHP-ActiveRecord++ https://github.com/roeitell/php-activerecord-plusplus https://github.com/roeitell/php-activerecord-plusplus
• vph 14y ago
  This seems to be a nice benchmark. For the Python group, I would suggest two things: (1) include a lightweight framework like Bottle, and (2) Try pypy.
  • bhauer 14y ago
    Thanks. Agreed, we'd love to get a Python micro-framework in the test. If you have some free time and feel like putting together a test for Bottle as a Github pull request, we'd really appreciate that.
  • apendleton 14y ago
    I'd be curious about a gevent-worker-driven server, too. Presumably they're running Django on either a thread- or processor-driven concurrency server, and gevent can show some pretty major gains. You could also do permutations of these: gevent on pypy (using pypycore), etc.
  • est 14y ago
    (3) use bjoern seriously, in my rough helloworld and sqlite value increment by 1 benchmark, Bjoern+wsgi app runs 2x as fast than nodejs.
• sergiotapia 14y ago
  Where is ASP.Net MVC? Odd that you list obscure frameworks like Wicket and leave out one of The Big Four frameworks. (The big four in my book are: ASP.Net MVC, Rails, Django and CakePHP)
  • bhauer 14y ago
    We'd love to have ASP.Net MVC included. One minor gotcha is that to do it justice, we'd need to spin up a Windows EC2 instance and figure out how to script that. It's on our to-do list! We did briefly test ASP.Net on Mono (see another comment in this thread) but didn't include it since we didn't believe that qualifies as a "production" grade ASP.Net MVC deployment.
    • voidlogic 14y ago
      You should include it; Lots of shops are deploying their products on .NET MVC with Mono.
      • Revex 14y ago
        I agree... Let's see what the numbers show for Mono, and on Windows.
    • sakopov 14y ago
      Just use AppHarbor.
  • Revex 14y ago
    I came here to say this. I don't understand why some of the development community likes to act like .Net doesn't exist.... It pains me to see charts and reporting done like this while leaving out my favorite framework.
    • pekk 14y ago
      Since these benchmarks are so wide-ranging, I agree. But that means setting up an entire new testbed on Windows, and then trying to make it comparable to the other platform testbed; possibly tuning. You need a Windows expert to do this. My question is, why aren't Windows experts setting these up?
• bioinfo 14y ago
  I think this benchmark is not that useful. If you seriously thought that the difference between a netty and django would be 4 times then you simply don't understand what these frameworks do in the first place. I would have guessed something like less than 100 times slower and that would have been still fine since the cost of all kinds of latencies in the system usually far outweigh the speed of the framework itself.
• chasing 14y ago
  What's the difference between "php" and "php-raw" in some of the data? Maybe I'm still in my morning fog, but having trouble thinking of what "php-raw" might mean. Sigh.
  • jtreminio 14y ago
    The code says php-raw refers to using PDO (which all project should be using) vs using an ORM or ActiveRecord. I am completely stunned by the performance cost of using ORM/AR, and will be using this to shame our team lead into giving it up and going for raw queries.
    • chasing 14y ago
      Gotcha. Thanks!
    • datasage 14y ago
      Some of the ORM cost can be mitigated by using caching. In most cases this is essential in a production deployment.
      • bhauer 14y ago
        Agreed. However, for our database tests we expressly wanted to stress-test the ORM and, where configurable, disabled caching. We plan a subsequent test, time permitting, that enables caching.
    • gaoshan 14y ago
      Important to note that OP doesn't mention using APC with this, something that any production code would be using in a real world case. This would have an impact on the numbers he uses and on the diff we are seeing.
    • mgkimsal 14y ago
      Don't be so hasty. As others pointed out, caching often plays a factor on larger projects. A degree of portability provided by ORMs is nice (test mode hits SQLite, prod is pg or mysql, etc). Also, the readability of ORM-oriented code shouldn't be overlooked, especially as new people come in to the team. Two or three expressive lines in ORM vs 15-20 of nested SQL hitting weird tables names and aliased column names and such isn't easy to decipher the intent of (especially when there's a problem). I typically use ORM for about 95% of a project, falling back to a few explicitly native SQL calls when performance can be shown to be a bottleneck in those locations.
• grannyg00se 14y ago
  Probably not a major factor but I'm curious to know why mongoose was thrown into the mix for the node.js test rather than going with the native mongo driver. It might be more of a real world test to include mongoose with the node+express test, but for the node-only test the native driver might be more appropriate.
  • pfalls 14y ago
    We wanted to use an ORM in all cases, it was only recently that we started working on native mysql access. We hope to add native mongodb results as well.
• codegeek 14y ago
  Nothing for Flask ?
  • stevenleeg 14y ago
    Yeah I was disappointed to see a lack of flash as well, especially since it advertises itself as a microframework and all.
    • bhauer 14y ago
      We received a Github pull request with a Flask test! So we'll aim to include that in a follow up post soon.
• neya 14y ago
  Thank you for such an informative comparison. You've helped shed light beyond the frameworks that are known to most of us - Ruby on Rails, Node.js, Django, Express, etc... Thanks again! :)
• rombdn 14y ago
  Very interesting! But isn't there a disavantage for node.js considering it's single threaded? Did you cluster it to use every core on the box ? (http://stackoverflow.com/a/8685968/1909827 http://stackoverflow.com/a/8685968/1909827 - "Scaling throughput on a webservice")
  • saym 14y ago
    From the OP's response to a similar question: >We attempted to take advantage of threading/multiple-processes as best we could (see the nodejs code for an example of using the cluster module). But we suspect there are additional areas of improvement here.
    • rombdn 14y ago
      Thanks
• desireco42 14y ago
  What I can take from this is that when you use ORM, it slows things down considerably. Also, looking at rails example, you didn't use active record, which is really wrong. I think you should tweak your tests to use more real world like examples. I realize it would be hard to do this across frameworks. Like let's have a database query pull user record from 100,000 users by username. And maybe do md5 on password.
• niteshade 14y ago
  Can't believe you didn't include Zend in there :(
  • craigching 14y ago
    Just read this above: "We've got Zend as a next target for PHP frameworks."
• systematical 14y ago
  These are some brutal results for CakePHP (which I use). However, in practice no intermediate developer would be issuing queries like that in a loop. They would loop through a set of data, building a IN statement example: WHERE field IN ('x','y','z'). Thus only sending a single query to the database. Still, the Cake developers really need to improve the speed of their framework.
  • pekk 14y ago
    I keep seeing numbers like this in benchmarks over the last few years, it would be great to see CakePHP's numbers "rehabilitated" using techniques like you discuss.
    • systematical 14y ago
      It's a fair way of testing, its just not the way a developer would write it in practice. Clearly disregards the "Big O".
  • slashclee 14y ago
    Send a pull request, then.
• pekk 14y ago
  Numerous irregularities plus a strong vested interest in the JVM make me doubt they have given adequate shrift to Go, here. Given the amount of interest in Haskell and Yesod around here, it is strange that it is missing.
  • goodwink 14y ago
    Could you elaborate? Their website seems to indicate that they are a very polyglot shop, not someone pushing a JVM agenda: "On the back-end, we use Java, Ruby, Python, .NET, PHP and others based on what makes sense balancing server performance, scalability, hosting costs, development efficiency, and your internal development team's capabilities."
    • pekk 14y ago
      "We have included our in-house Java web framework, Gemini, in our tests. ... " Then you see results for some languages that are completely out of whack with most other such benchmarks (they themselves mention the weirdness of Sinatra vs. Rails, for example). Then you see on a couple platforms that more performant mainstream options have been excluded, for no good reason. Then if you look at the repo, there are deployment choices and code mistakes in some of the other languages which go well beyond elementary incompetence...
      • kainsavage 14y ago
        Feel free to issue a pull request to help our testing. We are not trying to push Java-based frameworks over any others and we believe we are being fair across the board. That being said, if there are "code mistakes... which go well beyond elementary incompetence", then we would love to correct and retest these.
        • stephenhuey 14y ago
          Maybe you should've had the controllers execute raw SQL for a better comparison. I see you executing a regular query in Rails whereas your Java servlet is using prepared statements.
• richoffrails 14y ago
  If you're going to include evented web libs like Node.js and the frameworks created using it, wouldn't it be fair to include other evented libraries such as EventMachine and Twisted?
• waxjar 14y ago
  Im surprised to see that Sinatra on JRuby often performs worse than Sinatra on MRI, while Rack on the other hand performs much better.
  • bhauer 14y ago
    We were surprised by the sinatra-jruby tests as well. If you read our "expected questions" section, it's not clear to us why Sinatra's performance on JRuby was weak. We'd love to hear from a JRuby expert about how to address Sinatra's "wrong" looking numbers.
    • waxjar 14y ago
      Having had a look at the Gemfiles, it might be the choice of HTTP server used. I believe defaults to WEBrick, if no other server is present and I believe WEBrick is single-threaded. I don't have any experience with JRuby, but this might be a possibility.
    • ssmoot 14y ago
      Maybe the web-server choice (I didn't look to see what that was). But last I looked at it, the Sinatra router really is very naive/awful. For the size of the framework, that may be the right choice ultimately. On the other end of the spectrum is Rails, who've written their own Regexp engine for routing basically. It seems to be generally quite a bit faster. All major Ruby frameworks have just awful routers though really.
    • headius 14y ago
      Yeah, the Sinatra numbers really ought to be more in line with the straight Rack numbers. Something's amiss. Glad to see that on Rack, where I'd expect us to be fast...we are doing ok. On par with PHP (not a big thing to brag about, perhaps) and Play (worth bragging about, since it's Java/Scala end to end). Obviously there's more work needed not just in JRuby but in the servers that server it and the frameworks that run on it. The slow performance of Rails here, for example, is largely Rails' fault. But yeah...the sinatra numbers are wack.
• saym 14y ago
  What I'd love to see paired with this data is a cost comparison. At what scale does performance of ruby/python/php become cost prohibitive? Twitter made the move from Ruby to Java some years back, did they ever post a comparison of their numbers before and after? Also, the difference between EC2 and local i7 hardware is glaringly obvious. At what scale does owning the server hardware become imperative? I know these questions are beyond the scope of a performance review, but inquiring minds would like to know.
  • kainsavage 14y ago
    Actually, we had an early revision of the benchmark tests that had exactly that analysis, but we really felt it is a very interesting subject that deserves more focus.
  • bhauer 14y ago
    As you might imagine, during this exercise, we've had a lot of conversations about the points you raise. We have our own opinions, but we ultimately removed most of that content from the blog post because we didn't want it to be too editorial. We will be posting follow ups with some of our opinions. Some things are really difficult to answer in a vacuum. If you already have a competent devops staff, hosting your own hardware is probably beneficial. The increased performance per "server" is substantial. But no devops staff? Then it's either very risky or cost-prohibitive to own hardware.
    • ochoseis 14y ago
      I too was shocked at the deltas between Amazon and dedicated hardware. I think AWS runs on Xen, so I wonder if you could strike a more optimal performance/flexibility balance using lighter virtualization (LXC or OpenVZ) with either in-house hardware or another VPS provider.
      • spartango 14y ago
        While it is true that EC2 runs on a customized version of Xen, it's very unlikely that latency is being introduced by the hypervisor itself. With paravirtualized kernel and drivers, Xen introduces negligible overhead, and thus LXC would likely be no better. The reason that the virtualized setup performs more poorly than the dedicated setup is that you are fighting for CPU time with other AWS customers, so those other customers are introducing latency into your application. Any shared/virtualized host will have this problem. Interestingly, where Netflix really wants to squeeze CPU performance out of EC2 instances, they allocate the largest instance type so that they know that there's nobody else on the underlying machine.
• knappador 14y ago
  Utter crap. "Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling." But it's free, open-source software and we provide asynchronous database connection pooling for Postgres SQL: https://github.com/iiilx/django-psycopg2-pool https://github.com/iiilx/django-psycopg2-pool
  • andybak 14y ago
    Your comment is valid and would have been vastly improved without the first two words.
    • knappador 14y ago
      I'll elaborate on the first two words: They're shilling for their company with a config and tools I wouldn't be caught dead using. No idea what other craziness lurks in the other daemon configs. It's irresponsible and misleading. They're misrepresenting the framework I use to do my work and probably others while hoping I or someone else is going to do their work for them. "Outsourced CTO services?" Their trash. My lawn. Someone's going to eventually ask me to develop the rest of xyz in node and I'll have to repeat myself about articles like this. Bad enough when it's bloggers. Worse when it's self-shilling company that's obviously not willing to put the time in to be what they claim to offer.
  • rozap 14y ago
    Obviously if they optimized each and every one of these benchmarks we would see different results, but it would take a massive amount of time to learn the ins and outs of each framework to the point where you can do so effectively. For one person do a benchmark over this many samples, they have to just go with the out of the box setup for each.
    • kainsavage 14y ago
      This is really the point. As our blog post suggests, where we are not experts we had to rely on the tutorials provided by each framework's authors to build a test setup. If a specific framework seems low on the list, it could be due to the fact that the best practices guides we found for getting set up were not correctly configured for production use. Draw what conclusions you would like from this statement, but we did aim to be as fair and unbiased as possible.
• jerry2yu 14y ago
  I would suggest to include Tornado Web framework.
• dthakur 14y ago
  Could you kindly add error bars to your test results?
• thatthatis 14y ago
  A few additions/tweaks that would be great to see: 1) Add a raw http test -- no template compilation or html, just return "OK". Would give a relative idea of the cost to "just turn the thing on" 2) Don't json encode after the database tests. By json encoding you're doing two things but saying you're only testing one. I come from the python/django world, and I know that different python json packages have orders of magnitude differences in their performance. From that I can infer that there are probably similar or greater cross-language differences -- I suspect node.js having json as a native object type helps immensely.
• 33degrees 14y ago
  I'd be curious to see how ruby fares using a setup other than apache + passenger; I have a feeling that nginx with unicorn or puma would provide better results, and might give it a go when I have a bit more time.
• eweise 14y ago
  Was Play 1 or Play 2 tested? I thought Play ran on Netty so I'm surprised it didn't fair better.
  • jfim 14y ago
    They're using Play 2.1 with Java, without using futures for database queries. Not sure how representative of real usage that would be, though.
    • bhauer 14y ago
      We've got a bunch of feedback that the Play database test needs to be asynchronous, so we'll make that a priority for our next run. Perhaps I can entice you to submit a pull request to improve the Play configuration?
    • abalone 14y ago
      But the JSON test was 7X slower too.
• jfarmer 14y ago
  What would be much more interesting than "peak responses per second," which is a weird metric to begin with, is the actual histogram of sampled throughputs. Or at the very least a box-and-whisker plot (http://en.wikipedia.org/wiki/Box_plot http://en.wikipedia.org/wiki/Box_plot). Most folks who have run Rails at scale, for example, find that the untuned garbage collector in MRI (Ruby's default interpreter) introduces a large amount of variance, for example.
  • bhauer 14y ago
    We don't have a box plot, but we have line charts for all of the data (performance at multiple client concurrency levels). Just click the "All samples (line chart)" tab on the data panels. (Note that the very first data panel in the intro is an image and doesn't have tabs.)
• cridenour 14y ago
  I'm curious to see Django results when using the gevent worker for gunicorn. For these type of quick JSON calls, you can see huge performance increases.
  • jaddison 14y ago
    Along with gevent, it would be good to throw in psycogreen to improve DB (well, postgresql) evented connections.
    • bhauer 14y ago
      It looks as if we've got a Github pull request including these changes, so we'll be able to revise the Python-Django numbers soon.
  • deleted 14y ago
    [deleted]
• xal 14y ago
  What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment? Update: Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests. Update2: Ok it seems to run in production mode but still, passenger is not an idiomatic choice.
  • daenz 14y ago
    I'm not familiar with Rails. Could you explain what you mean in layman terms? Why is code being reloaded on every request by default?
    • matt_yoho 14y ago
      The Rails framework is written to reload (reevaluate) most application code between requests in development mode, so that changes during active dev are reflected. This is not the (default) behavior in production.
    • pmahoney 14y ago
      In development mode, it's nice to be able edit ruby code, then hit reload in the browser to see it in action without restarting the whole app. Sure there are other ways to accomplish this, but reloading code is typically how it's done in Ruby.
    • awj 14y ago
      In development mode Rails will reload code on each request to pick up on changes you have made to your application. That way you can interact with the application to help verify that your code is working properly. As of Rails 3.2 development mode watches for file changes and attempts to only reload those files, but it's still a significant performance issue. By default all Rails applications start in development mode, so one gotcha of benchmarking Rails is that some people will forget to set the mode correctly. That said, from the setup code[1] (line 14) it looks like they were running passenger in production mode. The max pool size seems excessive, especially when running on large ec2 instances, but I'm not fully convinced that it's out of line. [1] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rails/setup_ruby.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
• olegp 14y ago
  What I've also found is that Node (V8) tends to have much higher variance in response times when compared to Java Servlets (JVM): https://github.com/olegp/common-node#benchmarks https://github.com/olegp/common-node#benchmarks
• tnash 14y ago
  This is exactly why I decided to use PHP for my startup. I have something along these lines that I hope to blog about in the coming weeks (I tested php-fpm on nginx/go/node.js/silk.js and php won by a landslide when it came to speed). I would love to see php-fpm on nginx included in this test.
  • TylerE 14y ago
    The problem with php is that it looks great on (some) micro-benchmarks, but on real apps under real sustained load it certainly turns to cold dog shit from time to time for no apparent reason.
    • xd 14y ago
      What are you basing this on? I've been using PHP for well over a decade in high load environments and never experienced it turn "to cold dog shit" .. any issues I have experienced had a good reason, not "no apparent reason". But then, I've never used a PHP framework in all the time I've used it .. maybe that has something to do with me never having had negative issue with PHP.
  • snaky 14y ago
    When it comes to speed, considering you are using nginx, the way to go is using nginx as an app server not just fastcgi frontend. Lua-nginx-module combined with proper database module (async with connection pool support like ngx_drizzle or ngx_postgres) can give you speed. OpenResty provides the simplified preconfigured way to try it and adds some features too. http://agentzh.org/misc/slides/libdrizzle-lua-nginx http://agentzh.org/misc/slides/libdrizzle-lua-nginx
• saosebastiao 14y ago
  This is super awesome. I wish Yesod/Warp was available though :)
  • raphaelj 14y ago
    I've just send a pull request[1] with an Yesod implementation. It runs pretty well, scoring similar to webgo for the JSON pong benchmark, is almost at the same level as grails for the 1 query benchmark and is slightly faster that Play for the last benchmark. So, Yesod is in the same performance gap as Play or Grails, and is 3~4x faster that Django or Rails. But I've tested those on a mono-core Virtual Box, and I know Yesod scales pretty well on a multi-threaded environment. Also, keep in mind that most of the top performing frameworks are not fully featured web framework but asynchronous I/O libraries (netty, go, nodejs, vertx, ...) which implementations just write mindlessly the raw response directly on the socket whatever the HTTP request was. [1] https://github.com/TechEmpower/FrameworkBenchmarks/pull/39 https://github.com/TechEmpower/FrameworkBenchmarks/pull/39
    • bhauer 14y ago
      Thanks, Raphaelj! Pat (pfalls) will be in touch if he has any questions. Really appreciate the contribution!
• asbjornenge 14y ago
  It just goes to show that your framework of choice is pretty unimportant. The developers (and their skills), who are actually developing on and configuring the framework, outweigh the "baseline" for performance on multiple orders of magnitude. From that I'd say that developer friendliness (easily approachable concepts) and sensible defaults is by far the most important thing when choosing a framework.
• hnriot 14y ago
  haha, all those hipster developers using rails can now eat the php guys shorts :) Seriously though, this isn't news to anyone that does this professionally. The further up the abstraction curve you climb, the less performant the code will be. Ease of development vs run-time performance.
  • oneeyedpigeon 14y ago
    Exactly. And since human time is much more expensive than CPU time, very few people are writing their web apps in machine code.
• codenerdz 14y ago
  One glaring ommision: DOS on Dope [1] [1] http://dod.codeplex.com/ http://dod.codeplex.com/
• shanelja 14y ago
  I'm most surprised that PHP seems to be around an order of magnitude faster than Ruby on Rails, I knew it was faster, but didn't think it would be that much.
  • justinsteele 14y ago
    The comparison would be php-raw vs ruby. The actual "php" benchmark did just as poorly as rails.
• vanderZwan 14y ago
  "Among the many factors to consider when choosing a web development framework, raw performance is easy to objectively measure." Oh really? Then why did Zed write such an angry rant about how you are doing it wrong? http://zedshaw.com/essays/programmer_stats.html http://zedshaw.com/essays/programmer_stats.html Can we please see some standard deviations, at least?
  • k__ 14y ago
    What's wrong with that guy?
    • vanderZwan 14y ago
      It's a pretty serious problem with how we benchmark though. EDIT: when looking up what I vaguely remembered I somehow managed to come across a similar article that was published just today[1], even though I was referring to an older one[2] which was about microstuttering (basically: a high standard deviation in frame rate). The point still stands - in fact it applies to both cases in somewhat different ways. To give an example: Crossfire and SLI graphics card setups a few years ago[1]. It turned out that while both gave a similar performance increase in average framerates. Then was it discovered that one of them had a significantly lower minimum framerate than the other. A high minimum framerate is probably more important in shooters than peak performance, but that's not what we've been testing all of these years, is it? That's exactly the problem highlighted in the article by Zed. I know this is a gaming example, but I'm sure that in user perception of the performance this matters just as much for the responsiveness of webpages. [1] http://www.tomshardware.com/reviews/graphics-card-benchmarking-frame-rate,3466.html http://www.tomshardware.com/reviews/graphics-card-benchmarki... [2] http://www.tomshardware.com/reviews/radeon-geforce-stutter-crossfire,2995.html http://www.tomshardware.com/reviews/radeon-geforce-stutter-c...
      • spartango 14y ago
        Indeed, it's important to look not just at the average performance and performance extremes, but also the distribution of performance. Standard deviation helps with this. Also, often times looking at the latency at the 50, 90, and 99th percentiles is valuable as you can see events that would make your users unhappy. They're a very tangible series of metrics.
• neya 14y ago
  For anyone else curious on how Vert.x stacks up against the famous Node.js: http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-js-simple-http-benchmarks/ http://vertxproject.wordpress.com/2012/05/09/vert-x-vs-node-...
  • nyam 14y ago
    you haven't read discussion, right?
    • neya 14y ago
      I posted this way before the discussion happened.
• mmcclellan 14y ago
  The wsgi benchmark uses gunicorn, which is fine. But if you are surprised by the so-so performance, know that the worker class can be changed via the -k flag (-k gevent), and may improve performance. Of course, other wsgi servers, like uwsgi, are also available.
  • knappador 14y ago
    Are they not even using gevent? Someone please get these guys off the internet. The are worse than some personal blogs at everything except the comprehensiveness in their low-quality testing.
• kbar13 14y ago
  ...flask?
• JOnAgain 14y ago
  This is awesome. Thank you.
• reactor 14y ago
  Not to nitpick, but that (and another popular one http://www.techempower.com/blog/2013/03/26/everything-about-java-8/ http://www.techempower.com/blog/2013/03/26/everything-about-...) was a clever way to get people to techempower.com :) Useful comparison anyway. Seems Go struck a good balance.
• mariust 14y ago
  I am 99% sure that you haven't used contain=>false for cake php this way all related tables are loaded to serve the related data.
• suyash 14y ago
  Java (JVM) > JavaScript (Node.JS) > Ruby-Rails
  • chc 14y ago
    If all you're doing is trying to serve JSON as fast as possible.
• knes 14y ago
  No love for Flask? I would have love to see how it compare to Django and RoR
• jalopy 14y ago
  As a Rails developer and admirer, this is eye-opening. I love the framework (and Ruby especially), but these numbers bear some serious consideration. 30-50x performance difference gets really... real, no? The standard refrain of "throw more hardware at it" must reconcile with the fact that a factor of 30-50x means real dollars for the same amount of load. Is the developer productivity really that much greater?
  • tiedemann 14y ago
    Really depends on which kind of app you're working on. My main work app is 99% cached content so it would probably work just fine with almost anything. Developer time is certainly the biggest expense in my case so high-level it is.
  • bradleyland 14y ago
    Preface: This post is going to come across as a Rails apologist piece, but please read the entire thing before you reach a conclusion. Please also consider that you could apply these same arguments to just about any of the high-level language based frameworks on the list. I use Ruby on Rails in my comparisons, but I'm a huge fan of Node.js, Python/Django, and Go. I fully respect the JVM family of languages as well. I just think that Mark Twain said it best when he said: "There are three kinds of lies: lies, damned lies, and statistics." It's not that the numbers aren't true, it's that they may not matter as much, and in the way, that we initially perceive them. Performance is certainly something you should consider when selecting a language/framework, but it is not the only thing. ======================== You should undertake a detailed examination of these statistics before making any decisions. Issue #1) The 30-50x performance difference only exists in a very limited scenario that you're unlikely to encounter in the real world. Look carefully at the tests performed. The first test is an extraordinarily simple operation: take this string, serialize it, and send it to the client. This is the test in which we see massive differences: Gemini vs Rails 25,264/687 (gemini/rails-ruby) = 36.774 25,264/871 (gemini/rails-jruby) = 29.000 Node.js vs Rails 10,541/687 (nodejs/rails-ruby) = 15.343 10,541/871 (nodejs/rails-jruby) = 12.102 That's a 37x performance win for Gemini, and 15x for Node.js. Side note: You might be wondering why I didn't compare to the top performer, Netty. Netty is more like Rack. You build frameworks on top of Netty, not with Netty. As a Ruby dev, you could think of this in the same context of comparing Ruby on Rails with Rack; not a good comparison. Hence We won't compare Rails to Netty. The error would be in extrapolating that a move to Gemeni or Node.js would give you a 37x or 15x performance increase in your application. To understand why this is an error, we jump down to the "Database access test (multiple queries)" benchmark. Issue #2) Performance differences for one task doesn't always correlate proportionally with performance differences for all tasks. In the multi-query database access test, we start to see the top JSON performers slow down significantly when compared to the slow down for Rails: Gemini vs Rails 663/89 (gemini/rails-ruby) = 7.449 663/108 (gemini/rails-jruby) = 6.138 Node.js vs Rails 116/108 (nodejs-mysql-raw/rails-ruby) = 1.077 60/108 (nodejs-mysql/rails-ruby) = 0.555 In this scenario -- which is arguably much closer to the real world -- Ruby on Rails closes the gap and even beats some of the hip new kids. But why? The in-depth answer to this question would require a lot of space, but the really, really short version is kind of a "what's the sound of one hand clapping" response: Ruby isn't actually all that slow. To understand what the hell that means, check out this presentation from Alex Gaynor (of rdio/Topaz fame): https://speakerdeck.com/alex/why-python-ruby-and-javascript-are-slow https://speakerdeck.com/alex/why-python-ruby-and-javascript-... Ruby is just about as fast as C, provided you're comparing it to C that does exactly the same operations on the hardware as the Ruby code. Don't get me wrong, that's a HUGE provision. But it warrants close examination. The real benefit of lower level languages like C is that they give you the flexibility to drill down in to your actual bare-metal operations and optimize the way the program executes on the hardware. As Alex points out, we don't currently have that level of flexibility in languages like Ruby (without dropping down to inline C), so we suffer a performance penalty. This penalty is huge for simple tasks because they involve only a handful of operations that execute extremely quickly. As you add complexity, however, the benefits of micro-optimizations get lost in the vastness of the overall execution time. Look at it like this. When Gemini hits 36,717 req/s in the JSON test, each request only lasts about 1.6 ms. This is only possible because of the simplicity of the operations being done on the hardware. Ruby loses big here because there is a lower boundary to the way you can optimize without dropping down to C. gemini: 1.6 ms per request rails-ruby: 87.3 ms per request When we look at the multi-query database access test, we can see how the optimization at the low level gets lost in the sea of time taken to process the request. gemini: 90.5 ms per request rails-ruby: 674.2 ms per request Granted, that is still over a 7x performance win for Gemini, but this is where the Ruby arguments about programmer efficiency come in to play. I don't know Gemini, so it may very well beat Rails in that comparison too. Ruby is getting more performant with every release though, so it's easier to justify on the basis of preference alone when we're this close.
• dmoreno 14y ago
  Very nice benchmark. As developer of a toy "framework", but powerful http server library, I wanted to check on my project. I did a fast blog entry at http://bit.ly/10l9Smj http://bit.ly/10l9Smj. I got ~58288.10 for simple json, 8594.39 on db using sqlite, 503.23 for 20 requests. Anyway for me its more important speed of development than performance on the server. Maybe my servers do not get that many visits.
  • bhauer 14y ago
    Oh this is awesome! Very nice performance for a mobile CPU. Pull request, maybe? :)
• plainOldText 14y ago
  How can you write a web framework benchmark and not include some of the non mainstream languages with probably the most performant frameworks like Erlang (Cowboy, Mochiweb), Haskell (Yesod, Snap Framework)? That's just wrong; anyway.
  • brass9 14y ago
    Oh... and you think Erlang or Haskell frameworks are mainstream while you fail to mention ASP.net?
• lux 14y ago
  I'd love to see how PHP 5.4 compares. In my own app, I saw a noticeable speedup and RAM usage per request dropped by half.
• mixmastamyk 14y ago
  Wasn't it agreed a long time ago that benchmarking against a VM was a bad idea?
  • bhauer 14y ago
    Possibly, although we believe it represents precisely what we wanted to test: a realistic production environment. However, we also tested on our physical i7 hardware. Did you happen to scroll down? :)
    • mixmastamyk 14y ago
      I did, but didn't see a pretty graph for it. Full production results are interesting, but benchmarks should test on a single variable. A cloud VM adds several to the mix.
  • spullara 14y ago
    It is a great idea when your production environment is a VM.
• cloudwizard 14y ago
  It would be nice to see raw Python like you have raw PHP. I would expect Django to perform very poorly unless you optimize it's caching.
• redwolfb14 14y ago
  Wow, this benchmark is like comparing apples, oranges, mangos and cherries. Just because they are all fruit.
  • ldng 14y ago
    It's fine as long as people take it for what it is. I still find the fruit salad interesting :-)
• deleted 14y ago
  [deleted]
• bhauer 14y ago
  In case it's not obvious, you can hide frameworks/platforms of no interest to you to narrow the view. For example, here's Node.js versus Express: http://www.techempower.com/blog/2013/03/28/framework-benchmarks/#hide=1,2,3,0,27,7,30,11,12,13,15,14,17,16,18,19,21,20,22,23,24,25,26,29 http://www.techempower.com/blog/2013/03/28/framework-benchma... (Note the initial chart in the introduction section is an image and it won't be affected.)
• Yaggo 14y ago
  The raw json output merely tells about the performance of the language itself. Nice, but not that interesting if you need to get the job done and don't happen to know 10+ languages/frameworks.
• abimaelmartell 14y ago
  im not a PHP nazi, but you can't compare PHP and a framework
• jrussbowman 14y ago
  wow, these results and the performance of go is making me consider rewriting my tornado app again. I just put off doing it in order to build more stuff in to the python version. Maybe I'll reprioritize.
• codebeard 14y ago
  >And let us simply draw the curtain of charity over the Cake PHP results. Something about that phrase "Let us simply draw the curtain of charity..." resulted in an immediate spittake. Coffee everywhere.
• fatbat 14y ago
  I see alot of people saying how certain configs are missing/better/not tested. So what about a "crowdsourced benchmark" instead? Maybe a few elaborate scenarios are posted and people can simply submit their best setup/config/code to be benchmarked. I imagine devs would improve on it over time and eventually the most optimized would surface?
  • kainsavage 14y ago
    We are doing precisely that: if there are any mistakes or misunderstandings about the production-level best practices, we are accepting pull requests on our github (set up with all these tests, in case you want to run them yourself) to fix them, rerun them, and then report our updated findings.
• sequoia 14y ago
  > This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. I disagree with the implication here (that this is a good point for comparison because "real-world application's performance can only get worse."). Yes it can only get worse but how much worse (per unit of "features") is both significant and unaddressed. This isn't the best example but look at the gap between the top and bottom of the scale in the Database access test (single query) and Database access test (multiple queries) charts: In the first, Gemini is ~340x faster than Cake, in the second, only ~23x faster. There is still a big gap but it closed by an order of magnitude once you stepped past the most trivial possible DB access test. So nodejs or php-raw is faster than cake at a single DB access, but what about when you create a real world scenario with authentication, requirement to be able to update features faster (i.e. use an ORM), env. portability requirement, etc.? It seems to me this would look like a little slower, a little slower, a little slower in the {raw} versions, and already included, already included, already included in Rails or Cake. The full featured frameworks take a lot of their performance penalty up-front, with less of a hit as features are added (maybe? :P). My point is that it's not reasonable to assume that hackernews-benchmarks will actually reflect production use. That said I think the article is cool, and agree that it's good to keep framework authors' feet to the fire regarding performance!
  • campnic 14y ago
    This is completely loaded. Your implication is that the only viable test is a test which exercises all of the functionality of the most feature rich framework. How would that be a)viable and b) meaningful? We know that there is a set of common features and the benchmarks goal is to test least common denominator stuff on the networks. Authentication and portability are not LCD. The argument that they are is capricious. What if we made the requirement be that the framework is a lisp? Now we've completely changed the intent.
    • sequoia 14y ago
      I meant to suggest that comparing php-raw to Rails is apples & oranges- not "you must benchmark in a way that benefits larger frameworks", just "please acknowledge that LCD tests like this inherently cast Railsy frameworks in a bad light." It's like condemning a swiss army knife because it's not as efficient as a fixed blade at cutting apples. Well yeah that's true, but what about when you need to screw a screw or pull a cork? One is a multitool, it doesn't make sense to compare it to a specialized tool unless all you plan to do is cut apples.
• just2n 14y ago
  Quickly taking a glance -- these benchmarks (like most benchmarks) seem like they might be highly misleading. For instance, in the Express example code, they're sending JS objects rather than serializing them to raw data that the socket can just send. Instead, serialization/copying are happening on each request, which is a significant overhead. I don't have domain knowledge over many others, but I suspect a similar problem might exist with others.
• vinay_ys 14y ago
  For json serialization, this benchmark seems to indicate Netty is twice as fast as Golang. Don't think that's right. The golang benchmark code is not exactly equivalent to netty benchmark code. The netty code creates the ObjectMapper once and uses it for all requests, whereas golang code creates the json encoder for every request (enc := json.NewEncoder(w)). Just getting rid of that would make this trivial code so much faster.
• rbanffy 14y ago
  I still believe launching a usable web application next week is preferable to launching a really fast web application 3 months from now, if ever.
  • arkitaip 14y ago
    That's a straw man. Competent programmers should be able to code fast code in a fast manner.
    • rbanffy 14y ago
      Different languages and platforms yield very different productivity levels.
• deleted 14y ago
  [deleted]
• greenlaw110 14y ago
  Why Play benchmark is running on Resin? Shouldn't it be directly on top of Netty?
  • bhauer 14y ago
    Play is not running on Resin. Did we accidentally say that somewhere? If so, we'll correct it.
    • sharedliberty 14y ago
      Here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/play-java https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... The tests were run with: Java OpenJDK 1.7.0_09 Resin 4.0.34 Play 2.1.0
      • bhauer 14y ago
        Whoops! Fixing that up presently. Thanks for the catch.
• pacomerh 14y ago
  Go's performance is great, and hearing good things about it, does any one have any experience on the type of applications this language is meant for?.
• larkinrichards 14y ago
  I roughed up a flask benchmark but I haven't been able to test locally. If you want to help, check out this pull request: https://github.com/TechEmpower/FrameworkBenchmarks/pull/14/files https://github.com/TechEmpower/FrameworkBenchmarks/pull/14/f...
  • bhauer 14y ago
    Awesome, thank you! We'll aim to get this incorporated into a follow up post soon.
• xenonflash 14y ago
  1) The Python version has some basic newbie coding errors. This sort of code is what Python programmers call "Java written in Python". It may be a valid algorithm in Java, but it's the wrong way to do it in Python. Code like this will work, but it will be slow. Depending on the size of "queries", you are potentially allocating gobs of memory in two different places for no reason, and then throwing it away without using it. I wouldn't be surprised if the examples in other languages had similar problems. 2) The JSON serializer in Django 1.4 uses a method which is known to be very slow, but which is easily portable across different platforms and works with older versions of Python. They no doubt included for easy bundling. In a real application you would probably want to simply use the normal JSON serializer from the standard library (which is many times faster). 3) The examples are little more than "hello world". I did some benchmark tests with several Python async frameworks, Pypy, and Node.js for an application I was working on. With small JSON objects there wasn't much difference in performance. Once you started using large JSON objects the performance lines for all versions were indistinguishable from each other. The performance bottlenecks were in libraries, and those standard libraries were all written in 'C', so interpreter versus compiler versus JIT made little difference. 4) The problem with "toy" examples is that in real life there are two performance factors which must be taken into account. Think of as y = mx + b. With a toy example you are probably only measuring "b". With most real life applications it's "m" that matters. There are often different optimization approaches that are best for varying ratios of "b" and "m". You have to know your application intimately and benchmark using data which is realistic for that application. Python has a reputation for being "easy to learn". However, it is "easy" in the sense of being able to hack something together that works without knowing very much. There can be several different ways of doing things and doing it one way versus another way can mean a difference in performance of several orders of magnitude. The same may be true for some of the other languages, but I haven't examined them in enough detail to say.
• lobster_johnson 14y ago
  Some of the choices are unfair in an apples/oranges sort of way: They are testing Rails and Sinatra against Java servlets, but the Ruby equivalent of servlets is not a high-level framework like Rails, but Rack. (Also, the Ruby tests use Passenger, which is probably not a great choice for performance.)
• mythz 14y ago
  Would've loved to see http://servicestack.net http://servicestack.net on this list which has great performance on .NET and Mono: https://github.com/ServiceStack/ServiceStack/wiki/Real-world-performance https://github.com/ServiceStack/ServiceStack/wiki/Real-world... And also maintains .NET's fastest JSON and Text Serializers: http://theburningmonk.com/2011/11/performance-test-json-serializers-part-ii/ http://theburningmonk.com/2011/11/performance-test-json-seri...
  • bhauer 14y ago
    Thanks for the tips on these. We'll add ServiceStack to our to-do list. As you might imagine, that list is getting long as a result of some great community feedback. Pat (pfalls) is diving into the pull requests this morning.
• zeeg 14y ago
  You guys should feel bad about generating such awful benchmarks. How the hell can you compare accessing a MySQL database to accessing a MongoDB database? Its like comparing apples to piles of poop. Also when you're testing things like Django in web requests, you're testing gunicorn, not Django.
• baghali 14y ago
  We need a project similar to TodoMVC [1] for benchmarking different languages and frameworks. http://todomvc.com/ http://todomvc.com/
• dugmartin 14y ago
  I wonder how a cgi binary run from xinetd would perform?
• cdi 14y ago
  What about C++ Web frameworks?
  • bhauer 14y ago
    Don't think we didn't consider it! :) If you are willing to craft a comparable test for a C++ framework, we'd definitely like to get one into the next set of numbers. We especially don't feel confident in our C++ chops to do a C++ framework justice.
• throw_away_acc 14y ago
  Sequelize is one of the slowest (if not the slowest) SQL ORM for Node. node-orm or node-sql are much faster. This detail shows that this kind of benchmark as nice as it reads could misleads since the authors have much experience with JVM based stuff. Thus, the JVM based framework tests seem to be much more optimized or only most performant libs were chosen whilst the other framework tests just lack domain knowledge.
• cogwheel 14y ago
  How much of the difference is due to the random number generator calls? They aren't free or equal cost across platforms by any stretch. I really have a hard time trusting these numbers with how casually calls to RNGs are bandied about...
• a-b 14y ago
  I would love to see Dropwizard in this benchmarks!
• lennydizzy 14y ago
  How much does 'asynchronous' matters for the performance of a web framework? I mean netty, node.js are all async frameworks as I understand, but the source code for Go and Compojure seems was using a thread/process per request model?
• NicoJuicy 14y ago
  I kinda miss asp.net web api (okay, it's on Windows, but it's also a framework :))
• gary4gar 14y ago
  Woah....PHP is faster than Ruby in lot of results? I wonder why is this the case?
• sabe__ 14y ago
  Isn’t it that django and rails has flat curves because they are by nature blocking. So it wouldn’t matter how many request you throw at them as they are limited in how much they can handle at once. And to on top of this compare database requests is meaningless as the blocking nature of the framework itself is the major bottleneck, and not the database. Now, for a normal web application, the largest amount of request would come from static content, or cached content within the web app so the real gain would be a tiny fraction between python/ruby and go/java-based frameworks. That said, if you want to handle static content (images and such) from within your app, or build a java script-centric application with lots of tiny requests, or even persistent once...nor rails or django would do.
• programminggeek 14y ago
  I would like to point out that there is no reason you would benchmark a raw SQL query in PHP, and then only use ActiveRecord on the ruby side. At the very least you could have used the Sequel gem to just build a MySQL query for the Sinatra app. ActiveRecord is going to give you the same performance failure that it did in PHP too.
• richyzhang 14y ago
  JVM can use many cores at the same time node/rails only use one core. The test is totally nonsense. You should show us the cpu usage. If you had done so, you would have found how absurd the mistake you had make.
• egonschiele 14y ago
  Since a few people have asked about Haskell, here are some Haskell benchmarks in comparison: http://www.yesodweb.com/blog/2011/03/preliminary-warp-cross-language-benchmarks http://www.yesodweb.com/blog/2011/03/preliminary-warp-cross-...
  • nickknw 14y ago
    Still, it would be nice to have these results independently verified and put alongside all the rest. What do you think, bhauer? :)
    • bhauer 14y ago
      Our to-do list is getting unwieldy. :) We would love to get this added, though. Any chance a Haskell expert could create a test and submit it as a pull request?
      • nickknw 14y ago
        It sure sounds like it is! I'm a far cry from a Haskell expert unfortunately, so I'll have to leave it to someone else.
• iamjem 14y ago
  These sort of "benchmarks" are so subjective that they really don't provide any value. Looking at the framework lineup, a lot of the frameworks at the bottom have so many (tightly coupled) features (which also lead to bloat, like excessive middleware layers) that they're obviously not going to perform anywhere near as well as some of the barebones frameworks at the top. Neat post, but yeah, zero value.
• superuser2 14y ago
  It's interesting to note the time per request when you invert these. Even some of the most drastic differences (like Spring vs. Django) turn out to be the difference of a millisecond or two. That's pocket change in terms of user experience.
  • jonathanhefner 14y ago
    It is a good and interesting observation, but as the benchmarks approach real-world scenarios, the difference is more than a millisecond or two. For instance, if you're running on EC2 hardware (which is common enough) and you're executing ~20 DB queries/request (which is probably, unfortunately, common enough), the difference between Java servlets and Rails is more like 10ms. Then what happens when more than 89 real-world users start to hit your Rails server each second? (Note: I really like Ruby and Rails. Much more so than Java and its offerings.)
• lucian303 14y ago
  You confuse framewos, languages, and server technologies.
• bjouhier 14y ago
  node.js is single threaded and EC2 large instance has 4 CPUs. So to be fair you must either: a) bench on a single core processor (a small EC2 instance) b) configure node.js as a cluster with as many instances as processors.
• sGrabber 14y ago
  Thanks. We are reconsidering our stack and are in middle of evaluation, I am sure this post is going to be of great help.
• philliphaydon 14y ago
  Needs more NancyFX + Mono! Nice post tho, very interesting.
• mschuster91 14y ago
  You're using PDO in the PHP raw test - use direct mysqli_* API, it's lot faster than the PDO abstraction layer.
• purplefox 14y ago
  I haven't looked at this detail yet, but it should be noted that we haven't even optimised Vert.x yet so there should be plenty of scope for further improvement :) (Disclaimer: I'm the Vert.x project lead)
• eamodeorubio 14y ago
  Hi, great work! I only have two suggestions: a) Please, publish not only the requests per second, but the memory and CPU usage of the machine for each framework. b) For JAVA systems, can you publish de heap configuration of the JVM? Cheers!
• HallofFamer 14y ago
  Regarding PHP, well, isnt CakePHP by far the slowest major PHP framework? Why dont you test the other PHP frameworks? How about Yii, Codeigniter, Lithium and Symfony? Id say the result is kinda biased for PHP frameworks since you pick the slowest PHP framework available, instead of the fastest or at least the average one.
• HallofFamer 14y ago
  Also I dont think evaluating json based on a simple 'Hello World' is a valid test. In fact, 'hello world' should never be the standard to test the speed of any processes. If all your script does is to output 'hello world' to the screen, why bother using a server-side programming language at all? You are better off just writing it in HTML, its faster than using any server-side and client-side languages lol.
• deleted 14y ago
  [deleted]
• kenyee 14y ago
  In the FAQ for the benchmarks, could you also add a bit about how you configured the JVM (OpenJDK?). Curious about heap settings, etc.? Did you just use the default install? e.g. http://planet.jboss.org/post/rhel_openjdk_performance_tuning http://planet.jboss.org/post/rhel_openjdk_performance_tuning http://www.jaspersoft.com/sunopenjdk-jvm-garbage-collection-tuning-tutorial http://www.jaspersoft.com/sunopenjdk-jvm-garbage-collection-...