15 ms·

Frameworks Round 6

125 points by amarsahinovic 13y ago

bhauer 13y ago
Thanks for posting this, amarsahinovic. This is the latest round of our ongoing project to benchmark the performance of fundamental services provided by modern web application frameworks. With the continued contributions of the developer community, the number of frameworks and platforms included is now 74! Round 6 also adds the often-requested plaintext test with HTTP pipelining (thanks in large part to William Glozer of Wrk fame). The results web site also now includes a "framework overhead" tab that allows you to compare a framework versus its underlying platform. This was originally conceived by a Hacker News user named goodwink [1]. Like the round before, Round 6 gave us some technical challenges, including a period where we were unable to collect any latency data. Today, some problems persist, such as the Dart test (yes, there is server-side Dart) failing to complete for most of the tests. However, rather than continue to defer Round 6, we've decided it's better to move forward and use the round-over-round progression to iterate, fix, and improve. No problem that exists today is set in stone. To that end, as always, I would love to hear any feedback, advice, questions, criticism, and especially contributions. And we hope you find this information helpful! [1] https://news.ycombinator.com/item?id=5454970 https://news.ycombinator.com/item?id=5454970
- simonz05 13y ago
  Just want to say thanks for doing these tests. I think this is the first time we have such a thorough overview of the basic performance of web frameworks.
- jcampbell1 13y ago
  Luminus appears to be a clojure framework, not a php framework. Thanks again for posting these tests.
  - bhauer 13y ago
    Oops! My mistake in classifying. Many apologies to fans of Luminus. I've just corrected this.
- eranation 13y ago
  One request: I can't help but wonder how JAX-RS / Jersey on Tomcat / Glassfish perform. JAX-RS on Java EE is a serious contender to Spring, and I really want to know how a JAX-RS service (Servlet based) is compared to Spring and to raw Servlets.
- ditados 13y ago
  are you going to include gevent (I.e. bottle/flask + gunicorn with gevent workers, and uwsgi + gevent workers) ?
  - bhauer 13y ago
    We would certainly like to include that configuration. If you have time and are willing to contribute, we'd love to receive that as a pull request.
columbo 13y ago
I'm really surprised at the performance of C#... honest no-flame question: how is possible it that Python is beating C#? I've been out of the .NET world for several years, almost a decade, but I always assumed they would be at Java level performance. edit add: This is really fantastic work
- bhauer 13y ago
  Some caveats to point out: * Round 6 does not yet use SQL Server. A contributor has provided SQL Server test implementations, but we don't expect to include that until Round ~7 because it will still require some additional work. It is conceivable that the .NET MySQL driver is especially poor. * We do not yet have a test that is expressly heavy computation with no database connectivity. I suspect that C# and .NET would perform reasonably well on such a test (isolating it from its database connectivity). * We do not yet have Windows i7 tests. But performance would likely scale about the same as seen between Linux EC2 to Linux i7. All that said, it's also possible that the ASP.NET implementations of our tests could benefit from additional tuning. We have had a couple subject matter experts give contributions but we'd be happy to receive more.
- iaskwhy 13y ago
  I guess you need to change the Hardware to "Win" to see it shining.
  - TheAnimus 13y ago
    Indeed, mono performance isn't great. That is the problem with C#, you need ultimately to run it on IIS for web platform deployment, sadly under anything else it is rather slow. Also these tests are a bit unfair, as I think they give benefit in some areas to frameworks that cache by default, rather than by request. Ultimately, what I want to know is total cost, that means hardware, licenses / support agreements, development time, developer ease of hiring. That isn't to say these aren't interesting, but they are rather apples to oranges. As I posted in the other flamebate thread about MS stack > *, I have set a note in my calandar to help improve the example for MVC.Net (https://news.ycombinator.com/item?id=5976540 https://news.ycombinator.com/item?id=5976540). Hopefully I will be able to come then when not in an exclusive agreement, they suck I know, but it pays my mortgage!
deelowe 13y ago
Thanks for continuing to do these. As far as I know, no one else is doing something like this in such a comprehensive way.
rip747 13y ago
i would really like to see how CFML engines (Railo, Adobe ColdFusion) and frameworks (FW/1, CFWheels, ColdBox) stack up in this comparison. FD: I contribute to CFWheels
- bhauer 13y ago
  We'd love to receive these test cases as pull requests if you have the time to contribute them. :) https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
  - rip747 13y ago
    checking out the documentation now. is there a google group or some place I can post questions to if I need help?
    - bhauer 13y ago
      Indeed! https://groups.google.com/forum/?fromgroups=#!forum/framework-benchmarks https://groups.google.com/forum/?fromgroups=#!forum/framewor...
clhodapp 13y ago
The play-slick results seem to be missing. Was there some sort of problem?
- bhauer 13y ago
  I had not named it uniquely. I've changed its name to "play-slick." Thanks for pointing that out!
  - drummingds 13y ago
    Thanks for updating that. I noticed that the Slick version seems to outperform the regular Anorm version in most tests. Do you have any insight into why that might be? I figured it would've been the other way around.
npalli 13y ago
Does anyone have more info on "cpoll-cppsp"? Hearing about it for the first time, it seems to be smoking all the other frameworks.
- ambiate 13y ago
  I was actually just checking it out for myself. http://sourceforge.net/projects/cpollcppsp/ http://sourceforge.net/projects/cpollcppsp/ https://github.com/xaxaxa/workspace/ https://github.com/xaxaxa/workspace/ Seems to be a C++ framework with dynamics in the form of C++ code using an ASP style for embedding into the resource.
- deleted 13y ago
  [deleted]
camus 13y ago
Nice, Some solutions are quite fast , but as soon as the database needs to perform heavy stuffs , well the perfs decrease dramatically. The really bottleneck seems to be the database in most cases. However ,some frameworks are basically quite slow.
Historiopode 13y ago
Wonderful, thanks for the continuous updates. I hope you will keep gaining momentum around this initiative; having a solid community-supported set of benchmarks which routinely improves would be a true asset. By the way, I would like to offer a UX suggestion: it would be nice if you could select one or more frameworks and have them highlighted in the result listings, so as to be able to notice their relative position at a glance.
- bhauer 13y ago
  Thanks for the feedback. The highlight is a good idea and I've created an issue for the suggestion [1]. I'll try to get that added in time for the next round. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/359 https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...
VeejayRampay 13y ago
For such a young language, I'm really surprised by the performance of Go there. Quite awesome.
- YcombRegBroken 13y ago
  Round 4 of these tests is what triggered my dive into Go this past month, and it has been a revelation. If you add in the concurrency capabilities, the language simplicity, and nice balance between overly terse syntax (Python one liners...) and overly verbose typed languages (Java), the whole package is even more impressive. The only real thing I think it is missing is a "high productivity" application framework like Rails to elevate it from "damn, I'm impressed" to "it would be ill advised to use almost anything else". </reluctant fanboy rave>
  - bliker 13y ago
    how about http://robfig.github.io/revel/ http://robfig.github.io/revel/ ? I just skimmed thru their manual and it seems quite promising
    - h4pless 13y ago
      I was just looking at Revel too. But from the homepage: Development Status: Early adopters only. Pull requests welcome. Development is closing in on the "final" 1.0 design, but the rate of change is still high. Expect to get your hands dirty. Wouldn't quite appeal for people looking for a Go based Rails yet, but I would have to agree.. it does look very promising.
    - landr0id 13y ago
      Revel is pretty great so far (I actually just made my first contribution yesterday!). Some important things it needs though: support for HTTPS, better DB support (some type of ORM, although gorp is sufficient for now), HTTP auth, and better template engine (currently uses the built-in Go "template" package which is meh). Those are the major things that come to my mind.
bliker 13y ago
Honest question: Why are most of the frameworks that I know of in bottom half? When does speed becomes relevant?
- RyanZAG 13y ago
  Three reasons as to the hierarchy: 1) Most popular frameworks you would encounter run on higher level dynamic languages such as PHP, Ruby or Python. These languages are a decent bit slower than performance optimized languages. The languages towards the top are all C, Java, Go, etc. 2) The frameworks towards the top are generally lower level - they do less automatic 'magic' than the more popular frameworks that handle things like validation automatically. 3) In general, it's more likely you will have heard of older frameworks than newer ones. The frameworks towards the top are generally all less than 2 years old (excluding Java servlets which have just always been decent). This means they make use of newer designs such as non-locking and asynchronous connection handling. When does speed become relevant? It's actually always relevant as soon as your product gets traction. For example, 1 server at the top of that chart may be able to handle the same load as 10 servers towards the bottom. If you're using a framework at the bottom and paying heavily for 10 servers, you could increase your startups runway substantially by swapping. However, swapping may cost more in engineer-hours than the servers. Basically, the relevance of speed is directly related to your margin. If your margin is high (investment banking maybe?), then speed is almost irrelevant, you can afford a whole building of servers. If you're competing heavily on mass market web apps, speed is fairly critical as you can out price competition.
  - bhauer 13y ago
    Great response, RyanZAG. To add to this, I'd encourage you to look at the source code used to implement the tests in the higher performance frameworks. Ryan's third point is very relevant: you may find that modern frameworks leverage both high-performance platforms and much of the pragmatic thinking that was introduced a few years back in the frameworks with which you are more familiar. Source code link: https://github.com/TechEmpower/FrameworkBenchmarks/ https://github.com/TechEmpower/FrameworkBenchmarks/
  - candybar 13y ago
elsurudo 13y ago
I would love to see how the Goliath async server (Ruby version of Node.js is one way of looking at it - http://goliath.io http://goliath.io) performs, as I've recently started looking into it for a part of a project.
- ksec 13y ago
  I was just about to post this. And will the next round be tested on Rails 4.0?
  - mhixson 13y ago
    The best way to ensure Goliath is tested is to submit a pull request. We'd rather test Rails 4 than obsolete versions of Rails: https://github.com/TechEmpower/FrameworkBenchmarks/issues/352 https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...
dom96 13y ago
Finally! I was disappointed when this round was postponed. I couldn't wait to see how well my framework (Jester; https://github.com/dom96/jester https://github.com/dom96/jester) performed, not only because it was my framework but also because it is written in Nimrod. I am a bit disappointed by the results but at least there is a lot of room to improve now, I didn't have much time to properly implement concurrency for Jester so I opted for just spawning a couple of processes which is definitely not ideal. I hope that Round 7 will go better for Jester. In the meantime I hope that you will check out the Nimrod programming language (http://nimrod-code.org http://nimrod-code.org) despite the results that my framework achieved. To the team that carried out these benchmarks, thank you!
- bhauer 13y ago
  Finally indeed! In future rounds, I hope that we can spend less time trying to resolve problems--if there are problems, we'll just post the results and move on to the next round. Better to keep iterating, and allow the experts who work with each framework resolve issues as they come up. Thanks very much for contributing Jester to the test! I really appreciate your spirit and hope to see it improve in Round 7 as well. I'm looking forward to seeing the next pull request. :)
  - dom96 13y ago
    Indeed, although people may be disappointed if their framework fails and they have to wait another month for another round of results, but then I suppose the next round will come a lot quicker if you have less issues to fix. No problem at all. Thank you for merging my process spawning silliness. You will definitely see a lot more improvements from me :) EDIT: A little issue I just noticed: the jester "Front-end server" is shown as "None" when in fact it's "nginx".
    - MetaCosm 13y ago
      I think they would be less disappointed if it meant a consistent schedule. These benchmarks are really a lot of fun, not sure if they change hearts and minds, but they are beautiful and fun. I think being time-boxed rather than implementation-boxed make sense. If you miss a month, you got 30 days to fix it. Also, they could only "update" the test rules quarterly or every 6 months, give the authors a bit of time to respond.
neya 13y ago
An honest confession from someone inspired by these comparisons. I've been following these benchmarks very closely right from round one and ever since I've been waiting to see my favorite framework, Ruby on Rails perform decently to some extent. I waited till the last round to see some improvements and while I DID see some marginal improvements, it wasn't as expected. This then provoked me to do some basic math. Imagine a real world startup scenario wherein you need to your application to deliver API responses (JSON) or even ordinary HTTP responses. If you use a framework from the bottom of the list (like Rails), then you may be able to serve, say, 'X' customers on your site using your application at a time. Now, if you use something from the top of the list, you may be able to serve roughly about 20 times (20X) more number of customers on a single server. This not only means less cost, it also means you can worry about scaling and stuff much much later than you would have to, unlike on a low performant framework. For this reason, I decided to conduct an experiment for myself. Pick up a framework from the top of the benchmarks and experiment with the learning curve and the language. Well, after carefully evaluating, I decided JAVA wasn't for me (too much of a beast). But Scala seemed to hit the sweet spot with excellent language (1 line of Scala = 10 lines of Java on an average) features along with the trust of the JVM.[1] So, I chose Scala and started experimenting with it. I started with the play framework and was not fond of certain decisions that were made within the framework. Also, the play framework is nothing like Ruby on Rails (but it claims to be so) and was more of self-serve and also quite heavy for my taste. Hence, I went even farther up the list and chose Scalatra. Generally, framework selection is a matter of personal preference and each one should choose whatever aligns with their philosophy. For me, it was scalatra. Frankly, I haven't been able to ship out features at the speed at which I used to, with Rails, but there was something very important that I learnt too - Don't get comfortable with full stack frameworks without understanding what goes underneath. Rails suddenly seemed like an enemy to me because it was doing everything so magically that I literally had NO IDEA until I had to implement them myself with Scalatra. For example, do you know what AES-GCM is? Do you know HOW the secure cookie store in Rails works? What algorithm it uses?? Do you know you have to change the secret pretty often??? Do you know how authentication in Rails (Devise) works??? Can you build it from scratch???? I knew none of these, until I had to implement them myself for my application on top of Scalatra. It was seriously a pain because for the first one week, I did absolutely nothing, in terms of progress with features. But later, I started loving this way of development rather than relying on 'magic' frameworks. Now 'magic' seems scary for me because I cannot actually see what's happening underneath. So, to cut a long story short, Should you choose the fastest framework? Or should you follow the 'ship it first' policy with an average framework?? My advice - Start with a full stack framework, something like Rails or Django, but also in parallel, try to understand how every bit of your framework works and try to implement it into a micro framework based on something from the top of the benchmarks (like Go, Scala, etc.). Most importantly LEARN. Something new everyday! And slowly shift your development towards these high performant frameworks as and when you see fit. This is just based on my experience :) [1] 1 JVM = 10 Thins, Source: http://www.slideshare.net/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm http://www.slideshare.net/pcalcado/from-a-monolithic-ruby-on...
continuations 13y ago
cpoll-cppsp made huge performance gains from round 5 to 6. For example multiple queries went from 1,872 rps in round 5 to 7,252 in round 6. What caused that? Also what happened to cpoll-cppsp in the plaintext test? Its performance there took a deep dive.
- bhauer 13y ago
  I know that the contributor of those tests made several changes. We weren't able to identify why it fails dramatically on the plaintext test, but I suspect it's a combination of the two unique characteristics of that test: the use of HTTP pipelining and higher client-side concurrency. My guess is that the pipelining in particular is causing problems.
  - MetaCosm 13y ago
    I really hope you guys start doing more in-depth high-concurrency testing. This is a huge risk / problem with many frameworks and kits. It seems to be a blackhole in your tests for some reason, which is a shame because the are so well done in other regards. Having a framework which falls down horribly at high concurrency is something people need to know about -- in how they structure and design applications and how they structure deploys. It can also help them pick frameworks that fit them better (some frameworks will do great at high concurrency, some won't). Can't wait till you start adding high concurrency (which should NOT require any code changes from the implementation providers, you add a zeroes to the concurrency.. 100, 1000, 10000 tests).
- joeblau 13y ago
  Is there any reason why it is so slow on plain text?
  - bhauer 13y ago
    Looks like there was a problem in cpoll-cppsp: https://github.com/TechEmpower/FrameworkBenchmarks/pull/364 https://github.com/TechEmpower/FrameworkBenchmarks/pull/364 This will be merged in for Round 7.
antihero 13y ago
How are certain languages so much faster even when pulling from the database? Surely the database would be the bottleneck here? Surely if it's something cool the driver is doing, that could be implemented in other drivers? Also, why are some frameworks on there a bunch of times (e.g. ASP MVC)?
- continuations 13y ago
  > Surely the database would be the bottleneck here? The idea that "it doesn't matter which language you use because DB would be your bottleneck" might have been true 10 years ago but it's definitely not true anymore. Latest versions of MySQL and Postgresql are extremely fast. They are written in C/C++ and heavily optimized. MySQL serving data from RAM can easily do 500K+ rps on a commodity server. Even if you need to go to disks there are $200 SSD's that give you almost 100K IOPS. Put a few of them in a RAID and you're getting hundreds of thousands of IOPS for very little money. In all likelihood application code written in Ruby/Python/PHP with frameworks piled on top of abstractions is going to be orders of magnitude slower than the DB server written in C that has undergone 15+ years of heavy optimizations.
  - antihero 13y ago
    Ok, but say we use vanilla Python WSGI with a psycopg2 (C Python PostgreSQL extension) directly. Will this come close to the performance of the C++ framework thing?
    - continuations 13y ago
      I'd say a vanilla Python application without ORM would perform similar to a flask app without ORM. According to http://www.techempower.com/benchmarks/#section=data-r6&hw=i7&test=db http://www.techempower.com/benchmarks/#section=data-r6&hw=i7... flask (on Gunicorn): 6,138 flask (on PyPy): 8,167 Django + ORM : 4,026 cpoll-cppsp : 114,711 So while a light weight Python app without ORM might be 50% faster than a full-blown Django app, it's still almost 20 times slower than a C++ app. That shouldn't be surprising, by all accounts Python is at least an order of magnitude slower than C++. You could speed up your Python program by running PyPy. According to the benchmark that gives it a 30% speed boost, which is in line with my experience. Another way is to deploy on uWSGI or Meinheld instead of Gunicorn. No matter what you do, Python will not be close to the speed of C++. But what really matters is: is it fast enough for your need?
coolsunglasses 13y ago
I've followed yogthos' work on Luminus since the beginning. It doesn't merit mention independently of Compojure. It's just some sane defaults for pedagogical reasons for getting started writing web apps on top of Compojure. It adds nor removes no runtime or anything else. As it currently exists, it's a Leiningen project template generator. That's about it. Luminus is isomorphic, for the purposes of a benchmark, with Compojure. I'm sure yogthos will be flattered when I tell him his web app templating thingy was included though.
- bhauer 13y ago
  Sunglasses! Good to see a familiar name. Incidentally, yogthos should be aware that Luminus was included: https://github.com/TechEmpower/FrameworkBenchmarks/pull/293 https://github.com/TechEmpower/FrameworkBenchmarks/pull/293
cmccabe 13y ago
What about Cobol on COGS?
ditados 13y ago
i don't see any mention of uwsgi or gevent in these benchmarks. Deploying Python apps without properly configuring the container/webserver will skew the results considerably (probably applicable to a couple of other entries as well).
evab 13y ago
Can anyone explain why the "gemini" framework with an ORM outperforms raw servlets/queries in almost all the cases? What kind of optimizations are done ?
- bhauer 13y ago
  Evab, Gemini is our in-house framework [1] that we've included in these tests for our own learning experience. We're not obsessed with performance, but we do try to avoid problems that are easily avoided. :) For instance, where possible, we have used Java's concurrent data structures and reduced the use of locks. A difference in these tests is that we have our own connection pool. Speed was not a main objective when we wrote it--it's so old that if my memory is correct, it predates the availability of the connection pool that now ships with the MySQL driver, which is what we've used in the Servlet tests. Nevertheless, my conjecture is that perhaps it's just a tad quicker at selecting an idle connection. Incidentally, for a while in an earlier round, the Go guys had an alternate test implementation that used a concurrent queue for connections and it performed fantastically. We've been toying with changing Gemini to do the same at some point. For the benchmarks project, we classified ORMs as "raw" (no ORM), micro, or full. In Gemini, we have a micro ORM. The following are examples of its use in the test code [2]: // Get all rows from Fortunes table. final List<Fortune> fortunes = store.list(Fortune.class); // Get a random World row. worlds[i] = store.get(World.class, random.nextInt(DB_ROWS) + 1); // Run a batch of updates from a List of entity objects. store.putAll(Arrays.asList(worlds)); "Micro ORM" is loosely defined, but we've applied it to ORM options that offer some abstraction over plain old SQL, but aren't as comprehensive as the standard-bearers. For example, while we have a data structure that represents relationships, we do not have a higher-level query language that traverses relationships automatically nor any similar mechanism for easy relationship traversal. It turns out that across all of these frameworks, there are dozens of what we are calling Micro ORMs. Getting back to your question, the Gemini ORM does leverage a few tricks for performance such as ReflectASM for object creation and prepared statements for queries. But there isn't a whole lot of fancy work going on. It's not glamorous, but the most important high-performance elements we make use of in Gemini are the JVM platform and Java's concurrent data structures. As demonstrated by the likes of Undertow (the web server component of JBoss WildFly), mind-boggling performance is available on the JVM. [1] https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg-Ibk/fOjQMxESEJ0J https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg... [2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
mark_l_watson 13y ago
Disturbing results since I so frequently have used Sinatra for hosting REST APIs for customer projects. Rethink needed. For my own projects I use Clojure + Compojure and more recently added Node.js (often with Meteor) and was pleased to see that they benchmark well. One thing I especially like about Compojure with the Hiccup HTML generating library is that I see syntax errors immediately in the editor. I suppose that Scalatra with embedded HTML would provide the same benefit.
- bemurphy 13y ago
  Maybe benchmark it yourself? I think something is amiss, it should easily 'hello, world' at 2x higher rate than Rails. Poked around the setup a little a while back but nothing jumped out. Hunches: it's either going through webrick or not running with RACK_ENV production
  - bhauer 13y ago
    Unicorn, nginx, production environment. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/sinatra/setup_ruby.py#L18 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

125 points by amarsahinovic 13y ago

Thanks for posting this, amarsahinovic. This is the latest round of our ongoing project to benchmark the performance of fundamental services provided by modern web application frameworks. With the continued contributions of the developer community, the number of frameworks and platforms included is now 74! Round 6 also adds the often-requested plaintext test with HTTP pipelining (thanks in large part to William Glozer of Wrk fame). The results web site also now includes a "framework overhead" tab that allows you to compare a framework versus its underlying platform. This was originally conceived by a Hacker News user named goodwink [1]. Like the round before, Round 6 gave us some technical challenges, including a period where we were unable to collect any latency data. Today, some problems persist, such as the Dart test (yes, there is server-side Dart) failing to complete for most of the tests. However, rather than continue to defer Round 6, we've decided it's better to move forward and use the round-over-round progression to iterate, fix, and improve. No problem that exists today is set in stone. To that end, as always, I would love to hear any feedback, advice, questions, criticism, and especially contributions. And we hope you find this information helpful! [1] https://news.ycombinator.com/item?id=5454970 https://news.ycombinator.com/item?id=5454970

simonz05 13y ago
Just want to say thanks for doing these tests. I think this is the first time we have such a thorough overview of the basic performance of web frameworks.
jcampbell1 13y ago
Luminus appears to be a clojure framework, not a php framework. Thanks again for posting these tests.
- bhauer 13y ago
  Oops! My mistake in classifying. Many apologies to fans of Luminus. I've just corrected this.
eranation 13y ago
One request: I can't help but wonder how JAX-RS / Jersey on Tomcat / Glassfish perform. JAX-RS on Java EE is a serious contender to Spring, and I really want to know how a JAX-RS service (Servlet based) is compared to Spring and to raw Servlets.
ditados 13y ago
are you going to include gevent (I.e. bottle/flask + gunicorn with gevent workers, and uwsgi + gevent workers) ?
- bhauer 13y ago
  We would certainly like to include that configuration. If you have time and are willing to contribute, we'd love to receive that as a pull request.

columbo 13y ago

I'm really surprised at the performance of C#... honest no-flame question: how is possible it that Python is beating C#? I've been out of the .NET world for several years, almost a decade, but I always assumed they would be at Java level performance. edit add: This is really fantastic work

bhauer 13y ago
Some caveats to point out: * Round 6 does not yet use SQL Server. A contributor has provided SQL Server test implementations, but we don't expect to include that until Round ~7 because it will still require some additional work. It is conceivable that the .NET MySQL driver is especially poor. * We do not yet have a test that is expressly heavy computation with no database connectivity. I suspect that C# and .NET would perform reasonably well on such a test (isolating it from its database connectivity). * We do not yet have Windows i7 tests. But performance would likely scale about the same as seen between Linux EC2 to Linux i7. All that said, it's also possible that the ASP.NET implementations of our tests could benefit from additional tuning. We have had a couple subject matter experts give contributions but we'd be happy to receive more.
iaskwhy 13y ago
I guess you need to change the Hardware to "Win" to see it shining.
- TheAnimus 13y ago
  Indeed, mono performance isn't great. That is the problem with C#, you need ultimately to run it on IIS for web platform deployment, sadly under anything else it is rather slow. Also these tests are a bit unfair, as I think they give benefit in some areas to frameworks that cache by default, rather than by request. Ultimately, what I want to know is total cost, that means hardware, licenses / support agreements, development time, developer ease of hiring. That isn't to say these aren't interesting, but they are rather apples to oranges. As I posted in the other flamebate thread about MS stack > *, I have set a note in my calandar to help improve the example for MVC.Net (https://news.ycombinator.com/item?id=5976540 https://news.ycombinator.com/item?id=5976540). Hopefully I will be able to come then when not in an exclusive agreement, they suck I know, but it pays my mortgage!

deelowe 13y ago

Thanks for continuing to do these. As far as I know, no one else is doing something like this in such a comprehensive way.

rip747 13y ago

i would really like to see how CFML engines (Railo, Adobe ColdFusion) and frameworks (FW/1, CFWheels, ColdBox) stack up in this comparison. FD: I contribute to CFWheels

bhauer 13y ago
We'd love to receive these test cases as pull requests if you have the time to contribute them. :) https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks
- rip747 13y ago
  checking out the documentation now. is there a google group or some place I can post questions to if I need help?
  - bhauer 13y ago
    Indeed! https://groups.google.com/forum/?fromgroups=#!forum/framework-benchmarks https://groups.google.com/forum/?fromgroups=#!forum/framewor...

clhodapp 13y ago

The play-slick results seem to be missing. Was there some sort of problem?

bhauer 13y ago
I had not named it uniquely. I've changed its name to "play-slick." Thanks for pointing that out!
- drummingds 13y ago
  Thanks for updating that. I noticed that the Slick version seems to outperform the regular Anorm version in most tests. Do you have any insight into why that might be? I figured it would've been the other way around.

npalli 13y ago

Does anyone have more info on "cpoll-cppsp"? Hearing about it for the first time, it seems to be smoking all the other frameworks.

ambiate 13y ago
I was actually just checking it out for myself. http://sourceforge.net/projects/cpollcppsp/ http://sourceforge.net/projects/cpollcppsp/ https://github.com/xaxaxa/workspace/ https://github.com/xaxaxa/workspace/ Seems to be a C++ framework with dynamics in the form of C++ code using an ASP style for embedding into the resource.
deleted 13y ago
[deleted]

camus 13y ago

Nice, Some solutions are quite fast , but as soon as the database needs to perform heavy stuffs , well the perfs decrease dramatically. The really bottleneck seems to be the database in most cases. However ,some frameworks are basically quite slow.

Historiopode 13y ago

Wonderful, thanks for the continuous updates. I hope you will keep gaining momentum around this initiative; having a solid community-supported set of benchmarks which routinely improves would be a true asset. By the way, I would like to offer a UX suggestion: it would be nice if you could select one or more frameworks and have them highlighted in the result listings, so as to be able to notice their relative position at a glance.

bhauer 13y ago
Thanks for the feedback. The highlight is a good idea and I've created an issue for the suggestion [1]. I'll try to get that added in time for the next round. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/359 https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...

VeejayRampay 13y ago

For such a young language, I'm really surprised by the performance of Go there. Quite awesome.

YcombRegBroken 13y ago
Round 4 of these tests is what triggered my dive into Go this past month, and it has been a revelation. If you add in the concurrency capabilities, the language simplicity, and nice balance between overly terse syntax (Python one liners...) and overly verbose typed languages (Java), the whole package is even more impressive. The only real thing I think it is missing is a "high productivity" application framework like Rails to elevate it from "damn, I'm impressed" to "it would be ill advised to use almost anything else". </reluctant fanboy rave>
- bliker 13y ago
  how about http://robfig.github.io/revel/ http://robfig.github.io/revel/ ? I just skimmed thru their manual and it seems quite promising
  - h4pless 13y ago
    I was just looking at Revel too. But from the homepage: Development Status: Early adopters only. Pull requests welcome. Development is closing in on the "final" 1.0 design, but the rate of change is still high. Expect to get your hands dirty. Wouldn't quite appeal for people looking for a Go based Rails yet, but I would have to agree.. it does look very promising.
  - landr0id 13y ago
    Revel is pretty great so far (I actually just made my first contribution yesterday!). Some important things it needs though: support for HTTPS, better DB support (some type of ORM, although gorp is sufficient for now), HTTP auth, and better template engine (currently uses the built-in Go "template" package which is meh). Those are the major things that come to my mind.

bliker 13y ago

Honest question: Why are most of the frameworks that I know of in bottom half? When does speed becomes relevant?

RyanZAG 13y ago
Three reasons as to the hierarchy: 1) Most popular frameworks you would encounter run on higher level dynamic languages such as PHP, Ruby or Python. These languages are a decent bit slower than performance optimized languages. The languages towards the top are all C, Java, Go, etc. 2) The frameworks towards the top are generally lower level - they do less automatic 'magic' than the more popular frameworks that handle things like validation automatically. 3) In general, it's more likely you will have heard of older frameworks than newer ones. The frameworks towards the top are generally all less than 2 years old (excluding Java servlets which have just always been decent). This means they make use of newer designs such as non-locking and asynchronous connection handling. When does speed become relevant? It's actually always relevant as soon as your product gets traction. For example, 1 server at the top of that chart may be able to handle the same load as 10 servers towards the bottom. If you're using a framework at the bottom and paying heavily for 10 servers, you could increase your startups runway substantially by swapping. However, swapping may cost more in engineer-hours than the servers. Basically, the relevance of speed is directly related to your margin. If your margin is high (investment banking maybe?), then speed is almost irrelevant, you can afford a whole building of servers. If you're competing heavily on mass market web apps, speed is fairly critical as you can out price competition.
- bhauer 13y ago
  Great response, RyanZAG. To add to this, I'd encourage you to look at the source code used to implement the tests in the higher performance frameworks. Ryan's third point is very relevant: you may find that modern frameworks leverage both high-performance platforms and much of the pragmatic thinking that was introduced a few years back in the frameworks with which you are more familiar. Source code link: https://github.com/TechEmpower/FrameworkBenchmarks/ https://github.com/TechEmpower/FrameworkBenchmarks/
- candybar 13y ago

elsurudo 13y ago

I would love to see how the Goliath async server (Ruby version of Node.js is one way of looking at it - http://goliath.io http://goliath.io) performs, as I've recently started looking into it for a part of a project.

ksec 13y ago
I was just about to post this. And will the next round be tested on Rails 4.0?
- mhixson 13y ago
  The best way to ensure Goliath is tested is to submit a pull request. We'd rather test Rails 4 than obsolete versions of Rails: https://github.com/TechEmpower/FrameworkBenchmarks/issues/352 https://github.com/TechEmpower/FrameworkBenchmarks/issues/35...

dom96 13y ago

Finally! I was disappointed when this round was postponed. I couldn't wait to see how well my framework (Jester; https://github.com/dom96/jester https://github.com/dom96/jester) performed, not only because it was my framework but also because it is written in Nimrod. I am a bit disappointed by the results but at least there is a lot of room to improve now, I didn't have much time to properly implement concurrency for Jester so I opted for just spawning a couple of processes which is definitely not ideal. I hope that Round 7 will go better for Jester. In the meantime I hope that you will check out the Nimrod programming language (http://nimrod-code.org http://nimrod-code.org) despite the results that my framework achieved. To the team that carried out these benchmarks, thank you!

bhauer 13y ago
Finally indeed! In future rounds, I hope that we can spend less time trying to resolve problems--if there are problems, we'll just post the results and move on to the next round. Better to keep iterating, and allow the experts who work with each framework resolve issues as they come up. Thanks very much for contributing Jester to the test! I really appreciate your spirit and hope to see it improve in Round 7 as well. I'm looking forward to seeing the next pull request. :)
- dom96 13y ago
  Indeed, although people may be disappointed if their framework fails and they have to wait another month for another round of results, but then I suppose the next round will come a lot quicker if you have less issues to fix. No problem at all. Thank you for merging my process spawning silliness. You will definitely see a lot more improvements from me :) EDIT: A little issue I just noticed: the jester "Front-end server" is shown as "None" when in fact it's "nginx".
  - MetaCosm 13y ago
    I think they would be less disappointed if it meant a consistent schedule. These benchmarks are really a lot of fun, not sure if they change hearts and minds, but they are beautiful and fun. I think being time-boxed rather than implementation-boxed make sense. If you miss a month, you got 30 days to fix it. Also, they could only "update" the test rules quarterly or every 6 months, give the authors a bit of time to respond.

neya 13y ago

An honest confession from someone inspired by these comparisons. I've been following these benchmarks very closely right from round one and ever since I've been waiting to see my favorite framework, Ruby on Rails perform decently to some extent. I waited till the last round to see some improvements and while I DID see some marginal improvements, it wasn't as expected. This then provoked me to do some basic math. Imagine a real world startup scenario wherein you need to your application to deliver API responses (JSON) or even ordinary HTTP responses. If you use a framework from the bottom of the list (like Rails), then you may be able to serve, say, 'X' customers on your site using your application at a time. Now, if you use something from the top of the list, you may be able to serve roughly about 20 times (20X) more number of customers on a single server. This not only means less cost, it also means you can worry about scaling and stuff much much later than you would have to, unlike on a low performant framework. For this reason, I decided to conduct an experiment for myself. Pick up a framework from the top of the benchmarks and experiment with the learning curve and the language. Well, after carefully evaluating, I decided JAVA wasn't for me (too much of a beast). But Scala seemed to hit the sweet spot with excellent language (1 line of Scala = 10 lines of Java on an average) features along with the trust of the JVM.[1] So, I chose Scala and started experimenting with it. I started with the play framework and was not fond of certain decisions that were made within the framework. Also, the play framework is nothing like Ruby on Rails (but it claims to be so) and was more of self-serve and also quite heavy for my taste. Hence, I went even farther up the list and chose Scalatra. Generally, framework selection is a matter of personal preference and each one should choose whatever aligns with their philosophy. For me, it was scalatra. Frankly, I haven't been able to ship out features at the speed at which I used to, with Rails, but there was something very important that I learnt too - Don't get comfortable with full stack frameworks without understanding what goes underneath. Rails suddenly seemed like an enemy to me because it was doing everything so magically that I literally had NO IDEA until I had to implement them myself with Scalatra. For example, do you know what AES-GCM is? Do you know HOW the secure cookie store in Rails works? What algorithm it uses?? Do you know you have to change the secret pretty often??? Do you know how authentication in Rails (Devise) works??? Can you build it from scratch???? I knew none of these, until I had to implement them myself for my application on top of Scalatra. It was seriously a pain because for the first one week, I did absolutely nothing, in terms of progress with features. But later, I started loving this way of development rather than relying on 'magic' frameworks. Now 'magic' seems scary for me because I cannot actually see what's happening underneath. So, to cut a long story short, Should you choose the fastest framework? Or should you follow the 'ship it first' policy with an average framework?? My advice - Start with a full stack framework, something like Rails or Django, but also in parallel, try to understand how every bit of your framework works and try to implement it into a micro framework based on something from the top of the benchmarks (like Go, Scala, etc.). Most importantly LEARN. Something new everyday! And slowly shift your development towards these high performant frameworks as and when you see fit. This is just based on my experience :) [1] 1 JVM = 10 Thins, Source: http://www.slideshare.net/pcalcado/from-a-monolithic-ruby-on-rails-app-to-the-jvm http://www.slideshare.net/pcalcado/from-a-monolithic-ruby-on...

continuations 13y ago

cpoll-cppsp made huge performance gains from round 5 to 6. For example multiple queries went from 1,872 rps in round 5 to 7,252 in round 6. What caused that? Also what happened to cpoll-cppsp in the plaintext test? Its performance there took a deep dive.

bhauer 13y ago
I know that the contributor of those tests made several changes. We weren't able to identify why it fails dramatically on the plaintext test, but I suspect it's a combination of the two unique characteristics of that test: the use of HTTP pipelining and higher client-side concurrency. My guess is that the pipelining in particular is causing problems.
- MetaCosm 13y ago
  I really hope you guys start doing more in-depth high-concurrency testing. This is a huge risk / problem with many frameworks and kits. It seems to be a blackhole in your tests for some reason, which is a shame because the are so well done in other regards. Having a framework which falls down horribly at high concurrency is something people need to know about -- in how they structure and design applications and how they structure deploys. It can also help them pick frameworks that fit them better (some frameworks will do great at high concurrency, some won't). Can't wait till you start adding high concurrency (which should NOT require any code changes from the implementation providers, you add a zeroes to the concurrency.. 100, 1000, 10000 tests).
joeblau 13y ago
Is there any reason why it is so slow on plain text?
- bhauer 13y ago
  Looks like there was a problem in cpoll-cppsp: https://github.com/TechEmpower/FrameworkBenchmarks/pull/364 https://github.com/TechEmpower/FrameworkBenchmarks/pull/364 This will be merged in for Round 7.

antihero 13y ago

How are certain languages so much faster even when pulling from the database? Surely the database would be the bottleneck here? Surely if it's something cool the driver is doing, that could be implemented in other drivers? Also, why are some frameworks on there a bunch of times (e.g. ASP MVC)?

continuations 13y ago
> Surely the database would be the bottleneck here? The idea that "it doesn't matter which language you use because DB would be your bottleneck" might have been true 10 years ago but it's definitely not true anymore. Latest versions of MySQL and Postgresql are extremely fast. They are written in C/C++ and heavily optimized. MySQL serving data from RAM can easily do 500K+ rps on a commodity server. Even if you need to go to disks there are $200 SSD's that give you almost 100K IOPS. Put a few of them in a RAID and you're getting hundreds of thousands of IOPS for very little money. In all likelihood application code written in Ruby/Python/PHP with frameworks piled on top of abstractions is going to be orders of magnitude slower than the DB server written in C that has undergone 15+ years of heavy optimizations.
- antihero 13y ago
  Ok, but say we use vanilla Python WSGI with a psycopg2 (C Python PostgreSQL extension) directly. Will this come close to the performance of the C++ framework thing?
  - continuations 13y ago
    I'd say a vanilla Python application without ORM would perform similar to a flask app without ORM. According to http://www.techempower.com/benchmarks/#section=data-r6&hw=i7&test=db http://www.techempower.com/benchmarks/#section=data-r6&hw=i7... flask (on Gunicorn): 6,138 flask (on PyPy): 8,167 Django + ORM : 4,026 cpoll-cppsp : 114,711 So while a light weight Python app without ORM might be 50% faster than a full-blown Django app, it's still almost 20 times slower than a C++ app. That shouldn't be surprising, by all accounts Python is at least an order of magnitude slower than C++. You could speed up your Python program by running PyPy. According to the benchmark that gives it a 30% speed boost, which is in line with my experience. Another way is to deploy on uWSGI or Meinheld instead of Gunicorn. No matter what you do, Python will not be close to the speed of C++. But what really matters is: is it fast enough for your need?

coolsunglasses 13y ago

I've followed yogthos' work on Luminus since the beginning. It doesn't merit mention independently of Compojure. It's just some sane defaults for pedagogical reasons for getting started writing web apps on top of Compojure. It adds nor removes no runtime or anything else. As it currently exists, it's a Leiningen project template generator. That's about it. Luminus is isomorphic, for the purposes of a benchmark, with Compojure. I'm sure yogthos will be flattered when I tell him his web app templating thingy was included though.

bhauer 13y ago
Sunglasses! Good to see a familiar name. Incidentally, yogthos should be aware that Luminus was included: https://github.com/TechEmpower/FrameworkBenchmarks/pull/293 https://github.com/TechEmpower/FrameworkBenchmarks/pull/293

cmccabe 13y ago

What about Cobol on COGS?

ditados 13y ago

i don't see any mention of uwsgi or gevent in these benchmarks. Deploying Python apps without properly configuring the container/webserver will skew the results considerably (probably applicable to a couple of other entries as well).

evab 13y ago

Can anyone explain why the "gemini" framework with an ORM outperforms raw servlets/queries in almost all the cases? What kind of optimizations are done ?

bhauer 13y ago
Evab, Gemini is our in-house framework [1] that we've included in these tests for our own learning experience. We're not obsessed with performance, but we do try to avoid problems that are easily avoided. :) For instance, where possible, we have used Java's concurrent data structures and reduced the use of locks. A difference in these tests is that we have our own connection pool. Speed was not a main objective when we wrote it--it's so old that if my memory is correct, it predates the availability of the connection pool that now ships with the MySQL driver, which is what we've used in the Servlet tests. Nevertheless, my conjecture is that perhaps it's just a tad quicker at selecting an idle connection. Incidentally, for a while in an earlier round, the Go guys had an alternate test implementation that used a concurrent queue for connections and it performed fantastically. We've been toying with changing Gemini to do the same at some point. For the benchmarks project, we classified ORMs as "raw" (no ORM), micro, or full. In Gemini, we have a micro ORM. The following are examples of its use in the test code [2]: // Get all rows from Fortunes table. final List<Fortune> fortunes = store.list(Fortune.class); // Get a random World row. worlds[i] = store.get(World.class, random.nextInt(DB_ROWS) + 1); // Run a batch of updates from a List of entity objects. store.putAll(Arrays.asList(worlds)); "Micro ORM" is loosely defined, but we've applied it to ORM options that offer some abstraction over plain old SQL, but aren't as comprehensive as the standard-bearers. For example, while we have a data structure that represents relationships, we do not have a higher-level query language that traverses relationships automatically nor any similar mechanism for easy relationship traversal. It turns out that across all of these frameworks, there are dozens of what we are calling Micro ORMs. Getting back to your question, the Gemini ORM does leverage a few tricks for performance such as ReflectASM for object creation and prepared statements for queries. But there isn't a whole lot of fancy work going on. It's not glamorous, but the most important high-performance elements we make use of in Gemini are the JVM platform and Java's concurrent data structures. As demonstrated by the likes of Undertow (the web server component of JBoss WildFly), mind-boggling performance is available on the JVM. [1] https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg-Ibk/fOjQMxESEJ0J https://groups.google.com/d/msg/framework-benchmarks/p3PbUTg... [2] https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/gemini/Source/hello/home/handler/HelloHandler.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

mark_l_watson 13y ago

Disturbing results since I so frequently have used Sinatra for hosting REST APIs for customer projects. Rethink needed. For my own projects I use Clojure + Compojure and more recently added Node.js (often with Meteor) and was pleased to see that they benchmark well. One thing I especially like about Compojure with the Hiccup HTML generating library is that I see syntax errors immediately in the editor. I suppose that Scalatra with embedded HTML would provide the same benefit.

bemurphy 13y ago
Maybe benchmark it yourself? I think something is amiss, it should easily 'hello, world' at 2x higher rate than Rails. Poked around the setup a little a while back but nothing jumped out. Hunches: it's either going through webrick or not running with RACK_ENV production
- bhauer 13y ago
  Unicorn, nginx, production environment. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/sinatra/setup_ruby.py#L18 https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...