15 ms·

Web Framework Benchmarks Round 2

193 points by pfalls 13y ago

bhauer 13y ago
This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contributed! We hope this is useful information.
- krg 13y ago
  The contributions from the community have been great. To those who submitted pull requests that didn't make it into this blog post: We've been overwhelmed (in a good way) with the response and we're working to include as much as we can. Thank you!
- neya 13y ago
  Dear bhauer, You have no idea how valuable this is to everyone! I know it takes a lot of effort to consolidate all comments, requests, fixes suggestions, etc. Personally, I've even seen you respond on the Play! framework Google groups. Thank you for being such a down to earth person and helping out the community. You guys rock :) Thanks!
  - pfalls 13y ago
    Thanks so much for the kind words. We're obviously having a great time working on this, and we too think there's a lot of value to this for the community at large.
- e12e 13y ago
  It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second? Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC licenses, which I didn't find): http://zedshaw.com/essays/programmer_stats.html http://zedshaw.com/essays/programmer_stats.html For instance, you state that: > In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds. Only cake php requires more than 100 milliseconds on average. But look at Django: Average around 60 ms, standard deviation around 90 ms (!). Not to mention a "worst" score of 1.4 seconds.
  - pfalls 13y ago
    Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.
    - vanderZwan 13y ago
      As one of the people complaining about statistics last week (and also, by coincidence, citing Zed's rant), I'm glad to see you are working on it and open to more ideas and improvements! Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this. Would be nice if the community helps turn this into the de facto example of how to benchmark correctly. Now I'll just have to wait and see how Go 1.1 compares ;).
      - bhauer 13y ago
        Go 1.1 is looking very strong! Pat (pfalls) just showed me some very preliminary numbers and we are extremely happy with them. Thanks for the input and constructive criticism, vanderZwan. It has been very helpful to get feedback from yourself and everyone else. I too am particularly happy with the sportsmanlike competition vibe. You have no idea how fulfilling that is to us.
  - krg 13y ago
    Note that the results data is available as JSON if you want to play with different sorting yourself: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/results/ec2/20130404200504/results.json https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... As pfalls said, we're definitely open to different ways to present the data. More sorting would be cool.
- giulianob 13y ago
  Did you guys turn on byte code caching for all the PHP frameworks? If not, then I recommend everyone ignore these benchmarks until that is completely done.
  - krg 13y ago
    We did. PHP 5.4.13 with APC, using PHP-FPM, running behind nginx.
  - kainsavage 13y ago
    Yes, we are using APC for all the PHP tests in this round of benchmarking.
    - keammo1 13y ago
      Is there somewhere we can see the settings for APC? Thanks for the great work!
      - krg 13y ago
        https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/config https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... See php.ini, php-fpm.conf.
        keammo1 13y ago
        I could be missing something, but it looks like you are using the default settings. Have you tried tweaking it all? Specifically setting apc.stat=0, which will stop it from checking the mtime. You'll need to clear the cache with apc_clear_cache() when you make code changes though. You may also want to look at apc.php to check for fragmentation and adjust apc.shm_size if necessary
        ebiester 13y ago
        fork, benchmark, and create a pull request. :)
- DonnyV 13y ago
  Why no .NET C# and Mono?
  - errnoh 13y ago
    Their repo at https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks is open for pull requests I don't think they have anything against C#, probably no one just has committed C# version to be benched. (edit: whops, linked fork first)
    - tracker1 13y ago
      I'd be curious how it would perform on C# (windows/.net vs. linux/mono) .. though the environment setup would probably be a bit more involved.. and IIS is a very different beast. I'd think that it would probably land somewhere close to Java Servlets, but a bit slower. The framework stack for web requests in .Net is probably a bit more than it is in the servlet server in question. I would also think that Mono would be a bit slower than IIS, only because IIS does very well at pooling resources/threads for multiple requests. There's also the question of async .Net handling vs. blocking. Most .Net code I've seen is blocking, but there are async options, and as of the 4.x releases are much easier to use.
    - bhauer 13y ago
      Thanks for the link, errnoh. You're precisely right, we want to have C# and .Net tests included but didn't find the time to do so ourselves in the past week and have not yet received a pull request. I was not familiar with ServiceStack prior to feedback we received to last week's test, but from the looks of it, I'd personally like to see how it does.
  - fredkingham 13y ago
    In their previous test they said that it wasn't fair to test with C# on Mono as its not as performant as C# on windows.
- eranation 13y ago
  First time I'm actually happy for building something with Servlets (have a feeling I'm the only one here that uses it as the go to framework for hacking something quickly, I'm an old man), it's indeed dead annoying for REST services, but doable, and blazing fast. The question is - when does my productivity starts to be more important than performance (I think 99% of the lifetime of anything I'll ever build is the first and not the latter). I'm very surprised to see Play-Scala is not in the same playing field as other JVM based frameworks, had a lot of hope for it, I hope TypeSafe will take that into consideration... Node.js is the biggest surprise for me on the good side, I think it changes my plans a little as for what to learn next... Thanks for a very valuable study, this is great
- deleted 13y ago
  [deleted]
coolsunglasses 13y ago
Go 1.1 beta is up, really need to get some results with tip instead of 1.0.3.
- pfalls 13y ago
  Go 1.1 is very high on our list, we'd love to get that into the tests as soon as we can. Of course, anyone in the Go community is welcome to issue a pull request that updated the version to 1.1. Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64 https://github.com/TechEmpower/FrameworkBenchmarks/pull/64
  - TylerE 13y ago
    Don't think you'd need to change any code, just pull mercurial tip and recompile go and the app.
    - voidlogic 13y ago
      cd go/src hg pull hg update tip ./all.bash
      - coolsunglasses 13y ago
        That's not how their installer works. I submitted a PR to pull the 1.1 beta tarball.
    - cmsimike 13y ago
      Is 1.1 production ready? I'd feel weird about incorporating something still in a "beta" phase though.
      - coolsunglasses 13y ago
        People have been working off of tip for months.
      - 13y ago
voidlogic 13y ago
It is worth pointing out Go 1.0.3 not Go 1.1 beta is being used. I tested the code locally and on my machine 1.1 beta was about 20% "faster" (req/sec).
- cmsimike 13y ago
  Really? We're definitely going to try that out!
  - voidlogic 13y ago
    In addition to the improvements from the improved scheduler, use of accept4 on Linux, better GC and code generation, Brad Fitzpatrick has been giving a lot of love to the "net/http" package. Here is a small example: https://plus.google.com/u/0/115863474911002159675/posts/L3o9hEs8SAe https://plus.google.com/u/0/115863474911002159675/posts/L3o9...
  - errnoh 13y ago
    Yeah, 20% is even quite low estimate. Here's 1.0.3 vs 1.1 from just now (request worker (wrk) and Go http server running in separate machines) https://gist.github.com/errnoh/5320784 https://gist.github.com/errnoh/5320784
redtuesday 13y ago
Thanks for the tests. Especially the latency is interesting. I cant't wait for the next round, when the updates of the last two days are included (elli, grizzly, play-java, play-scala, snap etc).
- pfalls 13y ago
  Since we moved to a different benchmarking tool for round 2, we had to re-run all the tests, which is a time consuming process, especially on the EC2 hardware. Now that we've gotten that out of the way, we have an easier time running and processing results for individual tests. So stay tuned!
Uchikoma 13y ago
Very interesting - again. What I - again - find most interesting, is the large discrepancy in the 20-query test. Common wisdom seems to be that language performance is not important b/c every language hangs in the same IO/DB.
- malachismith 13y ago
  probably a combo of mysql drivers and connection pooling.
  - Uchikoma 13y ago
    I wonder why Play is so bad then.
nlh 13y ago
This is an awesome project, and thank you for the follow-up. In particular, this is awesome because it's introduced me to some new frameworks that I hadn't even heard of (ie vert.x) that seem extremely interesting. Keep it up and many thanks for this contribution!
- pfalls 13y ago
  Thanks for the kind words. Glad to hear that you've been introduced to some new frameworks, we feel the same way, and we're excited to see what else the community has up it's sleeve.
  - diminish 13y ago
    Pls could u include some C/C++ web frameworks... comparison with Java, could be interesting..
    - mitchi 13y ago
      yes please! There's got to be a C++ framework out there to test. Casablanca?
      - reactor 13y ago
        here we go! http://www.treefrogframework.org/ http://www.treefrogframework.org/
    - krg 13y ago
      We're interested in that, too. In progress is the onion http library pull request from davidmoreno: https://github.com/TechEmpower/FrameworkBenchmarks/pull/43 https://github.com/TechEmpower/FrameworkBenchmarks/pull/43 We hope to get more as well.
      - spyder 13y ago
        It would be interesting to see G-WAN too because it seems controversial. http://gwan.ch/ http://gwan.ch/
kevinburke 13y ago
It seems like you're still using Mongo DB as the datastore for some languages and Mysql as the datastore for other languages. This seems like it would bias the results.
- pfalls 13y ago
  The only framework that uses MongoDB exclusively is the vert.x test, the nodejs tests have tests for both MySQL and MongoDB
mythz 13y ago
Are C#/Mono web frameworks an option? Because I would love to see how http://servicestack.net http://servicestack.net fares: https://github.com/ServiceStack/ServiceStack/wiki/Real-world-performance https://github.com/ServiceStack/ServiceStack/wiki/Real-world...
- giulianob 13y ago
  Ditto. Would be great to also compare them to .NET VM
  - misterbwong 13y ago
    +1 again. I'd like to see how .NET fares as well but I understand that this is a Linux x64 test...
- pfalls 13y ago
  We've had a lot of requests Mono, and are actually very interested in trying this out. We've been hesitant to move forward only because we're unsure how stable the C# community believes Mono to be. We hope to have that test at some point though, but the fastest way to get a test included is to issue a pull request. So if anyone is interested in trying this out, let us know and we can help guide the process.
  - mythz 13y ago
    Cool, a note on the stability side: http://www.servicestack.net http://www.servicestack.net and the live demos http://www.servicestack.net/#livedemos http://www.servicestack.net/#livedemos have always been hosted on Linux/Mono in all the years it's been in existence. Is GitHub issues the best way to ask for guidance?
    - pfalls 13y ago
      Yes, Github issues is a good place to ask for guidance.
- voidlogic 13y ago
programminggeek 13y ago
One of the variations that is being tested here, that I'm not sure if it is legitimate is that for PHP specifically, Apache is likely spinning out many PHP threads, I'm not sure if the same behavior is happening on the ruby side. Thus, if you are comparing performance of 10 PHP threads to 1 ruby or python thread, your benchmark is not going to be terribly accurate.
- krg 13y ago
  PHP was tested behind nginx this time due to community feedback we received, not Apache.
  - programminggeek 13y ago
    Awesome, I did realize that. I stand corrected. Love the work you're doing!
jjm 13y ago
Hm... we'll need to get spray.io on this list
- bhauer 13y ago
  Yes, please! Spray.io folks, can we entice a pull request? :)
voidlogic 13y ago
It might be interesting to track average and peak server side memory usage during these tests. For example, if framework A is only 80% the "speed" of framework B, but uses 1/4 as much memory, for some users this might be a win for A.
- pfalls 13y ago
  Great suggestions, we've thought about capturing other system data as well, but memory seems like a great choice to start with.
- cmsimike 13y ago
  That would be an interesting metric indeed.
- kainsavage 13y ago
  This is definitely something we want to include in the tests, but it is difficult to manage a fair way of monitoring. Outside of the actual benchmarks, where we kept everything as fair as possible, we ran individual tests to make sure they worked prior to the benchmarking and would routinely have htop running at the same time to get an idea of what that sort of data looks like. It IS very interesting and is definitely something we have discussed methods for measuring for the sake of these benchmarks. Other areas that we discussed were cpu utilization, disk utilization, and network saturation.
  - kalmar 13y ago
    One idea that jumps to mind is putting the running frameworks in an LXC container (or similar) and monitoring the memory usage of that. Not sure how accurate that is, but it's one avenue. It also still might not be fair because some runtimes can let their heap get pretty big despite not actually needing the memory. You'd want a way to differentiate those from the ones with big heaps that would choke with some memory pressure.
    - voidlogic 13y ago
      Just make sure you measure resident memory not virtual memory :)
mdlthree 13y ago
As an amateur hacker and after 275 days of reading Hacker News I feel I can now navigate in the sea of client side Javascript frameworks. With the introduction of this data on web framework performance, I now have another set of choices that I am completely unqualified to comprehend. My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and many other introductory app server frameworks. Lets say after I was able to build an initial product successfully, would I then consider re-building the product in a higher performance framework as Go, Node, etc? The third level, netty and gemini and servlet etc, I am not familiar with. Googling "netty" I get --"Netty is an asynchronous event-driven network application framework"-- I thought that is what Node is (in js) and Go does with gophers. What are the use cases for these faster frameworks and do they follow an evolution of performance options that an app might go through?
- ConstantineXVI 13y ago
  Most of these are bound to a language: choose a language first (and my answer to that question is "whichever one you're happiest working in and can get the job done"), then figure out which backend to use. You can fret over the framework more after you've already built something that works.
- avenger123 13y ago
  All metrics are meaningless without context. I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone. I find its good "food for thought" as it provides me exposure to frameworks that perhaps I can use one day but also to learn more about. The most interesting aspect is all the code for each framework to do the simple page output, the queries, etc. can be viewed for each framework. I find this an incredible learning tool as it shows the personality of each framework/language.
  - pfalls 13y ago
    "I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone." This is exactly right, performance is one of many items to consider when building your own website, and while we're aiming to have a very comprehensive and useful set of performance data, we would agree that other factors need to be taken into account.
BryantD 13y ago
I apologize if this question has been asked and answered, but why is netty missing from the database tests? No ORM?
- pfalls 13y ago
  The short answer is that at some point, we kept adding more and more tests on our own, which kept the blog from going out, and we finally decided to have a cutoff date and let the community continue adding tests that they were interested in. I think adding a database test to Netty (as well as Go for that matter) would be a great addition.
  - BryantD 13y ago
    Thanks! And thank you very much for this work -- it's tremendously useful.
kbd 13y ago
What about Puma (http://puma.io http://puma.io), Goliath (http://goliath.io http://goliath.io) or Unicorn (http://unicorn.bogomips.org/ http://unicorn.bogomips.org/) instead of Passenger in the Ruby stack?
- pfalls 13y ago
  We're looking into moving to unicorn as the community has suggested. If anyone is interested in setting up Puma or Goliath, we'd be interested in testing those out as well.
  - cheald 13y ago
    For something like the query tests where you're IO-bound, Puma is going to annihilate the competition. For CPU-bound tests like JSON generation, unicorn with multiple workers is going to perform better. It's worth noting that Rails' default JSON solution is the compatible-everywhere pure-Ruby JSON generator. Using a drop-in replacement like the oj gem will drastically improve throughput there. I didn't get to the pull request for this round, but tweaking the GC parameters for the Ruby install should dramatically improve the general performance of the non-JRuby Ruby tests, as well. I'll see if I can get a PR in. :)
    - ksec 13y ago
      Well yes please do this, not sure if it is allowed though since Oj gem is written in C ( I think )
      - cheald 13y ago
        So is the mysql2 driver (at least the slow parts), and Passenger is C++. C extensions are a common part of your standard Ruby deployment stack.
boundlessdreamz 13y ago
This is superb. 1. More sorting options please :) I really want to sort by standard deviation. 2. Also in case of rails and other single threaded frameworks are you spinning up multiple processes? 3. Why is Go not being tested in the Database access tests? 4. Can you test rails with unicorn too?
- bhauer 13y ago
  1. Blame me. I put together the charts and tables. I'll try to get some sorting added for the next round! :) 2. Pat (pfalls) should know the answer to that more definitively, but we are definitely trying to use the CPU cores as fully as possible. I think you'll see some comments by Pat elsewhere in this thread suggesting some things coming in round 3. 3. Good question! Mostly because we haven't had the time to add it. I would really like to see it added, though. If you know Go and can write the code, might I entice you to submit a pull request? 4. I believe that is the plan, yes.
mike-- 13y ago
migth be interesting to add Erlang.
- kainsavage 13y ago
  We have at least one Erlang test that came in yesterday and will be included in the next round of benchmarks.
  - mike-- 13y ago
    thanks.
    - kainsavage 13y ago
      Don't thank us, we didn't write the code for it - one of you (the community) wrote it and submitted a pull request. We just ensured that the test would work in our benchmarks, then ran it! We are happy to do it, and super appreciative of the support from you guys!
charlesjshort 13y ago
Any thoughts of putting some Erlang in there?
- kainsavage 13y ago
  We got a pull request for an Erlang framework (my brain is not letting me pull up the name) yesterday that we accepted, but sadly it was too late for this round of benchmarks. We will include it in the next round.
milesf 13y ago
It's a useful data point to know the speed of frameworks, but what matters most to me is speed of development and whether or not I enjoy the process. Wish there was a way to benchmark those points.
- cmsimike 13y ago
  I 100% agree with this. It has been a huge point of discussion for us here internally. "Developer efficiently with language x." Ultimately, language performance does matter but there are ways to make up for less performant languages. If it takes you 2x as long to get something out the door though, then that is a huge problem. Just use the language and framework you are most efficient in and enjoy using. Worry about framework performance when you notice server load increasing over time.
  - malachismith 13y ago
    Sure... as long as you have the migration and rewrite plan done and ready to go, and have engineered the product for that rewrite. Otherwise you're gonna be wishing you'd failed rather than succeeded.
- columbo 13y ago
  >what matters most to me is speed of development and whether or not I enjoy the process. Matters most? There are 100x differences here. Take two companies writing the same application, one in Spring the other in Rails. Both have easy access to knowledgable people, both are industry standard. However the Spring application would be several orders of magnitude more scalable[1]. I'd consider that to be a more important consideration over enjoying the language. [1] All benchmarks are suspect until proven otherwise. You can write slow software in Java/C/C++, you can write fast software in Python/Ruby etc etc etc
  - superuser2 13y ago
    Performant, not scalable. You can throw money at Rails and it will handle the load, but it would cost a lot more.
portmanteaufu 13y ago
This is great stuff. I'd love to see some more frameworks written in compiled native languages like Haskell and C!
saosebastiao 13y ago
Yesod disappoints! Unfortunate for how much it is hyped. This is definitely good to know, even if it appears that the Warp server was not used.
- flyinRyan 13y ago
  If Warp wasn't used then this isn't a proper test of it, no?
- pilgrim689 13y ago
  That's the first thing I noticed. What was used instead of Warp? I also see that Snap got pulled in a couple days ago, so it will probably get tested in the next round. Should be interesting to see how it fares against Yesod.
  - mightybyte 13y ago
    The way the benchmark is constructed, it's really more about JSON and DB performance. Snap makes no decisions for you about those things, so the numbers really won't reflect the performance of Snap all that much. I don't have great expectations for the "Snap" code that is in there now because it is not using very fast JSON and DB libraries.
- raphaelj 13y ago
  I wrote the Yesod tests for this benchmark. It uses Warp. Actually, the performances of Yesod are not so bad, as Play's are. Top performing frameworks (netty, vertx, go) are not web frameworks but async programming frameworks. They don't carry a lot of things around -- just push the serialized data on the raw socket output stream. Yesod/Play/Django/Rails/... on the other hand must manage routes, content types, browser headers, etc etc....
apendleton 13y ago
I think I said something similar on the first round, but for the Python tests, I'd be curious to see alternate runtimes and/or concurrency strategies: pypy, gevent, and probably the confluence of the two (though it requires quite a bit more hacking, since gevent doesn't officially support pypy).
- pfalls 13y ago
  We're very curious about PyPy as well, and have it on our list https://github.com/TechEmpower/FrameworkBenchmarks/issues/15 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15
- nobodysfool 13y ago
  Yea, they have Flask in there, but it's on a braindead setup. gunicorn is using the 'sync' worker type, not gevent or eventlet (which is the whole point of gunicorn) and it's not preloading the app, so it's compiling the code every time the process is forked. No wonder Flask performs so poorly in this test. Yes, I've submitted a pull request with these changes.
cheriot 13y ago
As one would expect, as more data access is added, the scale of differences between web frameworks goes down. It would be interesting if you added 5 and 10 db query test runs as that starts to get closer to what a functional website would be doing.
- bhauer 13y ago
  Hi Cheriot, We do in fact have 5 and 10-query runs available in the data. Switch the views to "Data table" or "All samples (line chart)" and you should see the data for 1, 5, 10, 15, and 20 queries per request.
- deleted 13y ago
  [deleted]
eibrahim 13y ago
I would love to see how asp.net mvc compares to all this...
jimmytucson 13y ago
Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks? If not, can anyone explain in layman's terms why that should be the case?
- jcroll 13y ago
  Why shouldn't that be the case? PHP has been designed from the get go to be super fast and 5.4 introduced a bunch of optimizations that have sped it up even more so. Ruby was designed for programmer happiness, PHP was designed for performance and getting sh*t done.
  - pdeuchler 13y ago
    PHP technically wasn't designed
    - deleted 13y ago
      [deleted]
  - irahul 13y ago
    > PHP has been designed from the get go to be super fast PHP was never designed to be fast. As far as execution speed goes, PHP will be in the same ballpark as Perl?Python/Ruby. PHP comes up as faster in these tests because of all the extra things the frameworks are doing. See the_mitsuhiko comment above regarding flask for an example.
- rozap 13y ago
  Not sure about ruby, but django doesn't ship with connection pooling, so that is probably why its database performance is terrible. It could be configured to use pooling. With such a trivial request handler like the one in benchmark, only a small percentage of the execution is actually taken up in the view, so all of the time you're seeing in the benchmark is pure overhead of the framework. A longer running view across all of these frameworks would cause things to even out a bit. All that being said, however, it's not an unfair comparison. After all, a benchmark of this nature should be a measure of overhead because that's really all there is to measure.
alberth 13y ago
I wish Lua was tested in the benchmarks. In particular, Lua-CJSON [1] [1] http://www.kyne.com.au/~mark/software/lua-json-performance.html http://www.kyne.com.au/~mark/software/lua-json-performance.h... (I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3") EDIT: fix typo
- kainsavage 13y ago
  Personally, I really like Lua and I would love to see a pull request with a Lua framework test!
- pfalls 13y ago
  There will most certainly be a Round 3, and 4 and on and on as long as we continue to have feedback from the community. I can guarantee that we'll have a Lua test in there, but if we receive a pull request, we'll include it.
- snaky 13y ago
  ngx_lua [1] will beat all of them no matter which JSON library used, end of story. BTW, LuaRocks is supported [2] [1] https://github.com/chaoslawful/lua-nginx-module https://github.com/chaoslawful/lua-nginx-module [2] http://openresty.org/#UsingLuaRocks http://openresty.org/#UsingLuaRocks
bsnyder788 13y ago
I think that the play-java one would likely benefit greatly from using raw jdbc rather than the ebean ORM. In fact my best guess is this is why the scala/java versions are so far apart. In past projects, scrapping ebean always led to large performance improvements in throughput, etc.
- redtuesday 13y ago
  https://news.ycombinator.com/item?id=5499839 https://news.ycombinator.com/item?id=5499839 ps: i agree that raw jdbc would enhance the performance.
don_draper 13y ago
I code in Ruby, Python and Groovy/Grails. I'm amazed that Grails performs so much better than the others.
- pfg 13y ago
  I guess that's one of the big advantages JVM languages have. Using stuff like Spring or Hibernate (like Grails does), you benefit from thousands of person-hours spent on tuning those libraries and you still get native Java performance. Obviously there's a price you have to pay for all the things Grails built on top of Spring/Hibernate (in the benchmark it looks like Grails' performance = Spring / 2), but in general, it's still faster than just writing everything in Groovy (Ruby, Python, ...).
  - msgilligan 13y ago
    It would be interesting to see the results for Grails 2.2 (which uses Groovy 2 and Invoke Dynamic) One of the nice things about Grails is that you can easily drop down to the Spring and/or JVM level when needed for performance.
    - don_draper 13y ago
      Yup. I manage a Grails app where I wanted to speed up performance on a popular part of the app. So I replace the Grails controller with a servlet. I saw a bump in performance.
- vorg 13y ago
  Grails performs better than what? Ruby and Python are programming languages while Grails is a web framework.
thatthatis 13y ago
This is still a json API comparison, not a full framework comparison. In the first test it sees how fast frameworks can serialize json and in the second test it sees how fast frameworks can access the database AND serialize json. An inefficient json library and a framework is doomed across all tests. There are a lot of things frameworks do beyond serving json. These benchmarks are presented as representative of broad use when really they are only representative of the frameworks if used as a json api. A raw HTML hello world, and removing the json encoding step in the db test would go a long way towards getting results that can mean what the authors seem to want them to mean. There's nothing wrong with a json API benchmark, in fact its quite valuable in a lot of cases. But if that's what this is intended to be, the authors should say so.
- logic 13y ago
  Agreed; for example, in our (in-house) use case, switching a django app from using either stdlib's json or simplejson to using ujson[0] was a significant performance increase when serializing/deserializing large-ish (~100MB) JSON datasets. [0]: https://pypi.python.org/pypi/ujson https://pypi.python.org/pypi/ujson
Zak 13y ago
A note about Compojure: they're using Korma for database access. Korma is an abstraction layer that's more Clojure-friendly than using SQL directly, but lighter than an ORM. Using clojure.java.jdbc might produce higher performance.
- cygwin98 13y ago
  The performance of Compojure was really impressive, which was almost three times as fast as play-scala. I was expecting the opposite, considering that Scala is generally a bit faster than Clojure.
  - Zak 13y ago
    From what little I know about Play, it does a lot more than Compojure does. Compojure is really just a routing library. This is a benchmark of Ring (HTTP communication), Compojure (routing), Korma (database access/abstraction) and Cheshire (JSON serialization). I'm pretty sure a basic request in Play goes through more middleware than that. But yes, the performance of a reasonable web app stack on Clojure is very good.
- krg 13y ago
  Where possible on the tests, we tried to use something other than raw SQL because we wouldn't expect someone to use raw SQL everywhere when writing a real application. We chose Korma because it seemed decently popular for Clojure, but we're certainly open to a "compojure-raw" test that uses clojure/java.jdbc. Feel free to submit a pull request. :)
  - Zak 13y ago
    I'm not saying Korma isn't a good or realistic choice. I use it myself. There did seem to be some disagreement in freenode #clojure about whether that was the best way to go, or JDBC and prepared statements. Evidently, a lot of people actually do like SQL. It's just important for people to be aware that "Compojure" actually represents "one possible production-ready stack that uses Compojure as the routing component", as do several other items on the list. I don't mean that as a criticism - just clarification.
njharman 13y ago
I find the lack of Erlang frameworks disappointing.
- kainsavage 13y ago
  We accepted a pull request for an Erlang framework yesterday, so they will be in the next round of benchmarks. If you want to ensure your favorite is listed, you can issue a pull request yourself on our github page.
- taeric 13y ago
  Instead, find it motivational. :) They are accepting pull requests.
  - krg 13y ago
    In fact we've already got a pull request that includes Cowboy and Elli: https://github.com/TechEmpower/FrameworkBenchmarks/pull/50 https://github.com/TechEmpower/FrameworkBenchmarks/pull/50 We'll be including this in the next round.
    - chops 13y ago
      If I can find the time, I'll try to put together one for Nitrogen (http://nitrogenproject.com http://nitrogenproject.com) as well.
tantaman 13y ago
Does anyone know what on earth is Play! doing that makes it is so slow? I'd like to see Lift in there if you guys do another round.
- bsnyder788 13y ago
  My best guess is that the java one is slow due to ebean (I've always seen huge performance improvements from switching to raw jdbc). The scala one I have no clue.
  - redtuesday 13y ago
    More likely because the version uses only as many threads as the cpu has cores. Whereas the servlet version for example has 128 thanks to the default resin configuration.
    - bsnyder788 13y ago
      That would definitely be an issue as well. Although, this is easily configurable within the application.conf file if I remember correctly.
- maybenot 13y ago
  The Play JSON controller is using Futures to do JSON serialization. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Compare this to the Java Servlet, which just serializes directly in the Servlet callback. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/servlet/src/main/java/hello/JsonServlet.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... The former seems like overkill.
  - redtuesday 13y ago
    This is not the version which was used in the tests. The tests used a earlier version.
- 13y ago
jeltz 13y ago
I wonder where the bottleneck for the ruby benchmarks is. Is it in passenger, the framework, or somwhere else? I am sure I have gotten higher requests per second on my workstation when I tuned Ramaze (which should not be faster than using raw rack). My guess is that it is either due to a different method of measuring or the fact that I was using thin rather than passenger.
andybak 13y ago
Wow. I really need to use connection pooling with Django.
polskibus 13y ago
I would love to see ASP .NET and ASP .NET MVC included in the benchmark.
escaped_hn 13y ago
Is gemini really that awesome? Why arent more people talking about it?
- saym 13y ago
  Gemini is closed source. The company running these comparisons is the only team that is qualified to test/develop with it.
  - kainsavage 13y ago
    While it is true that our source code is closed at the moment, you are still free to check out the benchmark code from github and run it yourself to verify our results. It wouldn't really be a fair test if we put all these frameworks up for testing, suggested Gemini was better than sliced bread with auto-applying jam, and didn't back it up in a testable way ;-)
wheaties 13y ago
Is it against the rules to use different workers for Django or Flask? In other words, there's a big difference when I run things with Gunicorn+Gevent+psychopg (PostGres async driver) than when I run it just with Gunicorn alone.
- pfalls 13y ago
  Each test can be modified in isolation, if both tests see an improvement in that setup, we would like to include the changes for both though.
kriro 13y ago
Cool stuff. Only looked over it quickly but it seems JAVA stuff is dominating. Kind of curious why jruby doesn't provide more of a boost (especially given compojure doing relatively well, too) and would like to see Django+Jython in version 3 (and +1 to the .NET/Mono stuff)
seunosewa 13y ago
I wish you would have tested minimalist web frameworks in Python: CherryPy and Bottle. Flask is not really a micro-framework in the same way that CherryPy and Bottle are micro-frameworks because of its numerous dependencies.
jebblue 13y ago
Servlets are super fast, this shows that.
VeejayRampay 13y ago
Gemini conveniently leads the pack all the time. So either OP and his/her colleagues have revolutionized web programming or the benchmark is nothing but a PR piece.
- krg 13y ago
  Just ignore Gemini. In fact, you can click the "hide" link on the charts to remove it. From our first benchmarks post: http://www.techempower.com/blog/2013/03/28/framework-benchmarks/ http://www.techempower.com/blog/2013/03/28/framework-benchma... "Why include this Gemini framework I've never heard of?" We have included our in-house Java web framework, Gemini, in our tests. We've done so because it's of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we're proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.
fsiefken 13y ago
Again no mono with ServiceStack! It boggles my mind...
gourneau 13y ago
As a Django guy with hurt feeling right now, I just wanted to say that this is a great project. I look forward to the next versions.
tracker1 13y ago
Where's the source for the implementations? Of interest to me is the NodeJS instances.. was this single-process, or multi-process via cluster (which is recommended for higher performance). The only source I saw was for the testing framework. --- edit: found it. https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks Seems to be using cluster. I'm not surprised that Go and servlets were faster... just surprised to see NodeJS in the middle of the pack position. Though performance isn't the only reason I really like NodeJS
mojomama 13y ago
No love for Mojolicious? It would be great to see a little Perl here!
- voidlogic 13y ago
  Submit your pull request to add it?
just2n 13y ago
Once again, the Node examples are wrong. - Unnecessary parsing. - Writing strings instead of buffers (they're copied, they aren't sent as-is). - Using async. It does a lot of really nasty things, most of which break V8 optimization best practices. This is a perf benchmark, not a comparison of how concise your code can be. - Having the main request handler in a gigantic function that will never be properly optimized by V8. - Not getting helper functions that are clearly monomorphic warm before accepting requests. While the code itself is what I would consider fine, when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code.
- kaoD 13y ago
  On the one hand, you're completely right. On the other hand, shouldn't real code be tested? How accurate would a ugly-optimized-code be when you SHOULD write reusable code in your real life projects?
  - just2n 13y ago
    I wouldn't use async myself, so what's "real" code varies from person to person. This is why benchmarks like this are highly misleading. But ideally, yes. We'd be better informed to see how real world code performs rather than some idyllic perf-oriented creature.
- vanderZwan 13y ago
  So do a pull request and create a node.js equivalent of the stripped editions of rails and django. I'm sure they're open to it.
- ldh 13y ago
  > when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code That seems to defeat the purpose of benchmarking to begin with, or at least makes it less useful for real-world comparison.
  - 13y ago
saadazzz 13y ago
Any plans for ASP.NET?
superuser2 13y ago
This is definitely motivation to learn another framework besides Rails and Django. Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?
- Zak 13y ago
  I've used it for client work that was in production and for a side-project that's in production, but private with few users. It has been reliable for me. Note that Compojure is more of a routing library than a framework. It's similar to Flask or Sinatra. Clojure's philosophy includes aggressive separation of concerns such that full-stack frameworks aren't really "the Clojure way".
abalone 13y ago
Interesting that the Play code optimizations had virtually no effect. Almost identical absolute scores between tests. Clearly something heavy's going on within the Play request handling framework to slow things down, not the code we see. I was also surprised at the difference between the Java and Scala play test, since I thought they were supposed to be similar. But it looks like the approaches are quite different. The Java DB test uses the ebean ORM while Scala does not. The JSON code also looks a bit different, with the Java version surfacing more Jackson internals. I don't know if these are necessarily "wrong", but perhaps it's not as apples-to-apples a comparison as it could be.
- bhauer 13y ago
  Agreed, and I would certainly like to see the Play tests (both Scala and Java) improve versus what we have measured so far. I would not rule out a configuration glitch in our deployment either. But on that front, I'm really hoping the Play experts can lend a hand. I think we've received a couple tips about the database connection pool size. If the contributions we've seen so far are any indicator, we're going to need to get more clever with how to show and hide rows in our results tables! :) But I'd like to see a few more rows added to cover the various permutations of the ORM and JSON options, as you point out.
  - abalone 13y ago
    Thank you Brian for this so much. Really just a fantastic effort and a great contribution to the web dev community.
- redtuesday 13y ago
  I'am certain they didn't test the versions that got merged the last two days (for example the scala version has a working db test now). So we will probably see some improvements in the next version. EDIT: @json performance difference: Seems to me like they tested these two versions: Java - https://github.com/TechEmpower/FrameworkBenchmarks/blob/5203b6e936bfc96b62b73c8db6cb15999945bb71/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/5203... Scala (the version with the not working db) - https://github.com/TechEmpower/FrameworkBenchmarks/blob/b89fbf957da97d9deb2af014c1dbd35f16e85461/play-scala/app/controllers/Application.scala https://github.com/TechEmpower/FrameworkBenchmarks/blob/b89f... The scala version uses val, which ist like final in java. final enables the JVM to cache this object an run optimizations. So I suspect the object creation only happens once in the scala case, whereas in the java case on every request. I think this is happening here which results in much better performance for the scala version. But maybe I am wrong and the difference stems only from implementation differences in the controller etc. I'm a php guy and don't know the jvm or scala very well. ;o)
infinia 13y ago
I recently started transitioning from PHP to Rails. Should I be reconsidering?
- voidlogic 13y ago
  Maybe, maybe not, that depends on your priorities- If your concern is (or a major concern) is performance, then yes, you should reconsider (maybe look at Go?). If that is not a major concern, maybe not. (I personally am not a Ruby fan due to performance and security issues, but it works well for many people)
talloaktrees 13y ago
It should be mentioned that Go 1.1 is about to be released, which includes really big speed increases (sometimes over 30%)
apunic 13y ago
Sequelize (used for the node-mysql tests) is the slowest ORM out for Node, others are much faster (i.e. the node-mysql module). And I would love to see node-mongodb tests with the native Mongo driver and not with Mongoose (which is slower).
- bhauer 13y ago
  I know this will sound like a refrain, but would you be interested in preparing a test for Node that uses a better ORM? Or if you can demonstrate that another ORM would be clearly superior, perhaps it would make more sense to simply modify the existing Node test code, replacing Sequelize?
- bsingh4 13y ago
  Something like this would be way faster than Sequelize: https://github.com/mgutz/mapper https://github.com/mgutz/mapper
apunic 13y ago
Why are there no DB tests with Go?
- bhauer 13y ago
  Disappointing answer incoming: we haven't found the time. If you know Go sufficiently to write those tests (they should be fairly easy to write for someone who works with Go regularly), we'd be very happy to accept a pull request. Also, just to reiterate something I've said elsewhere. We've done some spot testing with Go 1.1 and its performance is super. We really look forward to showing Round 3 numbers next week!
cakeface 13y ago
I like to see that Tapestry is rising high in this latest round. I have a lot of production sites running Tapestry and I'm always impressed with how fast it runs.

193 points by pfalls 13y ago

This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway. Thanks especially to those who have contributed! We hope this is useful information.

krg 13y ago
The contributions from the community have been great. To those who submitted pull requests that didn't make it into this blog post: We've been overwhelmed (in a good way) with the response and we're working to include as much as we can. Thank you!
neya 13y ago
Dear bhauer, You have no idea how valuable this is to everyone! I know it takes a lot of effort to consolidate all comments, requests, fixes suggestions, etc. Personally, I've even seen you respond on the Play! framework Google groups. Thank you for being such a down to earth person and helping out the community. You guys rock :) Thanks!
- pfalls 13y ago
  Thanks so much for the kind words. We're obviously having a great time working on this, and we too think there's a lot of value to this for the community at large.
e12e 13y ago
It's great that you're doing this, and listing stuff like standard deviation in the tables -- but I'd say your focus/interpretation of the data isn't quite right. At least provide the option to sort by standard deviation -- as that might well be more interesting than requests/second? Maybe I'm just being mean because I was reminded of this essay by Zed Shaw earlier today (I was looking for his alluded rant on CC licenses, which I didn't find): http://zedshaw.com/essays/programmer_stats.html http://zedshaw.com/essays/programmer_stats.html For instance, you state that: > In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds. Only cake php requires more than 100 milliseconds on average. But look at Django: Average around 60 ms, standard deviation around 90 ms (!). Not to mention a "worst" score of 1.4 seconds.
- pfalls 13y ago
  Thanks for the suggestion, we're still in the early stages of getting the latency information incorporated, and having the ability to sort by the various metrics makes a lot of sense.
  - vanderZwan 13y ago
    As one of the people complaining about statistics last week (and also, by coincidence, citing Zed's rant), I'm glad to see you are working on it and open to more ideas and improvements! Also, I like the "sportsmanlike benchmarking game between different communities" vibe I'm getting from all this. Would be nice if the community helps turn this into the de facto example of how to benchmark correctly. Now I'll just have to wait and see how Go 1.1 compares ;).
    - bhauer 13y ago
      Go 1.1 is looking very strong! Pat (pfalls) just showed me some very preliminary numbers and we are extremely happy with them. Thanks for the input and constructive criticism, vanderZwan. It has been very helpful to get feedback from yourself and everyone else. I too am particularly happy with the sportsmanlike competition vibe. You have no idea how fulfilling that is to us.
- krg 13y ago
  Note that the results data is available as JSON if you want to play with different sorting yourself: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/results/ec2/20130404200504/results.json https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... As pfalls said, we're definitely open to different ways to present the data. More sorting would be cool.
giulianob 13y ago
Did you guys turn on byte code caching for all the PHP frameworks? If not, then I recommend everyone ignore these benchmarks until that is completely done.
- krg 13y ago
  We did. PHP 5.4.13 with APC, using PHP-FPM, running behind nginx.
- kainsavage 13y ago
  Yes, we are using APC for all the PHP tests in this round of benchmarking.
  - keammo1 13y ago
    Is there somewhere we can see the settings for APC? Thanks for the great work!
    - krg 13y ago
      https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/config https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast... See php.ini, php-fpm.conf.
      - keammo1 13y ago
        I could be missing something, but it looks like you are using the default settings. Have you tried tweaking it all? Specifically setting apc.stat=0, which will stop it from checking the mtime. You'll need to clear the cache with apc_clear_cache() when you make code changes though. You may also want to look at apc.php to check for fragmentation and adjust apc.shm_size if necessary
        ebiester 13y ago
        fork, benchmark, and create a pull request. :)
DonnyV 13y ago
Why no .NET C# and Mono?
- errnoh 13y ago
  Their repo at https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks is open for pull requests I don't think they have anything against C#, probably no one just has committed C# version to be benched. (edit: whops, linked fork first)
  - tracker1 13y ago
    I'd be curious how it would perform on C# (windows/.net vs. linux/mono) .. though the environment setup would probably be a bit more involved.. and IIS is a very different beast. I'd think that it would probably land somewhere close to Java Servlets, but a bit slower. The framework stack for web requests in .Net is probably a bit more than it is in the servlet server in question. I would also think that Mono would be a bit slower than IIS, only because IIS does very well at pooling resources/threads for multiple requests. There's also the question of async .Net handling vs. blocking. Most .Net code I've seen is blocking, but there are async options, and as of the 4.x releases are much easier to use.
  - bhauer 13y ago
    Thanks for the link, errnoh. You're precisely right, we want to have C# and .Net tests included but didn't find the time to do so ourselves in the past week and have not yet received a pull request. I was not familiar with ServiceStack prior to feedback we received to last week's test, but from the looks of it, I'd personally like to see how it does.
- fredkingham 13y ago
  In their previous test they said that it wasn't fair to test with C# on Mono as its not as performant as C# on windows.
eranation 13y ago
First time I'm actually happy for building something with Servlets (have a feeling I'm the only one here that uses it as the go to framework for hacking something quickly, I'm an old man), it's indeed dead annoying for REST services, but doable, and blazing fast. The question is - when does my productivity starts to be more important than performance (I think 99% of the lifetime of anything I'll ever build is the first and not the latter). I'm very surprised to see Play-Scala is not in the same playing field as other JVM based frameworks, had a lot of hope for it, I hope TypeSafe will take that into consideration... Node.js is the biggest surprise for me on the good side, I think it changes my plans a little as for what to learn next... Thanks for a very valuable study, this is great
deleted 13y ago
[deleted]

coolsunglasses 13y ago

Go 1.1 beta is up, really need to get some results with tip instead of 1.0.3.

pfalls 13y ago
Go 1.1 is very high on our list, we'd love to get that into the tests as soon as we can. Of course, anyone in the Go community is welcome to issue a pull request that updated the version to 1.1. Edit: Just saw this pull request come in, so expect some Go 1.1 love in the next round. https://github.com/TechEmpower/FrameworkBenchmarks/pull/64 https://github.com/TechEmpower/FrameworkBenchmarks/pull/64
- TylerE 13y ago
  Don't think you'd need to change any code, just pull mercurial tip and recompile go and the app.
  - voidlogic 13y ago
    cd go/src hg pull hg update tip ./all.bash
    - coolsunglasses 13y ago
      That's not how their installer works. I submitted a PR to pull the 1.1 beta tarball.
  - cmsimike 13y ago
    Is 1.1 production ready? I'd feel weird about incorporating something still in a "beta" phase though.
    - coolsunglasses 13y ago
      People have been working off of tip for months.
    - 13y ago

voidlogic 13y ago

It is worth pointing out Go 1.0.3 not Go 1.1 beta is being used. I tested the code locally and on my machine 1.1 beta was about 20% "faster" (req/sec).

cmsimike 13y ago
Really? We're definitely going to try that out!
- voidlogic 13y ago
  In addition to the improvements from the improved scheduler, use of accept4 on Linux, better GC and code generation, Brad Fitzpatrick has been giving a lot of love to the "net/http" package. Here is a small example: https://plus.google.com/u/0/115863474911002159675/posts/L3o9hEs8SAe https://plus.google.com/u/0/115863474911002159675/posts/L3o9...
- errnoh 13y ago
  Yeah, 20% is even quite low estimate. Here's 1.0.3 vs 1.1 from just now (request worker (wrk) and Go http server running in separate machines) https://gist.github.com/errnoh/5320784 https://gist.github.com/errnoh/5320784

redtuesday 13y ago

Thanks for the tests. Especially the latency is interesting. I cant't wait for the next round, when the updates of the last two days are included (elli, grizzly, play-java, play-scala, snap etc).

pfalls 13y ago
Since we moved to a different benchmarking tool for round 2, we had to re-run all the tests, which is a time consuming process, especially on the EC2 hardware. Now that we've gotten that out of the way, we have an easier time running and processing results for individual tests. So stay tuned!

Uchikoma 13y ago

Very interesting - again. What I - again - find most interesting, is the large discrepancy in the 20-query test. Common wisdom seems to be that language performance is not important b/c every language hangs in the same IO/DB.

malachismith 13y ago
probably a combo of mysql drivers and connection pooling.
- Uchikoma 13y ago
  I wonder why Play is so bad then.

nlh 13y ago

This is an awesome project, and thank you for the follow-up. In particular, this is awesome because it's introduced me to some new frameworks that I hadn't even heard of (ie vert.x) that seem extremely interesting. Keep it up and many thanks for this contribution!

pfalls 13y ago
Thanks for the kind words. Glad to hear that you've been introduced to some new frameworks, we feel the same way, and we're excited to see what else the community has up it's sleeve.
- diminish 13y ago
  Pls could u include some C/C++ web frameworks... comparison with Java, could be interesting..
  - mitchi 13y ago
    yes please! There's got to be a C++ framework out there to test. Casablanca?
    - reactor 13y ago
      here we go! http://www.treefrogframework.org/ http://www.treefrogframework.org/
  - krg 13y ago
    We're interested in that, too. In progress is the onion http library pull request from davidmoreno: https://github.com/TechEmpower/FrameworkBenchmarks/pull/43 https://github.com/TechEmpower/FrameworkBenchmarks/pull/43 We hope to get more as well.
    - spyder 13y ago
      It would be interesting to see G-WAN too because it seems controversial. http://gwan.ch/ http://gwan.ch/

kevinburke 13y ago

It seems like you're still using Mongo DB as the datastore for some languages and Mysql as the datastore for other languages. This seems like it would bias the results.

pfalls 13y ago
The only framework that uses MongoDB exclusively is the vert.x test, the nodejs tests have tests for both MySQL and MongoDB

mythz 13y ago

Are C#/Mono web frameworks an option? Because I would love to see how http://servicestack.net http://servicestack.net fares: https://github.com/ServiceStack/ServiceStack/wiki/Real-world-performance https://github.com/ServiceStack/ServiceStack/wiki/Real-world...

giulianob 13y ago
Ditto. Would be great to also compare them to .NET VM
- misterbwong 13y ago
  +1 again. I'd like to see how .NET fares as well but I understand that this is a Linux x64 test...
pfalls 13y ago
We've had a lot of requests Mono, and are actually very interested in trying this out. We've been hesitant to move forward only because we're unsure how stable the C# community believes Mono to be. We hope to have that test at some point though, but the fastest way to get a test included is to issue a pull request. So if anyone is interested in trying this out, let us know and we can help guide the process.
- mythz 13y ago
  Cool, a note on the stability side: http://www.servicestack.net http://www.servicestack.net and the live demos http://www.servicestack.net/#livedemos http://www.servicestack.net/#livedemos have always been hosted on Linux/Mono in all the years it's been in existence. Is GitHub issues the best way to ask for guidance?
  - pfalls 13y ago
    Yes, Github issues is a good place to ask for guidance.
voidlogic 13y ago

programminggeek 13y ago

One of the variations that is being tested here, that I'm not sure if it is legitimate is that for PHP specifically, Apache is likely spinning out many PHP threads, I'm not sure if the same behavior is happening on the ruby side. Thus, if you are comparing performance of 10 PHP threads to 1 ruby or python thread, your benchmark is not going to be terribly accurate.

krg 13y ago
PHP was tested behind nginx this time due to community feedback we received, not Apache.
- programminggeek 13y ago
  Awesome, I did realize that. I stand corrected. Love the work you're doing!

jjm 13y ago

Hm... we'll need to get spray.io on this list

bhauer 13y ago
Yes, please! Spray.io folks, can we entice a pull request? :)

voidlogic 13y ago

It might be interesting to track average and peak server side memory usage during these tests. For example, if framework A is only 80% the "speed" of framework B, but uses 1/4 as much memory, for some users this might be a win for A.

pfalls 13y ago
Great suggestions, we've thought about capturing other system data as well, but memory seems like a great choice to start with.
cmsimike 13y ago
That would be an interesting metric indeed.
kainsavage 13y ago
This is definitely something we want to include in the tests, but it is difficult to manage a fair way of monitoring. Outside of the actual benchmarks, where we kept everything as fair as possible, we ran individual tests to make sure they worked prior to the benchmarking and would routinely have htop running at the same time to get an idea of what that sort of data looks like. It IS very interesting and is definitely something we have discussed methods for measuring for the sake of these benchmarks. Other areas that we discussed were cpu utilization, disk utilization, and network saturation.
- kalmar 13y ago
  One idea that jumps to mind is putting the running frameworks in an LXC container (or similar) and monitoring the memory usage of that. Not sure how accurate that is, but it's one avenue. It also still might not be fair because some runtimes can let their heap get pretty big despite not actually needing the memory. You'd want a way to differentiate those from the ones with big heaps that would choke with some memory pressure.
  - voidlogic 13y ago
    Just make sure you measure resident memory not virtual memory :)

mdlthree 13y ago

As an amateur hacker and after 275 days of reading Hacker News I feel I can now navigate in the sea of client side Javascript frameworks. With the introduction of this data on web framework performance, I now have another set of choices that I am completely unqualified to comprehend. My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and many other introductory app server frameworks. Lets say after I was able to build an initial product successfully, would I then consider re-building the product in a higher performance framework as Go, Node, etc? The third level, netty and gemini and servlet etc, I am not familiar with. Googling "netty" I get --"Netty is an asynchronous event-driven network application framework"-- I thought that is what Node is (in js) and Go does with gophers. What are the use cases for these faster frameworks and do they follow an evolution of performance options that an app might go through?

ConstantineXVI 13y ago
Most of these are bound to a language: choose a language first (and my answer to that question is "whichever one you're happiest working in and can get the job done"), then figure out which backend to use. You can fret over the framework more after you've already built something that works.
avenger123 13y ago
All metrics are meaningless without context. I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone. I find its good "food for thought" as it provides me exposure to frameworks that perhaps I can use one day but also to learn more about. The most interesting aspect is all the code for each framework to do the simple page output, the queries, etc. can be viewed for each framework. I find this an incredible learning tool as it shows the personality of each framework/language.
- pfalls 13y ago
  "I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone." This is exactly right, performance is one of many items to consider when building your own website, and while we're aiming to have a very comprehensive and useful set of performance data, we would agree that other factors need to be taken into account.

BryantD 13y ago

I apologize if this question has been asked and answered, but why is netty missing from the database tests? No ORM?

pfalls 13y ago
The short answer is that at some point, we kept adding more and more tests on our own, which kept the blog from going out, and we finally decided to have a cutoff date and let the community continue adding tests that they were interested in. I think adding a database test to Netty (as well as Go for that matter) would be a great addition.
- BryantD 13y ago
  Thanks! And thank you very much for this work -- it's tremendously useful.

kbd 13y ago

What about Puma (http://puma.io http://puma.io), Goliath (http://goliath.io http://goliath.io) or Unicorn (http://unicorn.bogomips.org/ http://unicorn.bogomips.org/) instead of Passenger in the Ruby stack?

pfalls 13y ago
We're looking into moving to unicorn as the community has suggested. If anyone is interested in setting up Puma or Goliath, we'd be interested in testing those out as well.
- cheald 13y ago
  For something like the query tests where you're IO-bound, Puma is going to annihilate the competition. For CPU-bound tests like JSON generation, unicorn with multiple workers is going to perform better. It's worth noting that Rails' default JSON solution is the compatible-everywhere pure-Ruby JSON generator. Using a drop-in replacement like the oj gem will drastically improve throughput there. I didn't get to the pull request for this round, but tweaking the GC parameters for the Ruby install should dramatically improve the general performance of the non-JRuby Ruby tests, as well. I'll see if I can get a PR in. :)
  - ksec 13y ago
    Well yes please do this, not sure if it is allowed though since Oj gem is written in C ( I think )
    - cheald 13y ago
      So is the mysql2 driver (at least the slow parts), and Passenger is C++. C extensions are a common part of your standard Ruby deployment stack.

boundlessdreamz 13y ago

This is superb. 1. More sorting options please :) I really want to sort by standard deviation. 2. Also in case of rails and other single threaded frameworks are you spinning up multiple processes? 3. Why is Go not being tested in the Database access tests? 4. Can you test rails with unicorn too?

bhauer 13y ago
1. Blame me. I put together the charts and tables. I'll try to get some sorting added for the next round! :) 2. Pat (pfalls) should know the answer to that more definitively, but we are definitely trying to use the CPU cores as fully as possible. I think you'll see some comments by Pat elsewhere in this thread suggesting some things coming in round 3. 3. Good question! Mostly because we haven't had the time to add it. I would really like to see it added, though. If you know Go and can write the code, might I entice you to submit a pull request? 4. I believe that is the plan, yes.

mike-- 13y ago

migth be interesting to add Erlang.

kainsavage 13y ago
We have at least one Erlang test that came in yesterday and will be included in the next round of benchmarks.
- mike-- 13y ago
  thanks.
  - kainsavage 13y ago
    Don't thank us, we didn't write the code for it - one of you (the community) wrote it and submitted a pull request. We just ensured that the test would work in our benchmarks, then ran it! We are happy to do it, and super appreciative of the support from you guys!

charlesjshort 13y ago

Any thoughts of putting some Erlang in there?

kainsavage 13y ago
We got a pull request for an Erlang framework (my brain is not letting me pull up the name) yesterday that we accepted, but sadly it was too late for this round of benchmarks. We will include it in the next round.

milesf 13y ago

It's a useful data point to know the speed of frameworks, but what matters most to me is speed of development and whether or not I enjoy the process. Wish there was a way to benchmark those points.

cmsimike 13y ago
I 100% agree with this. It has been a huge point of discussion for us here internally. "Developer efficiently with language x." Ultimately, language performance does matter but there are ways to make up for less performant languages. If it takes you 2x as long to get something out the door though, then that is a huge problem. Just use the language and framework you are most efficient in and enjoy using. Worry about framework performance when you notice server load increasing over time.
- malachismith 13y ago
  Sure... as long as you have the migration and rewrite plan done and ready to go, and have engineered the product for that rewrite. Otherwise you're gonna be wishing you'd failed rather than succeeded.
columbo 13y ago
>what matters most to me is speed of development and whether or not I enjoy the process. Matters most? There are 100x differences here. Take two companies writing the same application, one in Spring the other in Rails. Both have easy access to knowledgable people, both are industry standard. However the Spring application would be several orders of magnitude more scalable[1]. I'd consider that to be a more important consideration over enjoying the language. [1] All benchmarks are suspect until proven otherwise. You can write slow software in Java/C/C++, you can write fast software in Python/Ruby etc etc etc
- superuser2 13y ago
  Performant, not scalable. You can throw money at Rails and it will handle the load, but it would cost a lot more.

portmanteaufu 13y ago

This is great stuff. I'd love to see some more frameworks written in compiled native languages like Haskell and C!

saosebastiao 13y ago

Yesod disappoints! Unfortunate for how much it is hyped. This is definitely good to know, even if it appears that the Warp server was not used.

flyinRyan 13y ago
If Warp wasn't used then this isn't a proper test of it, no?
pilgrim689 13y ago
That's the first thing I noticed. What was used instead of Warp? I also see that Snap got pulled in a couple days ago, so it will probably get tested in the next round. Should be interesting to see how it fares against Yesod.
- mightybyte 13y ago
  The way the benchmark is constructed, it's really more about JSON and DB performance. Snap makes no decisions for you about those things, so the numbers really won't reflect the performance of Snap all that much. I don't have great expectations for the "Snap" code that is in there now because it is not using very fast JSON and DB libraries.
raphaelj 13y ago
I wrote the Yesod tests for this benchmark. It uses Warp. Actually, the performances of Yesod are not so bad, as Play's are. Top performing frameworks (netty, vertx, go) are not web frameworks but async programming frameworks. They don't carry a lot of things around -- just push the serialized data on the raw socket output stream. Yesod/Play/Django/Rails/... on the other hand must manage routes, content types, browser headers, etc etc....

apendleton 13y ago

I think I said something similar on the first round, but for the Python tests, I'd be curious to see alternate runtimes and/or concurrency strategies: pypy, gevent, and probably the confluence of the two (though it requires quite a bit more hacking, since gevent doesn't officially support pypy).

pfalls 13y ago
We're very curious about PyPy as well, and have it on our list https://github.com/TechEmpower/FrameworkBenchmarks/issues/15 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15
nobodysfool 13y ago
Yea, they have Flask in there, but it's on a braindead setup. gunicorn is using the 'sync' worker type, not gevent or eventlet (which is the whole point of gunicorn) and it's not preloading the app, so it's compiling the code every time the process is forked. No wonder Flask performs so poorly in this test. Yes, I've submitted a pull request with these changes.

cheriot 13y ago

As one would expect, as more data access is added, the scale of differences between web frameworks goes down. It would be interesting if you added 5 and 10 db query test runs as that starts to get closer to what a functional website would be doing.

bhauer 13y ago
Hi Cheriot, We do in fact have 5 and 10-query runs available in the data. Switch the views to "Data table" or "All samples (line chart)" and you should see the data for 1, 5, 10, 15, and 20 queries per request.
deleted 13y ago
[deleted]

eibrahim 13y ago

I would love to see how asp.net mvc compares to all this...

jimmytucson 13y ago

Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks? If not, can anyone explain in layman's terms why that should be the case?

jcroll 13y ago
Why shouldn't that be the case? PHP has been designed from the get go to be super fast and 5.4 introduced a bunch of optimizations that have sped it up even more so. Ruby was designed for programmer happiness, PHP was designed for performance and getting sh*t done.
- pdeuchler 13y ago
  PHP technically wasn't designed
  - deleted 13y ago
    [deleted]
- irahul 13y ago
  > PHP has been designed from the get go to be super fast PHP was never designed to be fast. As far as execution speed goes, PHP will be in the same ballpark as Perl?Python/Ruby. PHP comes up as faster in these tests because of all the extra things the frameworks are doing. See the_mitsuhiko comment above regarding flask for an example.
rozap 13y ago
Not sure about ruby, but django doesn't ship with connection pooling, so that is probably why its database performance is terrible. It could be configured to use pooling. With such a trivial request handler like the one in benchmark, only a small percentage of the execution is actually taken up in the view, so all of the time you're seeing in the benchmark is pure overhead of the framework. A longer running view across all of these frameworks would cause things to even out a bit. All that being said, however, it's not an unfair comparison. After all, a benchmark of this nature should be a measure of overhead because that's really all there is to measure.

alberth 13y ago

I wish Lua was tested in the benchmarks. In particular, Lua-CJSON [1] [1] http://www.kyne.com.au/~mark/software/lua-json-performance.html http://www.kyne.com.au/~mark/software/lua-json-performance.h... (I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3") EDIT: fix typo

kainsavage 13y ago
Personally, I really like Lua and I would love to see a pull request with a Lua framework test!
pfalls 13y ago
There will most certainly be a Round 3, and 4 and on and on as long as we continue to have feedback from the community. I can guarantee that we'll have a Lua test in there, but if we receive a pull request, we'll include it.
snaky 13y ago
ngx_lua [1] will beat all of them no matter which JSON library used, end of story. BTW, LuaRocks is supported [2] [1] https://github.com/chaoslawful/lua-nginx-module https://github.com/chaoslawful/lua-nginx-module [2] http://openresty.org/#UsingLuaRocks http://openresty.org/#UsingLuaRocks

bsnyder788 13y ago

I think that the play-java one would likely benefit greatly from using raw jdbc rather than the ebean ORM. In fact my best guess is this is why the scala/java versions are so far apart. In past projects, scrapping ebean always led to large performance improvements in throughput, etc.

redtuesday 13y ago
https://news.ycombinator.com/item?id=5499839 https://news.ycombinator.com/item?id=5499839 ps: i agree that raw jdbc would enhance the performance.

don_draper 13y ago

I code in Ruby, Python and Groovy/Grails. I'm amazed that Grails performs so much better than the others.

pfg 13y ago
I guess that's one of the big advantages JVM languages have. Using stuff like Spring or Hibernate (like Grails does), you benefit from thousands of person-hours spent on tuning those libraries and you still get native Java performance. Obviously there's a price you have to pay for all the things Grails built on top of Spring/Hibernate (in the benchmark it looks like Grails' performance = Spring / 2), but in general, it's still faster than just writing everything in Groovy (Ruby, Python, ...).
- msgilligan 13y ago
  It would be interesting to see the results for Grails 2.2 (which uses Groovy 2 and Invoke Dynamic) One of the nice things about Grails is that you can easily drop down to the Spring and/or JVM level when needed for performance.
  - don_draper 13y ago
    Yup. I manage a Grails app where I wanted to speed up performance on a popular part of the app. So I replace the Grails controller with a servlet. I saw a bump in performance.
vorg 13y ago
Grails performs better than what? Ruby and Python are programming languages while Grails is a web framework.

thatthatis 13y ago

This is still a json API comparison, not a full framework comparison. In the first test it sees how fast frameworks can serialize json and in the second test it sees how fast frameworks can access the database AND serialize json. An inefficient json library and a framework is doomed across all tests. There are a lot of things frameworks do beyond serving json. These benchmarks are presented as representative of broad use when really they are only representative of the frameworks if used as a json api. A raw HTML hello world, and removing the json encoding step in the db test would go a long way towards getting results that can mean what the authors seem to want them to mean. There's nothing wrong with a json API benchmark, in fact its quite valuable in a lot of cases. But if that's what this is intended to be, the authors should say so.

logic 13y ago
Agreed; for example, in our (in-house) use case, switching a django app from using either stdlib's json or simplejson to using ujson[0] was a significant performance increase when serializing/deserializing large-ish (~100MB) JSON datasets. [0]: https://pypi.python.org/pypi/ujson https://pypi.python.org/pypi/ujson

Zak 13y ago

A note about Compojure: they're using Korma for database access. Korma is an abstraction layer that's more Clojure-friendly than using SQL directly, but lighter than an ORM. Using clojure.java.jdbc might produce higher performance.

cygwin98 13y ago
The performance of Compojure was really impressive, which was almost three times as fast as play-scala. I was expecting the opposite, considering that Scala is generally a bit faster than Clojure.
- Zak 13y ago
  From what little I know about Play, it does a lot more than Compojure does. Compojure is really just a routing library. This is a benchmark of Ring (HTTP communication), Compojure (routing), Korma (database access/abstraction) and Cheshire (JSON serialization). I'm pretty sure a basic request in Play goes through more middleware than that. But yes, the performance of a reasonable web app stack on Clojure is very good.
krg 13y ago
Where possible on the tests, we tried to use something other than raw SQL because we wouldn't expect someone to use raw SQL everywhere when writing a real application. We chose Korma because it seemed decently popular for Clojure, but we're certainly open to a "compojure-raw" test that uses clojure/java.jdbc. Feel free to submit a pull request. :)
- Zak 13y ago
  I'm not saying Korma isn't a good or realistic choice. I use it myself. There did seem to be some disagreement in freenode #clojure about whether that was the best way to go, or JDBC and prepared statements. Evidently, a lot of people actually do like SQL. It's just important for people to be aware that "Compojure" actually represents "one possible production-ready stack that uses Compojure as the routing component", as do several other items on the list. I don't mean that as a criticism - just clarification.

njharman 13y ago

I find the lack of Erlang frameworks disappointing.

kainsavage 13y ago
We accepted a pull request for an Erlang framework yesterday, so they will be in the next round of benchmarks. If you want to ensure your favorite is listed, you can issue a pull request yourself on our github page.
taeric 13y ago
Instead, find it motivational. :) They are accepting pull requests.
- krg 13y ago
  In fact we've already got a pull request that includes Cowboy and Elli: https://github.com/TechEmpower/FrameworkBenchmarks/pull/50 https://github.com/TechEmpower/FrameworkBenchmarks/pull/50 We'll be including this in the next round.
  - chops 13y ago
    If I can find the time, I'll try to put together one for Nitrogen (http://nitrogenproject.com http://nitrogenproject.com) as well.

tantaman 13y ago

Does anyone know what on earth is Play! doing that makes it is so slow? I'd like to see Lift in there if you guys do another round.

bsnyder788 13y ago
My best guess is that the java one is slow due to ebean (I've always seen huge performance improvements from switching to raw jdbc). The scala one I have no clue.
- redtuesday 13y ago
  More likely because the version uses only as many threads as the cpu has cores. Whereas the servlet version for example has 128 thanks to the default resin configuration.
  - bsnyder788 13y ago
    That would definitely be an issue as well. Although, this is easily configurable within the application.conf file if I remember correctly.
maybenot 13y ago
The Play JSON controller is using Futures to do JSON serialization. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... Compare this to the Java Servlet, which just serializes directly in the Servlet callback. https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/servlet/src/main/java/hello/JsonServlet.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... The former seems like overkill.
- redtuesday 13y ago
  This is not the version which was used in the tests. The tests used a earlier version.
13y ago

jeltz 13y ago

I wonder where the bottleneck for the ruby benchmarks is. Is it in passenger, the framework, or somwhere else? I am sure I have gotten higher requests per second on my workstation when I tuned Ramaze (which should not be faster than using raw rack). My guess is that it is either due to a different method of measuring or the fact that I was using thin rather than passenger.

andybak 13y ago

Wow. I really need to use connection pooling with Django.

polskibus 13y ago

I would love to see ASP .NET and ASP .NET MVC included in the benchmark.

escaped_hn 13y ago

Is gemini really that awesome? Why arent more people talking about it?

saym 13y ago
Gemini is closed source. The company running these comparisons is the only team that is qualified to test/develop with it.
- kainsavage 13y ago
  While it is true that our source code is closed at the moment, you are still free to check out the benchmark code from github and run it yourself to verify our results. It wouldn't really be a fair test if we put all these frameworks up for testing, suggested Gemini was better than sliced bread with auto-applying jam, and didn't back it up in a testable way ;-)

wheaties 13y ago

Is it against the rules to use different workers for Django or Flask? In other words, there's a big difference when I run things with Gunicorn+Gevent+psychopg (PostGres async driver) than when I run it just with Gunicorn alone.

pfalls 13y ago
Each test can be modified in isolation, if both tests see an improvement in that setup, we would like to include the changes for both though.

kriro 13y ago

Cool stuff. Only looked over it quickly but it seems JAVA stuff is dominating. Kind of curious why jruby doesn't provide more of a boost (especially given compojure doing relatively well, too) and would like to see Django+Jython in version 3 (and +1 to the .NET/Mono stuff)

seunosewa 13y ago

I wish you would have tested minimalist web frameworks in Python: CherryPy and Bottle. Flask is not really a micro-framework in the same way that CherryPy and Bottle are micro-frameworks because of its numerous dependencies.

jebblue 13y ago

Servlets are super fast, this shows that.

VeejayRampay 13y ago

Gemini conveniently leads the pack all the time. So either OP and his/her colleagues have revolutionized web programming or the benchmark is nothing but a PR piece.

krg 13y ago
Just ignore Gemini. In fact, you can click the "hide" link on the charts to remove it. From our first benchmarks post: http://www.techempower.com/blog/2013/03/28/framework-benchmarks/ http://www.techempower.com/blog/2013/03/28/framework-benchma... "Why include this Gemini framework I've never heard of?" We have included our in-house Java web framework, Gemini, in our tests. We've done so because it's of interest to us. You can consider it a stand-in for any relatively lightweight minimal-locking Java framework. While we're proud of how it performs among the well-established field, this exercise is not about Gemini. We routinely use other frameworks on client projects and we want this data to inform our recommendations for new projects.

fsiefken 13y ago

Again no mono with ServiceStack! It boggles my mind...

gourneau 13y ago

As a Django guy with hurt feeling right now, I just wanted to say that this is a great project. I look forward to the next versions.

tracker1 13y ago

Where's the source for the implementations? Of interest to me is the NodeJS instances.. was this single-process, or multi-process via cluster (which is recommended for higher performance). The only source I saw was for the testing framework. --- edit: found it. https://github.com/TechEmpower/FrameworkBenchmarks https://github.com/TechEmpower/FrameworkBenchmarks Seems to be using cluster. I'm not surprised that Go and servlets were faster... just surprised to see NodeJS in the middle of the pack position. Though performance isn't the only reason I really like NodeJS

mojomama 13y ago

No love for Mojolicious? It would be great to see a little Perl here!

voidlogic 13y ago
Submit your pull request to add it?

just2n 13y ago

Once again, the Node examples are wrong. - Unnecessary parsing. - Writing strings instead of buffers (they're copied, they aren't sent as-is). - Using async. It does a lot of really nasty things, most of which break V8 optimization best practices. This is a perf benchmark, not a comparison of how concise your code can be. - Having the main request handler in a gigantic function that will never be properly optimized by V8. - Not getting helper functions that are clearly monomorphic warm before accepting requests. While the code itself is what I would consider fine, when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code.

kaoD 13y ago
On the one hand, you're completely right. On the other hand, shouldn't real code be tested? How accurate would a ugly-optimized-code be when you SHOULD write reusable code in your real life projects?
- just2n 13y ago
  I wouldn't use async myself, so what's "real" code varies from person to person. This is why benchmarks like this are highly misleading. But ideally, yes. We'd be better informed to see how real world code performs rather than some idyllic perf-oriented creature.
vanderZwan 13y ago
So do a pull request and create a node.js equivalent of the stripped editions of rails and django. I'm sure they're open to it.
ldh 13y ago
> when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code That seems to defeat the purpose of benchmarking to begin with, or at least makes it less useful for real-world comparison.
- 13y ago

saadazzz 13y ago

Any plans for ASP.NET?

superuser2 13y ago

This is definitely motivation to learn another framework besides Rails and Django. Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?

Zak 13y ago
I've used it for client work that was in production and for a side-project that's in production, but private with few users. It has been reliable for me. Note that Compojure is more of a routing library than a framework. It's similar to Flask or Sinatra. Clojure's philosophy includes aggressive separation of concerns such that full-stack frameworks aren't really "the Clojure way".

abalone 13y ago

Interesting that the Play code optimizations had virtually no effect. Almost identical absolute scores between tests. Clearly something heavy's going on within the Play request handling framework to slow things down, not the code we see. I was also surprised at the difference between the Java and Scala play test, since I thought they were supposed to be similar. But it looks like the approaches are quite different. The Java DB test uses the ebean ORM while Scala does not. The JSON code also looks a bit different, with the Java version surfacing more Jackson internals. I don't know if these are necessarily "wrong", but perhaps it's not as apples-to-apples a comparison as it could be.

bhauer 13y ago
Agreed, and I would certainly like to see the Play tests (both Scala and Java) improve versus what we have measured so far. I would not rule out a configuration glitch in our deployment either. But on that front, I'm really hoping the Play experts can lend a hand. I think we've received a couple tips about the database connection pool size. If the contributions we've seen so far are any indicator, we're going to need to get more clever with how to show and hide rows in our results tables! :) But I'd like to see a few more rows added to cover the various permutations of the ORM and JSON options, as you point out.
- abalone 13y ago
  Thank you Brian for this so much. Really just a fantastic effort and a great contribution to the web dev community.
redtuesday 13y ago
I'am certain they didn't test the versions that got merged the last two days (for example the scala version has a working db test now). So we will probably see some improvements in the next version. EDIT: @json performance difference: Seems to me like they tested these two versions: Java - https://github.com/TechEmpower/FrameworkBenchmarks/blob/5203b6e936bfc96b62b73c8db6cb15999945bb71/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/5203... Scala (the version with the not working db) - https://github.com/TechEmpower/FrameworkBenchmarks/blob/b89fbf957da97d9deb2af014c1dbd35f16e85461/play-scala/app/controllers/Application.scala https://github.com/TechEmpower/FrameworkBenchmarks/blob/b89f... The scala version uses val, which ist like final in java. final enables the JVM to cache this object an run optimizations. So I suspect the object creation only happens once in the scala case, whereas in the java case on every request. I think this is happening here which results in much better performance for the scala version. But maybe I am wrong and the difference stems only from implementation differences in the controller etc. I'm a php guy and don't know the jvm or scala very well. ;o)

infinia 13y ago

I recently started transitioning from PHP to Rails. Should I be reconsidering?

voidlogic 13y ago
Maybe, maybe not, that depends on your priorities- If your concern is (or a major concern) is performance, then yes, you should reconsider (maybe look at Go?). If that is not a major concern, maybe not. (I personally am not a Ruby fan due to performance and security issues, but it works well for many people)

talloaktrees 13y ago

It should be mentioned that Go 1.1 is about to be released, which includes really big speed increases (sometimes over 30%)

apunic 13y ago

Sequelize (used for the node-mysql tests) is the slowest ORM out for Node, others are much faster (i.e. the node-mysql module). And I would love to see node-mongodb tests with the native Mongo driver and not with Mongoose (which is slower).

bhauer 13y ago
I know this will sound like a refrain, but would you be interested in preparing a test for Node that uses a better ORM? Or if you can demonstrate that another ORM would be clearly superior, perhaps it would make more sense to simply modify the existing Node test code, replacing Sequelize?
bsingh4 13y ago
Something like this would be way faster than Sequelize: https://github.com/mgutz/mapper https://github.com/mgutz/mapper

apunic 13y ago

Why are there no DB tests with Go?

bhauer 13y ago
Disappointing answer incoming: we haven't found the time. If you know Go sufficiently to write those tests (they should be fairly easy to write for someone who works with Go regularly), we'd be very happy to accept a pull request. Also, just to reiterate something I've said elsewhere. We've done some spot testing with Go 1.1 and its performance is super. We really look forward to showing Round 3 numbers next week!

cakeface 13y ago

I like to see that Tapestry is rising high in this latest round. I have a lot of production sites running Tapestry and I'm always impressed with how fast it runs.