7 ms·

I'd love to see your framework optimization index. Honestly, all of this would be a wonderful thing to automate and put in a web app - a readily-accessible, up-

by cacois 14y ago

I'd love to see your framework optimization index. Honestly, all of this would be a wonderful thing to automate and put in a web app - a readily-accessible, up-to-date measure of the current performance of the state of the art in languages and frameworks. I bet it would really change some of the technology choices made.

goodwink 14y ago
Here's a quick version of the framework optimization index. Higher is better (ratio of framework performance to raw platform performance, multiplied by 100 for scale): Framework Framework Index Gemini 87.88 Vert.x 76.29 Express 68.85 Sinatra-Ruby 67.88 WebGo 51.08 Compojure 45.69 Rails-Ruby 31.75 Wicket 29.33 Rails-Jruby 20.09 Play 18.02 Sinatra-Jruby 15.96 Tapestry 13.57 Spring 13.48 Grails 7.11 Cake 1.17
- goodwink 14y ago
  These could probably be further broken down into micro-frameworks (like Express, Sinatra, Vert.x etc.) and large MVC frameworks (like Play and Rails). Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected. Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young. Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
  - jholla14 14y ago
    The play test was written totally incorrectly since it used blocking database calls. Since play is really just a layer on top of Netty it should perform nearly as well if correctly written.
    - goodwink 14y ago
      I believe they're encouraging pull requests to fix that sort of thing. It will be interesting to see if it helps to that degree; I hope so!
    - abalone 14y ago
      But Play's trivial JSON test was much slower than Netty's.
      - mrspeaker 14y ago
        That's because they inexplicably use the Jackson library for the simple test, rather than Play's built in JSON support (they use the built-in JSON for the other benchmarks).
        rzidane 14y ago
        Both Netty and Play use Jackson though one the Netty version uses a single ObjectMapper and the Play version uses a new ObjectNode per request (created through Play's Json library).
        ms-tg 14y ago
        I hope this gets fixed!
        abalone 14y ago
        No, they use Play's JSON lib. It's kind of a moot point because Play's lib is in fact a wrapper for Jackson. Here's the source: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... So the question stands: If Play & Netty are using the same JSON serialization code, why is Play seven times slower?
  - ctide 14y ago
    I'd imagine that being relatively young is an advantage in a test like this. You're not utilizing any features, and features are what slow down requests. The less features something has, the faster it should perform in these trivial tests.
    - goodwink 14y ago
      That's a very good point; I hadn't thought of it that way. Maybe this is some small part of why we seem to keep flocking to the new kids on the block.
- bhauer 14y ago
  It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily interpreted as too much editorializing. Nevertheless, as you point out, it's interesting to know how using a framework impacts your performance versus the underlying platform. I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
  - sluukkonen 14y ago
    For Play, you'll want to either 1) handle the (blocking) database queries using Futures/Actors that use a separate thread pool (this might be easier to do in Scala) or 2) bump the default thread pool sizes considerably. The default configuration is optimized purely for non-blocking I/O. See e.g. https://github.com/playframework/Play20/blob/master/documentation/manual/detailledTopics/configuration/ThreadPools.md https://github.com/playframework/Play20/blob/master/document... and https://gist.github.com/guillaumebort/2973705 https://gist.github.com/guillaumebort/2973705 for more info.
    - rzidane 14y ago
      Why would the JSON serialization test perform so poorly though?
  - kclay 14y ago
    I'm pretty shocked that Play scored so low. One would think that being built on netty would put Play in a higher rank. Database access for sure needs to be in async block
    - bhauer 14y ago
      Agreed. We need to get the Play benchmark code updated to do the database queries in an asynchronous block. Accepting pull requests! :)
- rallison 14y ago
  In the same vein, I was curious to compare the max responses/second on dedicated hardware vs ec2 on a per framework basis. The following is percentage throughput of ec2 vs dedicated (in res/s): cake 18.9% (312 vs 59) compojure 12.1% (108588 vs 13135) django 16.8% (6879 vs 1156) express 16.9% (42867 vs 7258) gemini 12.5% (202727 vs 25264) go 13.3% (100948 vs 13472) grails 7.1% (28995 vs 2045) netty 18% (203970 vs 36717) nodejs 15.6% (67491 vs 10541) php 11.6% (43397 vs 5054) play 20.6% (25164 vs 5181) rack-jruby 15.6% (27874 vs 4336) rack-ruby 22.7% (9513 vs 2164) rails-jruby 22.7% (3841 vs 871) rails-ruby 20.7% (3324 vs 687) servlet 13.4% (213322 vs 28745) sinatra-jruby 21.2% (3261 vs 692) sinatra-ruby 22.2% (6619 vs 1469) spring 7.1% (54679 vs 3874) tapestry 5.2% (75002 vs 3901) vertx 22.3% (125711 vs 28012) webgo 13.5% (51091 vs 6881) wicket 12.7% (66441 vs 8431) wsgi 14.8% (21139 vs 3138) I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown. Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2. Disclosure: I am a colleague of the author of the article.
  - oldpond 14y ago
    You're saying that running a query across the internet to ec2 is 5 times faster than running it on dedicated hardware in the lab? I find that hard to believe.
    - rallison 14y ago
      Sorry, maybe my original post was not entirely clear. Let's take tapestry, for example. On dedicated hardware, the peak throughput in responses per second was 75,002. On ec2, it was 3,901 responses per second. So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.

by cacois 14y ago

goodwink 14y ago

Here's a quick version of the framework optimization index. Higher is better (ratio of framework performance to raw platform performance, multiplied by 100 for scale): Framework Framework Index Gemini 87.88 Vert.x 76.29 Express 68.85 Sinatra-Ruby 67.88 WebGo 51.08 Compojure 45.69 Rails-Ruby 31.75 Wicket 29.33 Rails-Jruby 20.09 Play 18.02 Sinatra-Jruby 15.96 Tapestry 13.57 Spring 13.48 Grails 7.11 Cake 1.17

goodwink 14y ago
These could probably be further broken down into micro-frameworks (like Express, Sinatra, Vert.x etc.) and large MVC frameworks (like Play and Rails). Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected. Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young. Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
- jholla14 14y ago
  The play test was written totally incorrectly since it used blocking database calls. Since play is really just a layer on top of Netty it should perform nearly as well if correctly written.
  - goodwink 14y ago
    I believe they're encouraging pull requests to fix that sort of thing. It will be interesting to see if it helps to that degree; I hope so!
  - abalone 14y ago
    But Play's trivial JSON test was much slower than Netty's.
    - mrspeaker 14y ago
      That's because they inexplicably use the Jackson library for the simple test, rather than Play's built in JSON support (they use the built-in JSON for the other benchmarks).
      - rzidane 14y ago
        Both Netty and Play use Jackson though one the Netty version uses a single ObjectMapper and the Play version uses a new ObjectNode per request (created through Play's Json library).
        ms-tg 14y ago
        I hope this gets fixed!
      - abalone 14y ago
        No, they use Play's JSON lib. It's kind of a moot point because Play's lib is in fact a wrapper for Jackson. Here's the source: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/play-java/app/controllers/Application.java https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... So the question stands: If Play & Netty are using the same JSON serialization code, why is Play seven times slower?
- ctide 14y ago
  I'd imagine that being relatively young is an advantage in a test like this. You're not utilizing any features, and features are what slow down requests. The less features something has, the faster it should perform in these trivial tests.
  - goodwink 14y ago
    That's a very good point; I hadn't thought of it that way. Maybe this is some small part of why we seem to keep flocking to the new kids on the block.
bhauer 14y ago
It's funny you should put this together because in an earlier draft of this blog entry I had created a tongue-in-cheek unit to express the average cost per additional line of code. Based on our Cake PHP numbers, I wanted to describe PHP as having the highest average cost per line of code. But we dropped this because I felt it ultimately wasn't fair to say that based on the limited data we had and it could be easily interpreted as too much editorializing. Nevertheless, as you point out, it's interesting to know how using a framework impacts your performance versus the underlying platform. I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
- sluukkonen 14y ago
  For Play, you'll want to either 1) handle the (blocking) database queries using Futures/Actors that use a separate thread pool (this might be easier to do in Scala) or 2) bump the default thread pool sizes considerably. The default configuration is optimized purely for non-blocking I/O. See e.g. https://github.com/playframework/Play20/blob/master/documentation/manual/detailledTopics/configuration/ThreadPools.md https://github.com/playframework/Play20/blob/master/document... and https://gist.github.com/guillaumebort/2973705 https://gist.github.com/guillaumebort/2973705 for more info.
  - rzidane 14y ago
    Why would the JSON serialization test perform so poorly though?
- kclay 14y ago
  I'm pretty shocked that Play scored so low. One would think that being built on netty would put Play in a higher rank. Database access for sure needs to be in async block
  - bhauer 14y ago
    Agreed. We need to get the Play benchmark code updated to do the database queries in an asynchronous block. Accepting pull requests! :)
rallison 14y ago
In the same vein, I was curious to compare the max responses/second on dedicated hardware vs ec2 on a per framework basis. The following is percentage throughput of ec2 vs dedicated (in res/s): cake 18.9% (312 vs 59) compojure 12.1% (108588 vs 13135) django 16.8% (6879 vs 1156) express 16.9% (42867 vs 7258) gemini 12.5% (202727 vs 25264) go 13.3% (100948 vs 13472) grails 7.1% (28995 vs 2045) netty 18% (203970 vs 36717) nodejs 15.6% (67491 vs 10541) php 11.6% (43397 vs 5054) play 20.6% (25164 vs 5181) rack-jruby 15.6% (27874 vs 4336) rack-ruby 22.7% (9513 vs 2164) rails-jruby 22.7% (3841 vs 871) rails-ruby 20.7% (3324 vs 687) servlet 13.4% (213322 vs 28745) sinatra-jruby 21.2% (3261 vs 692) sinatra-ruby 22.2% (6619 vs 1469) spring 7.1% (54679 vs 3874) tapestry 5.2% (75002 vs 3901) vertx 22.3% (125711 vs 28012) webgo 13.5% (51091 vs 6881) wicket 12.7% (66441 vs 8431) wsgi 14.8% (21139 vs 3138) I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown. Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2. Disclosure: I am a colleague of the author of the article.
- oldpond 14y ago
  You're saying that running a query across the internet to ec2 is 5 times faster than running it on dedicated hardware in the lab? I find that hard to believe.
  - rallison 14y ago
    Sorry, maybe my original post was not entirely clear. Let's take tapestry, for example. On dedicated hardware, the peak throughput in responses per second was 75,002. On ec2, it was 3,901 responses per second. So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.