7 ms·

Seeking new test ideas for Framework Benchmarks Round 4+

38 points by bhauer 14y ago

bhauer 14y ago
The community has contributed even more pull requests since Round 2 of our web application framework benchmarks. We're planning to start Round 3 tests on Monday 4/15 using a new build of Wrk that allows time-limited tests (rather than request-count limited) so that all frameworks are tested for a uniform amount of time. In the meantime, this Github issue request is seeking thoughts you may have concerning additional simple tests that we can introduce in Round 4 and beyond. We want to define tests that continue to exercise typical web application functionality but remain fairly simple to implement on a ever-widening field of frameworks. If you have thoughts, please add them here or at Github. Thanks!
- wheaties 14y ago
  I have got to do a fork/pull before then in my infinite spare time but a quick question, why only ORM? Excuse my ignorance but there are almost no ORM that I've used which are not materially slower than straight SQL or json, the exception being g Salat on MongoDB. I wonder what not having an ORM would do to some of the poor performance experienced.
  - bhauer 14y ago
    We expect that most modern webapps are (still) developed with an ORM, but we're not ORM hard-liners. A few of the tests are run without an ORM and are identified with the suffix "raw."
- Terretta 14y ago
  Thoughts of incremental web app functionality to test: 1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D. 2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example. (The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.) 3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc. 4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered. 5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache. For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.
  - bhauer 14y ago
    This is great input, Terretta. Exactly the kind of thinking I wanted to tap into. Some quick thoughts in response: For #1, the conceptual test included reading and writing in a 1:1 R:W ratio (well, to be more accurate 1:1 R:U). I like the idea of extending this a little bit to include C and D. For the sake of benchmark run-time, I'm looking to restrain the growth of permutations. But on the other hand, I like your idea of a tunable ratio. Something to think about! I like #2 and #3 as well. I'll think about those some more too. I really like the idea of incorporating some bcrypt and session state (#4). We have a few caching tests in mind, but like elsewhere, we'll start out simple and then add complexity. Thanks for the great input! This planning is to have some good long-term ideas in mind.
coolsunglasses 14y ago
Can't wait to see how Go 1.1 does in round 3 :) For anybody else interested in nonsense like this I have a Clojure template benchmark repository on Github here: https://github.com/bitemyapp/clojure-template-benchmarks https://github.com/bitemyapp/clojure-template-benchmarks I should probably update the clabango benchmark, some changes were made not too long ago.
- bhauer 14y ago
  My oh my those are some awesome sunglasses on that bear. We've got Go 1.1 in Round 3 and it's amazing.
darkchasma 14y ago
ASP.Net MVC?
- bhauer 14y ago
  Sadly, we still haven't fit this in and no pull requests to-date. A Mono pull request would cause heartfelt cheers.
  - streetwiseherc 14y ago
    If you're interested, I have replicated the /json and /db services for a few of the popular ASP.NET options for creating JSON services. - mono-httphandler - mono-mvc-dapper - mono-mvc-nhibernate - mono-mvc-raw - mono-servicestack Notes: 1) All projects use MySQL as the database. 2) mono-httphandler uses bare-bones HttpHandlers and will probably be the best performer because there is no real "framework" involved. It mirrors the Java servlet example. As I believe JSON serialization is one of the key indicators of performance, it has the option of using 3 different JSON serializers popular in the .NET world: NewtonSoft, ServiceStack and the default ASP.NET MVC serializer from System.Web.Extensions.dll. At present the rest of the ASP.NET MVC projects are using the default JSON serializer and should probably changed to use the fastest option (as servlet uses Jackson) which is probably going to be ServiceStack's serializer. 3) Dapper is a micro ORM created by Sam Saffron of StackOverflow fame. More info here: http://samsaffron.com/archive/2011/03/30/How+I+learned+to+stop+worrying+and+write+my+own+ORM http://samsaffron.com/archive/2011/03/30/How+I+learned+to+st... 4) Nhibernate is a popular full-featured ORM that is similar to Java's Hibernate ORM. 5) mono-mvc-raw uses bare-bones ADO.NET calls for database access. 6) mono-servicestack uses the ServiceStack framework which is supposed to have top performance for creating REST web services using .NET. It has it's own micro-ORM and JSON serializer which are supposed to be high performers. More info here: http://www.servicestack.net/benchmarks/ http://www.servicestack.net/benchmarks/ 7) AFAIK, the new Web API is not supported on Mono. See here: http://stackoverflow.com/questions/14126855/is-mono-capable-of-hosting-asp-net-mvc-webapi-in-mod-mono-yet http://stackoverflow.com/questions/14126855/is-mono-capable-...
  - streetwiseherc 14y ago
    https://github.com/streetwiseherc/TechEmpowerFrameworkBenchmarks https://github.com/streetwiseherc/TechEmpowerFrameworkBenchm... I've run these locally and they appear to work. I'll test them on an EC2 instance later and write the setup/config scripts. When is the next time you're running the benchmarks?
CoffeeDregs 14y ago
As a passive consumer of the benchmarks/info, I've got to say a huge "Thanks" for this! I've been a watcher of the Debian benchmarks game for a long time (since its inception), but, after using Python, Ruby, Haskell, etc, had written off JVM languages as either slow (e.g. Groovy) or non-expressive (e.g. Java)... This benchmark has got me seriously interested in JVM languages again. (Note: I don't want to use Scala^H^H^H^H^HFrankenstein) Given the computational ability of browsers, the "front end" is rich and the "back end" is shrinking down to just the API. I'm not sure that the extra verbosity of Java is too large a cost for the performance it offers. I currently have an application doing 1500 requests a second on Django and it's using 3 servers. I could have used only one server if I'd used Java or Clojure? Hmmm...
- coolsunglasses 14y ago
  > I could have used only one server if I'd used Java or Clojure? Hmmm... Use Clojure, but keep in mind that if you're not caching aggressively you can saturate persistence-side quickly - making your choice of application language irrelevant.
  - spullara 14y ago
    Sadly that isn't very true anymore. Databases (and I/O — think SSD) really are faster than most of the dynamic languages these days.
    - coolsunglasses 14y ago
      That's a pretty nonsense sentiment. How can a database be "faster" than a language?
      - spullara 14y ago
        For example, going to the database and retrieving the data uses less CPU and time than deserializing the data into an object into Ruby. That is the sense I am talking about. The bottleneck becomes the code talking to the database rather than the database itself.
happyhappy007 14y ago
I would like to see a simple CRUD blog built with different frameworks. Building a blog is like the "Hello world!" for dynamic web development.
- coolsunglasses 14y ago
  Given the scope and breadth involved in these benchmarks, that's a helluva tall order. I'm sure nothing's stopping anybody from doing it themselves though.
  - kbenson 14y ago
    I think that this could be specced out in stages, and implemented an a number of rounds. First would be a schema for a blog, with authors, posts an comments. Next would be a rest API for posts and comments. Finally mock pages to be used for posting, reading and commenting in HTML to test the included templating system, if there is one. You really need to go at least this far. This will also give you an approximate code size for thus sample project as well, which is is at least as important as performance to some people.
    - coolsunglasses 14y ago
      >You really need to go at least this far. Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Why not just go do it, if it's so essential and straightforward?
      - kbenson 14y ago
        > Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Okay, let me clarify. When I say "you really need to go this far" I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. I agree you never see anything approaching that in other reviews/benchmarks. Is that a good reason to not try it here? > Why not just go do it, if it's so essential and straightforward? I'm envisioning this as a community process, not a "Go off and write this in 20 frameworks or you're useless" sort or ultimatum. As such, just speccing out a possibly route is helping. Also, I plan to help with the existing benchmarks. After the second round, I pointed them out to the author of my favorite framework in the hopes he would have time to put together something for the benchmark, otherwise I was going to in the next week or two when I had time. I still plan to. Oh, and that framework author's answer? That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. Sure, there's possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far?
crypto5 14y ago
Do we have any chance to look at round 3 results?
- bhauer 14y ago
  Of course! We are starting the tests next Monday and you can expect them to be complete some time around the middle of next week.
ckluis 14y ago
@bhauer - this may be the best marketing I have ever seen a company do in the tech space (and by marketing I mean marketing to developers). This is a recruiting goldmine. Sheer genius.
voidlogic 14y ago
What about tracking memory usage? Peak, average, etc. Previously you tested EC2 vs Local HW. What about adding local KVM virtual machine as well? I also think a graph showing latency as a function of concurrency would be very interesting.
- bhauer 14y ago
  Thanks for the ideas, voidlogic. We do want to capture server statistics and have that as an issue in Github [1]. I am particularly interested in capturing CPU and I/O utilization because in spot checks, we've observed some frameworks do not fully saturate the 8 HT cores on our i7 hardware, suggesting lock or resource contention. As for a variety of other hardware and VM environments, the data would be interesting. Related: we plan to migrate the charts and tables to a stand-alone page. Right now, the blog entries are hard-coded to fetch two specific results.json files for rendering the charts/tables. But when we build a stand-alone page, I would like to enhance the script to allow selection of one or two results.json files from a menu for comparing side-by-side. And to your point, the community could then contribute their own results files. Imagine being able to compare EC2 large vs xlarge or vs Xeon E5s or ... Right now, as you noticed, the latency is only displayed at 256 concurrency. I'll make a note to myself to include a chart for latency versus concurrency when we move to a stand-alone page [2]. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/149 https://github.com/TechEmpower/FrameworkBenchmarks/issues/14...

38 points by bhauer 14y ago

The community has contributed even more pull requests since Round 2 of our web application framework benchmarks. We're planning to start Round 3 tests on Monday 4/15 using a new build of Wrk that allows time-limited tests (rather than request-count limited) so that all frameworks are tested for a uniform amount of time. In the meantime, this Github issue request is seeking thoughts you may have concerning additional simple tests that we can introduce in Round 4 and beyond. We want to define tests that continue to exercise typical web application functionality but remain fairly simple to implement on a ever-widening field of frameworks. If you have thoughts, please add them here or at Github. Thanks!

wheaties 14y ago
I have got to do a fork/pull before then in my infinite spare time but a quick question, why only ORM? Excuse my ignorance but there are almost no ORM that I've used which are not materially slower than straight SQL or json, the exception being g Salat on MongoDB. I wonder what not having an ORM would do to some of the poor performance experienced.
- bhauer 14y ago
  We expect that most modern webapps are (still) developed with an ORM, but we're not ORM hard-liners. A few of the tests are run without an ORM and are identified with the suffix "raw."
Terretta 14y ago
Thoughts of incremental web app functionality to test: 1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D. 2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example. (The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.) 3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc. 4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered. 5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache. For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.
- bhauer 14y ago
  This is great input, Terretta. Exactly the kind of thinking I wanted to tap into. Some quick thoughts in response: For #1, the conceptual test included reading and writing in a 1:1 R:W ratio (well, to be more accurate 1:1 R:U). I like the idea of extending this a little bit to include C and D. For the sake of benchmark run-time, I'm looking to restrain the growth of permutations. But on the other hand, I like your idea of a tunable ratio. Something to think about! I like #2 and #3 as well. I'll think about those some more too. I really like the idea of incorporating some bcrypt and session state (#4). We have a few caching tests in mind, but like elsewhere, we'll start out simple and then add complexity. Thanks for the great input! This planning is to have some good long-term ideas in mind.

coolsunglasses 14y ago

Can't wait to see how Go 1.1 does in round 3 :) For anybody else interested in nonsense like this I have a Clojure template benchmark repository on Github here: https://github.com/bitemyapp/clojure-template-benchmarks https://github.com/bitemyapp/clojure-template-benchmarks I should probably update the clabango benchmark, some changes were made not too long ago.

bhauer 14y ago
My oh my those are some awesome sunglasses on that bear. We've got Go 1.1 in Round 3 and it's amazing.

darkchasma 14y ago

ASP.Net MVC?

bhauer 14y ago
Sadly, we still haven't fit this in and no pull requests to-date. A Mono pull request would cause heartfelt cheers.
- streetwiseherc 14y ago
  If you're interested, I have replicated the /json and /db services for a few of the popular ASP.NET options for creating JSON services. - mono-httphandler - mono-mvc-dapper - mono-mvc-nhibernate - mono-mvc-raw - mono-servicestack Notes: 1) All projects use MySQL as the database. 2) mono-httphandler uses bare-bones HttpHandlers and will probably be the best performer because there is no real "framework" involved. It mirrors the Java servlet example. As I believe JSON serialization is one of the key indicators of performance, it has the option of using 3 different JSON serializers popular in the .NET world: NewtonSoft, ServiceStack and the default ASP.NET MVC serializer from System.Web.Extensions.dll. At present the rest of the ASP.NET MVC projects are using the default JSON serializer and should probably changed to use the fastest option (as servlet uses Jackson) which is probably going to be ServiceStack's serializer. 3) Dapper is a micro ORM created by Sam Saffron of StackOverflow fame. More info here: http://samsaffron.com/archive/2011/03/30/How+I+learned+to+stop+worrying+and+write+my+own+ORM http://samsaffron.com/archive/2011/03/30/How+I+learned+to+st... 4) Nhibernate is a popular full-featured ORM that is similar to Java's Hibernate ORM. 5) mono-mvc-raw uses bare-bones ADO.NET calls for database access. 6) mono-servicestack uses the ServiceStack framework which is supposed to have top performance for creating REST web services using .NET. It has it's own micro-ORM and JSON serializer which are supposed to be high performers. More info here: http://www.servicestack.net/benchmarks/ http://www.servicestack.net/benchmarks/ 7) AFAIK, the new Web API is not supported on Mono. See here: http://stackoverflow.com/questions/14126855/is-mono-capable-of-hosting-asp-net-mvc-webapi-in-mod-mono-yet http://stackoverflow.com/questions/14126855/is-mono-capable-...
- streetwiseherc 14y ago
  https://github.com/streetwiseherc/TechEmpowerFrameworkBenchmarks https://github.com/streetwiseherc/TechEmpowerFrameworkBenchm... I've run these locally and they appear to work. I'll test them on an EC2 instance later and write the setup/config scripts. When is the next time you're running the benchmarks?

CoffeeDregs 14y ago

As a passive consumer of the benchmarks/info, I've got to say a huge "Thanks" for this! I've been a watcher of the Debian benchmarks game for a long time (since its inception), but, after using Python, Ruby, Haskell, etc, had written off JVM languages as either slow (e.g. Groovy) or non-expressive (e.g. Java)... This benchmark has got me seriously interested in JVM languages again. (Note: I don't want to use Scala^H^H^H^H^HFrankenstein) Given the computational ability of browsers, the "front end" is rich and the "back end" is shrinking down to just the API. I'm not sure that the extra verbosity of Java is too large a cost for the performance it offers. I currently have an application doing 1500 requests a second on Django and it's using 3 servers. I could have used only one server if I'd used Java or Clojure? Hmmm...

coolsunglasses 14y ago
> I could have used only one server if I'd used Java or Clojure? Hmmm... Use Clojure, but keep in mind that if you're not caching aggressively you can saturate persistence-side quickly - making your choice of application language irrelevant.
- spullara 14y ago
  Sadly that isn't very true anymore. Databases (and I/O — think SSD) really are faster than most of the dynamic languages these days.
  - coolsunglasses 14y ago
    That's a pretty nonsense sentiment. How can a database be "faster" than a language?
    - spullara 14y ago
      For example, going to the database and retrieving the data uses less CPU and time than deserializing the data into an object into Ruby. That is the sense I am talking about. The bottleneck becomes the code talking to the database rather than the database itself.

happyhappy007 14y ago

I would like to see a simple CRUD blog built with different frameworks. Building a blog is like the "Hello world!" for dynamic web development.

coolsunglasses 14y ago
Given the scope and breadth involved in these benchmarks, that's a helluva tall order. I'm sure nothing's stopping anybody from doing it themselves though.
- kbenson 14y ago
  I think that this could be specced out in stages, and implemented an a number of rounds. First would be a schema for a blog, with authors, posts an comments. Next would be a rest API for posts and comments. Finally mock pages to be used for posting, reading and commenting in HTML to test the included templating system, if there is one. You really need to go at least this far. This will also give you an approximate code size for thus sample project as well, which is is at least as important as performance to some people.
  - coolsunglasses 14y ago
    >You really need to go at least this far. Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Why not just go do it, if it's so essential and straightforward?
    - kbenson 14y ago
      > Oh please. Nobody's ever done such a comparison across more than 2 or 3 languages or frameworks. Have you even seen the benchmarks in question? Okay, let me clarify. When I say "you really need to go this far" I mean that stopping at any point before that (but after the simple metric of how many requests a second it can serve which they already do) makes no sense, IMHO. If you are going to compare frameworks and you want to go beyond that initial performance metric, you might as well aim high enough to be useful. I agree you never see anything approaching that in other reviews/benchmarks. Is that a good reason to not try it here? > Why not just go do it, if it's so essential and straightforward? I'm envisioning this as a community process, not a "Go off and write this in 20 frameworks or you're useless" sort or ultimatum. As such, just speccing out a possibly route is helping. Also, I plan to help with the existing benchmarks. After the second round, I pointed them out to the author of my favorite framework in the hopes he would have time to put together something for the benchmark, otherwise I was going to in the next week or two when I had time. I still plan to. Oh, and that framework author's answer? That these benchmarks are laughable because all they measure is performance, and there's a clear performance to convenience trade-off shown in the results, and that of course there's a performance hit when the framework handles most the work for you. I have to say I agree. Sure, there's possibly some that are clear winners giving good performance with lots of conveniences for common operations, but is there any way to tell as much from the data presented so far?

crypto5 14y ago

Do we have any chance to look at round 3 results?

bhauer 14y ago
Of course! We are starting the tests next Monday and you can expect them to be complete some time around the middle of next week.

ckluis 14y ago

@bhauer - this may be the best marketing I have ever seen a company do in the tech space (and by marketing I mean marketing to developers). This is a recruiting goldmine. Sheer genius.

voidlogic 14y ago

What about tracking memory usage? Peak, average, etc. Previously you tested EC2 vs Local HW. What about adding local KVM virtual machine as well? I also think a graph showing latency as a function of concurrency would be very interesting.

bhauer 14y ago
Thanks for the ideas, voidlogic. We do want to capture server statistics and have that as an issue in Github [1]. I am particularly interested in capturing CPU and I/O utilization because in spot checks, we've observed some frameworks do not fully saturate the 8 HT cores on our i7 hardware, suggesting lock or resource contention. As for a variety of other hardware and VM environments, the data would be interesting. Related: we plan to migrate the charts and tables to a stand-alone page. Right now, the blog entries are hard-coded to fetch two specific results.json files for rendering the charts/tables. But when we build a stand-alone page, I would like to enhance the script to allow selection of one or two results.json files from a menu for comparing side-by-side. And to your point, the community could then contribute their own results files. Imagine being able to compare EC2 large vs xlarge or vs Xeon E5s or ... Right now, as you noticed, the latency is only displayed at 256 concurrency. I'll make a note to myself to include a chart for latency versus concurrency when we move to a stand-alone page [2]. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/149 https://github.com/TechEmpower/FrameworkBenchmarks/issues/14...