2 ms·

The community has contributed even more pull requests since Round 2 of our web application framework benchmarks. We're planning to start Round 3 tests on Monda

by bhauer 14y ago

The community has contributed even more pull requests since Round 2 of our web application framework benchmarks. We're planning to start Round 3 tests on Monday 4/15 using a new build of Wrk that allows time-limited tests (rather than request-count limited) so that all frameworks are tested for a uniform amount of time. In the meantime, this Github issue request is seeking thoughts you may have concerning additional simple tests that we can introduce in Round 4 and beyond. We want to define tests that continue to exercise typical web application functionality but remain fairly simple to implement on a ever-widening field of frameworks. If you have thoughts, please add them here or at Github. Thanks!

wheaties 14y ago
I have got to do a fork/pull before then in my infinite spare time but a quick question, why only ORM? Excuse my ignorance but there are almost no ORM that I've used which are not materially slower than straight SQL or json, the exception being g Salat on MongoDB. I wonder what not having an ORM would do to some of the poor performance experienced.
- bhauer 14y ago
  We expect that most modern webapps are (still) developed with an ORM, but we're not ORM hard-liners. A few of the tests are run without an ORM and are identified with the suffix "raw."
Terretta 14y ago
Thoughts of incremental web app functionality to test: 1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D. 2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example. (The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.) 3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc. 4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered. 5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache. For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.
- bhauer 14y ago
  This is great input, Terretta. Exactly the kind of thinking I wanted to tap into. Some quick thoughts in response: For #1, the conceptual test included reading and writing in a 1:1 R:W ratio (well, to be more accurate 1:1 R:U). I like the idea of extending this a little bit to include C and D. For the sake of benchmark run-time, I'm looking to restrain the growth of permutations. But on the other hand, I like your idea of a tunable ratio. Something to think about! I like #2 and #3 as well. I'll think about those some more too. I really like the idea of incorporating some bcrypt and session state (#4). We have a few caching tests in mind, but like elsewhere, we'll start out simple and then add complexity. Thanks for the great input! This planning is to have some good long-term ideas in mind.

by bhauer 14y ago

wheaties 14y ago

I have got to do a fork/pull before then in my infinite spare time but a quick question, why only ORM? Excuse my ignorance but there are almost no ORM that I've used which are not materially slower than straight SQL or json, the exception being g Salat on MongoDB. I wonder what not having an ORM would do to some of the poor performance experienced.

bhauer 14y ago
We expect that most modern webapps are (still) developed with an ORM, but we're not ORM hard-liners. A few of the tests are run without an ORM and are identified with the suffix "raw."

Terretta 14y ago

Thoughts of incremental web app functionality to test: 1. Exercising a randomized mix of reading and writing. I think you already said you were planning a CRUD test. Consider a tunable ratio here, something like 10000 R to 100 U to 10 C to 1 D. 2. Exercising synchronous web service (JSONP) calls in two modes: (a) to some web service that is consistently fast and low latency, say, the initial JSON example from this test suite running in servlet mode, and (b) to a web service written in the same framework as the one being tested, again using the initial JSON example. (The idea here is that many frameworks fall on their faces when confronted with latency. This is why synthetic tests are usually so poorly predictive of real world behavior -- people forget that latency causes backlogs and backlogs cause all parts of the stack to misbehave in interesting ways.) 3. Test async ability if the framework has it, with a system call (sleep?) that takes a randomized 0 - 60 seconds to return. Would help understand when a framework is likely to blow up calling out to a credit card processor, doing server side image processing, etc. 4. Exercising authentication (standardize on bcrypt, but only create passwords on 1 in 10K requests), authorization, and session state, if offered. 5. Exercising any built-in support for caching, where 1 in rand(X) requests invalidates the DB query cache, 1 in rand(X) requests invalidates the WS call cache, 1 in rand(X) requests invalidates the long term async system call cache, and 1 in rand(Y) requests blows away the whole cache. For the enterprise legacy integrators, it would also be interesting to test XML as well (in particular, SOAP), anywhere we're testing JSON.

bhauer 14y ago
This is great input, Terretta. Exactly the kind of thinking I wanted to tap into. Some quick thoughts in response: For #1, the conceptual test included reading and writing in a 1:1 R:W ratio (well, to be more accurate 1:1 R:U). I like the idea of extending this a little bit to include C and D. For the sake of benchmark run-time, I'm looking to restrain the growth of permutations. But on the other hand, I like your idea of a tunable ratio. Something to think about! I like #2 and #3 as well. I'll think about those some more too. I really like the idea of incorporating some bcrypt and session state (#4). We have a few caching tests in mind, but like elsewhere, we'll start out simple and then add complexity. Thanks for the great input! This planning is to have some good long-term ideas in mind.