5 ms·

I love what you guys are doing. This is by far the most comprehensive (in terms of number of frameworks) comparison of web frameworks. I also am a big fan of th

I love what you guys are doing. This is by far the most comprehensive (in terms of number of frameworks) comparison of web frameworks. I also am a big fan of the new filtering metadata. However, I'm starting to think that all of the advocates of various frameworks are now conspiring independently to make this comparison meaningless...any framework (except Cake for some reason) can be superoptimized towards a small set of tasks. If you do another round, could you increase the number of different tasks? Some examples could be: 1) Mixed bag of queries of various complexity 2) Static file serving 3) A few computation/memory-intense benchmarks (such as those in the Language Benchmarks Game) 4) Templating

pfalls 14y ago
Thanks for the kind words. We're very interested in adding additional tests, this round even includes a new test dubbed "Fortunes" which does in fact do server-side templating. We have an open github issue[1] asking for the community's input for just this sort of thing, and we'd love to have your feedback included. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...
- deleted 14y ago
  [deleted]
bhauer 14y ago
Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framework with realistic production-style implementation of the tests. No doubt there is temptation to trim out unnecessary functionality and focus on the benchmark's particular behavior. We have attempted to identify any such tests that remove framework features to target the benchmark as "Stripped" and those can now be filtered out from the list. In other words, our aim is that the implementation of each framework's test is idiomatic to that framework and platform. And if that's not the case for a test, we want to correct it. Your concern could be clarified by pointing out that framework authors may be tuning up their JSON serialization, database connection pools, and template processing in order to improve their position on these charts. And, to be clear, I have already seen evidence of that in my interaction with framework authors. To that concern, however, I would say: That is awesome. I want those features to be fast.
- jakejake 14y ago
  I would like to pile my thanks onto this list as well. I'm the author of Phreeze and I can say that I'm grateful that fairness is being encouraged. There is certainly glory in ranking well on any benchmark and I have to admit, as I was implementing the tests in Phreeze, I saw many opportunities to "cheat." For example, skipping the framework routing, not using the "proper" way to communicate between the layers, etc and substituting things with "raw" code would have potential to skew the result. I feel that would be missing the entire point of a benchmark, so I'm glad that is being considered. I can also say that this benchmark inspired me to take a hard look at class loading and I was able to make some improvements to the framework's efficiency in general. So, in a way, I did some tuning - not for the benchmark, but rather as a result of the benchmark. Thanks to this benchmark all Phreeze users will gain a little performance. I would also like to suggest a test idea. I think the biggest challenge for frameworks comes into play when you have to do table joins. Something like looping through all purchase orders and displaying the customer name from a 2nd table - that would be a very real-world type of test. I think foreign key type of queries are more telling about an ORM than a single table query. Thanks again!
  - camus 14y ago
    Can you share with us the tuning you did with class loading for instance ? thanks for your comment.
    - jakejake 14y ago
      Oh sure, nothing to complicated. Basically I just happened to notice that I loaded several classes that were not always needed. I was able to tune up the framework to load some of them on-demand instead. One example is that the framework loaded an lot of MySQL classes whether or not you do a DB query. So, now I wait to initialize the DB stuff until after you make a call that requires it. Phreeze has always been lazy about opening the DB connection, but now it's even lazier and doesn't even load the classes until you need them! There were some other utility-type classes like XML parsing and such that probably don't even get used much. So that is lazy loaded now too. For a non-DB request I was able to get it down from about 37 files that loaded to around 20. For a DB request I think it's still around 30 files, but I definitely consider that a performance improvement. The benchmark led me to scrutinize what is being loaded so I think it has already improved the framework.
mrgoldenbrown 14y ago
Without advocates "conspiring", the results will favor those frameworks which are most suited to the chosen tasks, as configured by the testers. With conspiring, the results will favor those frameworks with a community who cares about contributing a superoptimized microbenchmark config. In either case, the results might be good discussion fodder, but should be taken with a grain of salt.

pfalls 14y ago

Thanks for the kind words. We're very interested in adding additional tests, this round even includes a new test dubbed "Fortunes" which does in fact do server-side templating. We have an open github issue[1] asking for the community's input for just this sort of thing, and we'd love to have your feedback included. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/133 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...

deleted 14y ago
[deleted]

bhauer 14y ago

Hi saosebastiao, As Pat points out, we definitely look forward to implementing some more computationally-intense request types in the future. This round does include the first server-side template test. We'd like to hear the community's opinions about more tests. That said, I feel most of the frameworks' implementations of the existing tests are not cheating. Our objective in this project is to measure every framework with realistic production-style implementation of the tests. No doubt there is temptation to trim out unnecessary functionality and focus on the benchmark's particular behavior. We have attempted to identify any such tests that remove framework features to target the benchmark as "Stripped" and those can now be filtered out from the list. In other words, our aim is that the implementation of each framework's test is idiomatic to that framework and platform. And if that's not the case for a test, we want to correct it. Your concern could be clarified by pointing out that framework authors may be tuning up their JSON serialization, database connection pools, and template processing in order to improve their position on these charts. And, to be clear, I have already seen evidence of that in my interaction with framework authors. To that concern, however, I would say: That is awesome. I want those features to be fast.

jakejake 14y ago
I would like to pile my thanks onto this list as well. I'm the author of Phreeze and I can say that I'm grateful that fairness is being encouraged. There is certainly glory in ranking well on any benchmark and I have to admit, as I was implementing the tests in Phreeze, I saw many opportunities to "cheat." For example, skipping the framework routing, not using the "proper" way to communicate between the layers, etc and substituting things with "raw" code would have potential to skew the result. I feel that would be missing the entire point of a benchmark, so I'm glad that is being considered. I can also say that this benchmark inspired me to take a hard look at class loading and I was able to make some improvements to the framework's efficiency in general. So, in a way, I did some tuning - not for the benchmark, but rather as a result of the benchmark. Thanks to this benchmark all Phreeze users will gain a little performance. I would also like to suggest a test idea. I think the biggest challenge for frameworks comes into play when you have to do table joins. Something like looping through all purchase orders and displaying the customer name from a 2nd table - that would be a very real-world type of test. I think foreign key type of queries are more telling about an ORM than a single table query. Thanks again!
- camus 14y ago
  Can you share with us the tuning you did with class loading for instance ? thanks for your comment.
  - jakejake 14y ago
    Oh sure, nothing to complicated. Basically I just happened to notice that I loaded several classes that were not always needed. I was able to tune up the framework to load some of them on-demand instead. One example is that the framework loaded an lot of MySQL classes whether or not you do a DB query. So, now I wait to initialize the DB stuff until after you make a call that requires it. Phreeze has always been lazy about opening the DB connection, but now it's even lazier and doesn't even load the classes until you need them! There were some other utility-type classes like XML parsing and such that probably don't even get used much. So that is lazy loaded now too. For a non-DB request I was able to get it down from about 37 files that loaded to around 20. For a DB request I think it's still around 30 files, but I definitely consider that a performance improvement. The benchmark led me to scrutinize what is being loaded so I think it has already improved the framework.

mrgoldenbrown 14y ago

Without advocates "conspiring", the results will favor those frameworks which are most suited to the chosen tasks, as configured by the testers. With conspiring, the results will favor those frameworks with a community who cares about contributing a superoptimized microbenchmark config. In either case, the results might be good discussion fodder, but should be taken with a grain of salt.