Hi mdasen. Thank you for the feedback. We really appreciate these kinds of thoughtful contributions. To address some of your concerns: The php-raw test, as

Hi mdasen. Thank you for the feedback. We really appreciate these kinds of thoughtful contributions. To address some of your concerns: The php-raw test, as with all tests that have the "raw" suffix is not using an ORM. The servlet-raw test uses raw JDBC. The tests without the "raw" suffix are assumed to be using an ORM or something ORM-like. There is a separate PHP test (named just "php") that uses PHP ActiveRecord. To be clear: without the "raw" suffix, we expect the test to be exercising the framework's preferred ORM or something analogous to an ORM. For example, several but not all of the Java tests are using Hibernate. I am of the opinion that it's of great value to include the raw tests alongside the ORM tests for comparison. Later versions of the results view will allow filtering of the results (e.g., filtering out the "raw" tests or filtering out all Java frameworks) [1] [2]. Django is being used with MySQL and does not (presently) have a connection pool [3]. We'd gladly accept a pull request that adds a MySQL connection pool. Separately, we aim to eventually add Postgres tests. As for frameworks that use process-based concurrency, we have attempted to configure each according to the capacity of the hardware. For example, for a given process-concurrency framework on EC2 large, with two virtual processors, we may use two workers; on i7 with eight HT cores, we may use eight workers. You can review the configuration details in the repository and submit pull requests if they are wrong. We aim to add server-side statistics capturing in a later round [4]. However, for the time being, I can say that anecdotal observations show that some frameworks do not saturate all CPU cores, but we have not observed any running into out-of-memory situations. Regarding the curious Play1 and Go database test results, see elsewhere in this thread [5]. In both cases, the communities have offered to help address the problems the tests are running into. The Play community has already fixed the Play1 database test and we expect it to perform in line with Play1 + Siena in Round 4. I agree that testing more components is desirable. The next test will include some minor work with collections and server-side templates [6]. We have started implementing Test 4 on a few frameworks and have been very pleasantly surprised to see the community has already started submitting implementations in their favorites as well. Thanks again for your detailed thoughts and we look forward to continuously improving this project for as long as we have the time to do so. Please feel welcome to join in the conversation on any of the Github issues or create new ones. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/152 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/126 https://github.com/TechEmpower/FrameworkBenchmarks/issues/12... [3] http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation [4] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10... [5] https://news.ycombinator.com/item?id=5590132 https://news.ycombinator.com/item?id=5590132 [6] https://github.com/TechEmpower/FrameworkBenchmarks/issues/134 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...