3 ms·

The logic you have previously posted on HN for these benchmarks is that they measure the minimum overhead available on the platform, so that you cannot get fas

by pekk 14y ago

The logic you have previously posted on HN for these benchmarks is that they measure the minimum overhead available on the platform, so that you cannot get faster than the benchmarked numbers. If a framework is too slow, the framework-chooser can exclude it from consideration for because the resulting project just can't be any faster than the framework benchmark. Sounds reasonable. Except now it is clear that you are refusing optimizations for some frameworks due to a vague, aesthetic judgement of 'stripped'. Which now means that you actually aren't measuring the minimum framework overhead. You are measuring the overhead of the defaults, or the overhead of not taking optimization seriously, with large amounts of performance left on the table. Worse, selectively applying optimizations means you are comparing one framework's defaults to another framework's minimum overhead. And since you have abandoned minimum overhead, it now makes very little sense about why we are measuring performance independent of normal first-resort tactics like caching (who is running Cake without caching?) If you were going to do that, you should have benchmarked defaults right down the line and allowed a full, normal range of simple deployment optimizations. Instead we have selective optimization and totally unrealistic deploys, so it really indicates very little.

bhauer 14y ago
Hi Pekk, I'm not sure where you get the impression that we are refusing tuned tests (what we call "Stripped" tests). We have accepted two of those and would accept further tests of that nature. An implementation of course still needs to work and meet the obligations of the test scenario. For example, each row must be fetched from the database individually and the response must be serialized JSON. We did "reject" one test that fetched all 20 rows using a WHERE IN clause, but that implementation was quickly reconfigured by the submitter to match our specification. We are expressly not including reverse proxy caches in these tests. We're not benchmarking the performance of the nginx proxy cache, Apache HTTPD's proxy cache, Varnish, or anything similar. You can find such benchmarks elsewhere. We are benchmarking the performance of the application framework for requests that do reach the application server. The tests are intended to be a viable minimum stand-in for application functionality in order to fulfill requests that, for whatever reason, reach your application server. If the scenario is difficult to conceive, imagine your site cannot leverage a proxy cache because every request is providing private user information. To be clear: none of the frameworks are being tested with a front-end cache. Also presently, none of the tests use a back-end cache either, but future tests will include tests of back-end in-memory and near-memory caches.
- apkdn 14y ago
  I think quite a few of these frameworks were tuned for this benchmark but it is not marked as stripped. For example, Yesod has client session and logging disabled. I'm also sure that quite a few frameworks have logging disabled. Does that not count as "stripped" since it deviates from the norm for deployment?
  - bhauer 14y ago
    Hi Apkdn, These are very good points you bring up and I will need to address them in the site's FAQ in addition to this response. I would appreciate any follow-ups as I am open to revising the opinions I include below. First, if there are any specific examples of frameworks that have been mis-characterized, I would appreciate that we address each individually as a Github issue. For example, I will create an issue to discuss the Yesod test and its session configuration [1]. Here is our basic thinking on sessions. None of the current test types exercise sessions, but if the test types were changed to make use of sessions, session functionality should remain available within the framework. If the a particular test implementation/configuration has gone out of its way to remove support for sessions from the framework, we consider that Stripped. If session functionality remains available but simply isn't being exercised because the test types we've created to-date don't use sessions, then at least with respect to sessions, that is Realistic. Logging is an important point that we need to address. We intentionally disabled logging in all of the tests we created and will need to be careful to review the configuration of community-contributed tests to do the same. You're correct, disabling logging is not consistent with the production-class goal. So, why did we opt to disable logging? A few reasons: * We didn't want to deal with cleaning up old log files in the test scripts. * We didn't want to deal with normalizing the logging granularity across frameworks. (Or deal with not doing so.) * In spot checks, we didn't observe much performance differential when logging is enabled. We're not unmovable on logging, however, and if there is sufficiency community demand, we would switch to leaving logging [2]. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/252 https://github.com/TechEmpower/FrameworkBenchmarks/issues/25... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/253 https://github.com/TechEmpower/FrameworkBenchmarks/issues/25...

by pekk 14y ago

bhauer 14y ago

Hi Pekk, I'm not sure where you get the impression that we are refusing tuned tests (what we call "Stripped" tests). We have accepted two of those and would accept further tests of that nature. An implementation of course still needs to work and meet the obligations of the test scenario. For example, each row must be fetched from the database individually and the response must be serialized JSON. We did "reject" one test that fetched all 20 rows using a WHERE IN clause, but that implementation was quickly reconfigured by the submitter to match our specification. We are expressly not including reverse proxy caches in these tests. We're not benchmarking the performance of the nginx proxy cache, Apache HTTPD's proxy cache, Varnish, or anything similar. You can find such benchmarks elsewhere. We are benchmarking the performance of the application framework for requests that do reach the application server. The tests are intended to be a viable minimum stand-in for application functionality in order to fulfill requests that, for whatever reason, reach your application server. If the scenario is difficult to conceive, imagine your site cannot leverage a proxy cache because every request is providing private user information. To be clear: none of the frameworks are being tested with a front-end cache. Also presently, none of the tests use a back-end cache either, but future tests will include tests of back-end in-memory and near-memory caches.

apkdn 14y ago
I think quite a few of these frameworks were tuned for this benchmark but it is not marked as stripped. For example, Yesod has client session and logging disabled. I'm also sure that quite a few frameworks have logging disabled. Does that not count as "stripped" since it deviates from the norm for deployment?
- bhauer 14y ago
  Hi Apkdn, These are very good points you bring up and I will need to address them in the site's FAQ in addition to this response. I would appreciate any follow-ups as I am open to revising the opinions I include below. First, if there are any specific examples of frameworks that have been mis-characterized, I would appreciate that we address each individually as a Github issue. For example, I will create an issue to discuss the Yesod test and its session configuration [1]. Here is our basic thinking on sessions. None of the current test types exercise sessions, but if the test types were changed to make use of sessions, session functionality should remain available within the framework. If the a particular test implementation/configuration has gone out of its way to remove support for sessions from the framework, we consider that Stripped. If session functionality remains available but simply isn't being exercised because the test types we've created to-date don't use sessions, then at least with respect to sessions, that is Realistic. Logging is an important point that we need to address. We intentionally disabled logging in all of the tests we created and will need to be careful to review the configuration of community-contributed tests to do the same. You're correct, disabling logging is not consistent with the production-class goal. So, why did we opt to disable logging? A few reasons: * We didn't want to deal with cleaning up old log files in the test scripts. * We didn't want to deal with normalizing the logging granularity across frameworks. (Or deal with not doing so.) * In spot checks, we didn't observe much performance differential when logging is enabled. We're not unmovable on logging, however, and if there is sufficiency community demand, we would switch to leaving logging [2]. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/252 https://github.com/TechEmpower/FrameworkBenchmarks/issues/25... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/253 https://github.com/TechEmpower/FrameworkBenchmarks/issues/25...