6 ms·

I'm having a hard time making conclusions based off of this data, except that Python/Ruby/JavaScript are slow, JVM is fast, and the rest are in between... It's

I'm having a hard time making conclusions based off of this data, except that Python/Ruby/JavaScript are slow, JVM is fast, and the rest are in between... It's odd how Haskell (Yesod) and C (Onion) are both slower than the JVM frameworks. Is it just a maturity issue? I know Yesod is very new relative to some of these JVM frameworks. Also, minor typo: the link to Erlang from the Environment tab is not the right URL. You probably meant to reference http://www.erlang.org/ http://www.erlang.org/, not http://www.erlang.com/ http://www.erlang.com/

bhauer 13y ago
Thanks pilgrim689. I'll get that fixed up.
mdasen 13y ago
Because the benchmarks are quite narrow, sometimes you get something that has simply been optimized or isn't doing the same work across platforms. For example, the last time I checked their php-raw code, it was executing a database query and appending the raw results to an array. Then, it took that array and passed it to json_encode which I'm pretty sure is implemented in C (PHP's built-in functions are in C, yes?). Anyway, comparing that to something like Rails' ActiveRecord, the work is quite different. With Rails, it's going to go through and call the accessor methods on the objects to get the data because we often override the default data stored in the database for the representation we want on an object level. So, they aren't doing the same thing. Rails at least used to have a way to get back an array of hashes rather than getting the AR objects, but I haven't been able to find it recently. That could be an interesting comparison. So, in that case, it just isn't the same functionality. Similarly, it's hard to know exactly how they're setting things up. Django will perform significantly better if you set up something like pgBouncer for connection pooling. Rails includes connection pooling, IIRC. Without knowing something like that, it's hard to gauge whether it reflects real-world usage (I mean, I'd argue that defaults matter, but if you're looking to scale Django, this isn't an onerous addition). Similarly, with process-based concurrency, it's important to get settings that maximize CPU usage while not overloading the RAM. Since they don't talk about memory or CPU usage, it's hard to figure out whether they've dealt with this. It might be in their repository, but I haven't been able to go through it yet. Heck, the multiple-query test in go seems really odd. I mean, coming in last place, serving less than half the requests of Rails? Similarly, Play 1 hits a nice 28% on EC2 and then can't serve a single request on the dedicated hardware. Plus, the tests really stress JSON performance and don't use things like a framework's templating system. One doesn't need to argue that JSON performance is useful, but so is template performance and that just isn't being tested. So, yeah, it would be hard to make conclusions off of this data except possibly that benchmarking is hard. None of this comment is meant as a dig against the people doing these benchmarks. They're improving them, they're making their code open, they're doing the kind of stuff that allows good, open questioning on their results. But I think it's still early to consider these to be really meaningful. I think a test that combined a bunch of different types of requests with database access, JSON, templates, etc. would be interesting and it seems like they might go there as they have more time.
- deleted 13y ago
  [deleted]
- bhauer 13y ago
  Hi mdasen. Thank you for the feedback. We really appreciate these kinds of thoughtful contributions. To address some of your concerns: The php-raw test, as with all tests that have the "raw" suffix is not using an ORM. The servlet-raw test uses raw JDBC. The tests without the "raw" suffix are assumed to be using an ORM or something ORM-like. There is a separate PHP test (named just "php") that uses PHP ActiveRecord. To be clear: without the "raw" suffix, we expect the test to be exercising the framework's preferred ORM or something analogous to an ORM. For example, several but not all of the Java tests are using Hibernate. I am of the opinion that it's of great value to include the raw tests alongside the ORM tests for comparison. Later versions of the results view will allow filtering of the results (e.g., filtering out the "raw" tests or filtering out all Java frameworks) [1] [2]. Django is being used with MySQL and does not (presently) have a connection pool [3]. We'd gladly accept a pull request that adds a MySQL connection pool. Separately, we aim to eventually add Postgres tests. As for frameworks that use process-based concurrency, we have attempted to configure each according to the capacity of the hardware. For example, for a given process-concurrency framework on EC2 large, with two virtual processors, we may use two workers; on i7 with eight HT cores, we may use eight workers. You can review the configuration details in the repository and submit pull requests if they are wrong. We aim to add server-side statistics capturing in a later round [4]. However, for the time being, I can say that anecdotal observations show that some frameworks do not saturate all CPU cores, but we have not observed any running into out-of-memory situations. Regarding the curious Play1 and Go database test results, see elsewhere in this thread [5]. In both cases, the communities have offered to help address the problems the tests are running into. The Play community has already fixed the Play1 database test and we expect it to perform in line with Play1 + Siena in Round 4. I agree that testing more components is desirable. The next test will include some minor work with collections and server-side templates [6]. We have started implementing Test 4 on a few frameworks and have been very pleasantly surprised to see the community has already started submitting implementations in their favorites as well. Thanks again for your detailed thoughts and we look forward to continuously improving this project for as long as we have the time to do so. Please feel welcome to join in the conversation on any of the Github issues or create new ones. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/152 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/126 https://github.com/TechEmpower/FrameworkBenchmarks/issues/12... [3] http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation [4] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10... [5] https://news.ycombinator.com/item?id=5590132 https://news.ycombinator.com/item?id=5590132 [6] https://github.com/TechEmpower/FrameworkBenchmarks/issues/134 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...
orclev 13y ago
On the topic of the JVM frameworks, you have to understand that the JVM is a very finely tuned machine at this point, it has something like 10+ years of performance tuning by a large portion of the entire industry behind it. Yesod just recently cracked the 1.0 release and has been going through massive changes in the last year or so, the fact that it performs as well as it does is a huge testament not only to Haskell as a language, but also to the work of Michael Snoyman and the others who's work he incorporated into Yesod. That said, there are still a lot of rough patches with Yesod, and I'm sure there are lots of opportunities to improve performance in various small ways. The main Haskell compiler (ghc) is constantly improving and many of the changes allow for smarter automatic optimizations to be applied by both the compiler and the runtime. With continued refinement I feel it's definitely possible for Yesod to match, and even exceed the performance of any of the JVM based languages, but as I said, there are a lot of performance optimizations baked into the JVM and it's probably going to take a while before Yesod (and WAI which it's built on) have similar levels of optimization in them.

bhauer 13y ago

Thanks pilgrim689. I'll get that fixed up.

mdasen 13y ago

Because the benchmarks are quite narrow, sometimes you get something that has simply been optimized or isn't doing the same work across platforms. For example, the last time I checked their php-raw code, it was executing a database query and appending the raw results to an array. Then, it took that array and passed it to json_encode which I'm pretty sure is implemented in C (PHP's built-in functions are in C, yes?). Anyway, comparing that to something like Rails' ActiveRecord, the work is quite different. With Rails, it's going to go through and call the accessor methods on the objects to get the data because we often override the default data stored in the database for the representation we want on an object level. So, they aren't doing the same thing. Rails at least used to have a way to get back an array of hashes rather than getting the AR objects, but I haven't been able to find it recently. That could be an interesting comparison. So, in that case, it just isn't the same functionality. Similarly, it's hard to know exactly how they're setting things up. Django will perform significantly better if you set up something like pgBouncer for connection pooling. Rails includes connection pooling, IIRC. Without knowing something like that, it's hard to gauge whether it reflects real-world usage (I mean, I'd argue that defaults matter, but if you're looking to scale Django, this isn't an onerous addition). Similarly, with process-based concurrency, it's important to get settings that maximize CPU usage while not overloading the RAM. Since they don't talk about memory or CPU usage, it's hard to figure out whether they've dealt with this. It might be in their repository, but I haven't been able to go through it yet. Heck, the multiple-query test in go seems really odd. I mean, coming in last place, serving less than half the requests of Rails? Similarly, Play 1 hits a nice 28% on EC2 and then can't serve a single request on the dedicated hardware. Plus, the tests really stress JSON performance and don't use things like a framework's templating system. One doesn't need to argue that JSON performance is useful, but so is template performance and that just isn't being tested. So, yeah, it would be hard to make conclusions off of this data except possibly that benchmarking is hard. None of this comment is meant as a dig against the people doing these benchmarks. They're improving them, they're making their code open, they're doing the kind of stuff that allows good, open questioning on their results. But I think it's still early to consider these to be really meaningful. I think a test that combined a bunch of different types of requests with database access, JSON, templates, etc. would be interesting and it seems like they might go there as they have more time.

deleted 13y ago
[deleted]
bhauer 13y ago
Hi mdasen. Thank you for the feedback. We really appreciate these kinds of thoughtful contributions. To address some of your concerns: The php-raw test, as with all tests that have the "raw" suffix is not using an ORM. The servlet-raw test uses raw JDBC. The tests without the "raw" suffix are assumed to be using an ORM or something ORM-like. There is a separate PHP test (named just "php") that uses PHP ActiveRecord. To be clear: without the "raw" suffix, we expect the test to be exercising the framework's preferred ORM or something analogous to an ORM. For example, several but not all of the Java tests are using Hibernate. I am of the opinion that it's of great value to include the raw tests alongside the ORM tests for comparison. Later versions of the results view will allow filtering of the results (e.g., filtering out the "raw" tests or filtering out all Java frameworks) [1] [2]. Django is being used with MySQL and does not (presently) have a connection pool [3]. We'd gladly accept a pull request that adds a MySQL connection pool. Separately, we aim to eventually add Postgres tests. As for frameworks that use process-based concurrency, we have attempted to configure each according to the capacity of the hardware. For example, for a given process-concurrency framework on EC2 large, with two virtual processors, we may use two workers; on i7 with eight HT cores, we may use eight workers. You can review the configuration details in the repository and submit pull requests if they are wrong. We aim to add server-side statistics capturing in a later round [4]. However, for the time being, I can say that anecdotal observations show that some frameworks do not saturate all CPU cores, but we have not observed any running into out-of-memory situations. Regarding the curious Play1 and Go database test results, see elsewhere in this thread [5]. In both cases, the communities have offered to help address the problems the tests are running into. The Play community has already fixed the Play1 database test and we expect it to perform in line with Play1 + Siena in Round 4. I agree that testing more components is desirable. The next test will include some minor work with collections and server-side templates [6]. We have started implementing Test 4 on a few frameworks and have been very pleasantly surprised to see the community has already started submitting implementations in their favorites as well. Thanks again for your detailed thoughts and we look forward to continuously improving this project for as long as we have the time to do so. Please feel welcome to join in the conversation on any of the Github issues or create new ones. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/152 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15... [2] https://github.com/TechEmpower/FrameworkBenchmarks/issues/126 https://github.com/TechEmpower/FrameworkBenchmarks/issues/12... [3] http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation [4] https://github.com/TechEmpower/FrameworkBenchmarks/issues/108 https://github.com/TechEmpower/FrameworkBenchmarks/issues/10... [5] https://news.ycombinator.com/item?id=5590132 https://news.ycombinator.com/item?id=5590132 [6] https://github.com/TechEmpower/FrameworkBenchmarks/issues/134 https://github.com/TechEmpower/FrameworkBenchmarks/issues/13...

orclev 13y ago

On the topic of the JVM frameworks, you have to understand that the JVM is a very finely tuned machine at this point, it has something like 10+ years of performance tuning by a large portion of the entire industry behind it. Yesod just recently cracked the 1.0 release and has been going through massive changes in the last year or so, the fact that it performs as well as it does is a huge testament not only to Haskell as a language, but also to the work of Michael Snoyman and the others who's work he incorporated into Yesod. That said, there are still a lot of rough patches with Yesod, and I'm sure there are lots of opportunities to improve performance in various small ways. The main Haskell compiler (ghc) is constantly improving and many of the changes allow for smarter automatic optimizations to be applied by both the compiler and the runtime. With continued refinement I feel it's definitely possible for Yesod to match, and even exceed the performance of any of the JVM based languages, but as I said, there are a lot of performance optimizations baked into the JVM and it's probably going to take a while before Yesod (and WAI which it's built on) have similar levels of optimization in them.