5 ms·

This is the most recent update to our ongoing project measuring the performance of web application platforms and frameworks. In this round we've received sever

by bhauer 14y ago

This is the most recent update to our ongoing project measuring the performance of web application platforms and frameworks. In this round we've received several more community-contributed tests in Perl, PHP, Python, Java, and JavaScript. Go is a comeback champion thanks to changes made by Brad Fitzpatrick [1] and others in the Go community. A new "Fortunes" test was also added (implemented in 17 of the frameworks) that exercises server-side templates and collections. With 57 total frameworks being tested, we have implemented some filtering to allow you to narrow your view to only those you care about. As always, we'd really like to hear your questions, suggestions, and criticisms. And we hope you enjoy this latest round of data. [1] https://code.google.com/p/go/source/detail?r=45c12efb46 https://code.google.com/p/go/source/detail?r=45c12efb46

nopal 14y ago
Any reason you didn't test ASP.NET MVC or ASP Web Forms?
- JulienSchmidt 14y ago
  Maybe because the server is not running Windows? Edit: yeah right Mono and w/e
  - cmircea 14y ago
    Would it be too much to ask to also run a Windows VM? Or use Mono?
    - platz 14y ago
      http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation "As with the previous question, we'd love to. We have heard tentative word from a reader/contributor that a pull request may be incoming soon that will include several .NET frameworks on Mono, which we assume will be as easy to include as any other pull request. One challenge we face is that the test infrastructure we've built assumes a Linux deployment that we can automate using ssh and Python. To do a proper .NET test on Windows Servers, we will need to work on adapting that platform to automate Windows Servers as well. Community assistance on this would be greatly appreciated."
      - iknight 14y ago
        Can you share who the Mono contributor is?
        bhauer 14y ago
        Sure. We now have two pending issues, one of which is a pull request. If you can weigh in and perhaps contribute some review time to the PR in particular, that would be very appreciated! https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
    - voidlogic 14y ago
      Linux amd64 is the dominate internet serving platform; supporting a non-free OS for the benefit of a single test entry seems dumb. I'm all for a Mono entry however.
      - diroussel 14y ago
        I don't think it seems dumb. It would be a very useful comparison between platforms. Stackoverlfow runs on .Net and Windows servers and they say it's very performant. So why not compare with other frameworks on the same hardware.
        bhauer 14y ago
        Voidlogic is right that we are waiting to get a pull request that will include some .Net frameworks [1]. If you can help, it would be greatly appreciated. We do want to include .Net. We will test on Mono to start. We also want to test on .Net's native Windows platform. But we need to work on the testing platform we've built in order to automate a Windows server in the same way we presently automate a Linux server. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
        egeozcan 14y ago
        Nancy[1] comes to mind. [1]: http://nancyfx.org/ http://nancyfx.org/
        bhauer 14y ago
        I like the looks of that! Could I convince you to put together a test as a pull request? :)
        egeozcan 14y ago
        I can't promise to deliver but I'll sure look into it this weekend =)
- AndrewGaspar 14y ago
  I had the same thought. ASP.NET MVC and Web Forms are both very popular frameworks, and they're free to develop for. Might be harder to set up on a non-Windows machine, which could be why they were not tested.
  - voidlogic 14y ago
    They were not tested because no one has submitted a pull request for them.
- pfalls 14y ago
  We would like to have a .NET test running on mono. We were hoping to get a pull request for round 4, but unfortunately we have yet to receive one.
- apalmer 14y ago
  ASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other test implementation, the results really wont be comparable in any fair way. Microsoft Issue #1, dont know if it still holds now a days but in past official EULA for .Net prohibited publishing benchmark results. PERIOD. I am a .net developer and as much as I like ASP.Net I dont think the effort of adding a .Net implementation really would pay off.
  - bsaul 14y ago
    I completely understand your point, but I think it's fair to say that most .Net code will run on Windows server, and that pretty much everything else will run on some kind of linux flavor. Just like you have a "keep the default framework setting" approach to help compare very different frameworks because that's how the majority of people will use them, you may very well assume that comparing frameworks on their preferred OS is fair enough. I know that i wouldn't mind switching to a windows+.Net environment if it proved to be much much faster than what i'm using right now.
    - DennisP 14y ago
      Exactly, and if there is such a performance boost, having the numbers would help you figure out whether hardware savings would be worth the licensing cost.
  - voidlogic 14y ago
    Care to backup your Mono claims with evidence? I have a friend who went to a Mono talk at MS MIX where a Mono developer was speaking. The Mono developer said that while Mono is a little slower than .NET (and he was talking a couple percent) Mono often ends up being faster on the same hardware because the Linux system calls the runtime uses are faster. There were a few ROFLs in the audience. I also agree with you that investing in a .NET (Windows) test isn't good bang for the buck here.
diroussel 14y ago
The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How not to measure latency" [1] for more info. Also be sure you are not falling into the "Coordinated Omission" trap where you end up measureing the latency wrong. [1] http://www.infoq.com/presentations/latency-pitfalls http://www.infoq.com/presentations/latency-pitfalls
- bhauer 14y ago
  Hello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide that in some form. And perhaps the author of Wrk could add that in time. However, for the time being, I consider the matter somewhat academic. Not to be dismissive--I value your opinion--but I don't believe that would measurably impact my assessment of each framework's performance. Though, it would be fascinating to be able to validate my suspicion that Onion, being written in C, does not suffer even the tiny garbage collection pauses of the Java frameworks. [1] https://github.com/wg/wrk https://github.com/wg/wrk
  - diroussel 14y ago
    Ok, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31 https://github.com/wg/wrk/issues/31 Perhaps you could upvote or something? Thanks for all the great work in these benchmarks. A useful resource.
- tlarkworthy 14y ago
  "average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.
laumars 14y ago
There's two main mysql drivers for Go mymysql and go-mysql-driver. I've found the concurrency performance of the former to be abysmal when doing my benchmarking. Then the moment I switched to the latter, Go's performance went through the roof.

by bhauer 14y ago

nopal 14y ago

Any reason you didn't test ASP.NET MVC or ASP Web Forms?

JulienSchmidt 14y ago
Maybe because the server is not running Windows? Edit: yeah right Mono and w/e
- cmircea 14y ago
  Would it be too much to ask to also run a Windows VM? Or use Mono?
  - platz 14y ago
    http://www.techempower.com/benchmarks/#section=motivation http://www.techempower.com/benchmarks/#section=motivation "As with the previous question, we'd love to. We have heard tentative word from a reader/contributor that a pull request may be incoming soon that will include several .NET frameworks on Mono, which we assume will be as easy to include as any other pull request. One challenge we face is that the test infrastructure we've built assumes a Linux deployment that we can automate using ssh and Python. To do a proper .NET test on Windows Servers, we will need to work on adapting that platform to automate Windows Servers as well. Community assistance on this would be greatly appreciated."
    - iknight 14y ago
      Can you share who the Mono contributor is?
      - bhauer 14y ago
        Sure. We now have two pending issues, one of which is a pull request. If you can weigh in and perhaps contribute some review time to the PR in particular, that would be very appreciated! https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/pull/239 https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
  - voidlogic 14y ago
    Linux amd64 is the dominate internet serving platform; supporting a non-free OS for the benefit of a single test entry seems dumb. I'm all for a Mono entry however.
    - diroussel 14y ago
      I don't think it seems dumb. It would be a very useful comparison between platforms. Stackoverlfow runs on .Net and Windows servers and they say it's very performant. So why not compare with other frameworks on the same hardware.
      - bhauer 14y ago
        Voidlogic is right that we are waiting to get a pull request that will include some .Net frameworks [1]. If you can help, it would be greatly appreciated. We do want to include .Net. We will test on Mono to start. We also want to test on .Net's native Windows platform. But we need to work on the testing platform we've built in order to automate a Windows server in the same way we presently automate a Linux server. [1] https://github.com/TechEmpower/FrameworkBenchmarks/issues/158 https://github.com/TechEmpower/FrameworkBenchmarks/issues/15...
        egeozcan 14y ago
        Nancy[1] comes to mind. [1]: http://nancyfx.org/ http://nancyfx.org/
        bhauer 14y ago
        I like the looks of that! Could I convince you to put together a test as a pull request? :)
        egeozcan 14y ago
        I can't promise to deliver but I'll sure look into it this weekend =)
AndrewGaspar 14y ago
I had the same thought. ASP.NET MVC and Web Forms are both very popular frameworks, and they're free to develop for. Might be harder to set up on a non-Windows machine, which could be why they were not tested.
- voidlogic 14y ago
  They were not tested because no one has submitted a pull request for them.
pfalls 14y ago
We would like to have a .NET test running on mono. We were hoping to get a pull request for round 4, but unfortunately we have yet to receive one.
apalmer 14y ago
ASP.Net kind of put themselves out of the benchmark game here: Mono Issue #1, since the vast vast majority of ASP.Net websites run on windows a Mono performance test even if accurate is going to be of dubious value. Mono Issue #2, since Mono is nowhere near as polished as the Microsoft .Net implementation the numbers wont really be meaningful. Windows Issue #1, if you do the test on a different OS than every other test implementation, the results really wont be comparable in any fair way. Microsoft Issue #1, dont know if it still holds now a days but in past official EULA for .Net prohibited publishing benchmark results. PERIOD. I am a .net developer and as much as I like ASP.Net I dont think the effort of adding a .Net implementation really would pay off.
- bsaul 14y ago
  I completely understand your point, but I think it's fair to say that most .Net code will run on Windows server, and that pretty much everything else will run on some kind of linux flavor. Just like you have a "keep the default framework setting" approach to help compare very different frameworks because that's how the majority of people will use them, you may very well assume that comparing frameworks on their preferred OS is fair enough. I know that i wouldn't mind switching to a windows+.Net environment if it proved to be much much faster than what i'm using right now.
  - DennisP 14y ago
    Exactly, and if there is such a performance boost, having the numbers would help you figure out whether hardware savings would be worth the licensing cost.
- voidlogic 14y ago
  Care to backup your Mono claims with evidence? I have a friend who went to a Mono talk at MS MIX where a Mono developer was speaking. The Mono developer said that while Mono is a little slower than .NET (and he was talking a couple percent) Mono often ends up being faster on the same hardware because the Linux system calls the runtime uses are faster. There were a few ROFLs in the audience. I also agree with you that investing in a .NET (Windows) test isn't good bang for the buck here.

diroussel 14y ago

The requests per second is importanct, but some frameworks seem to get high average throughput but at the expense of a few slow requests. Also when measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely. Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms. See Gil Tene's talk "How not to measure latency" [1] for more info. Also be sure you are not falling into the "Coordinated Omission" trap where you end up measureing the latency wrong. [1] http://www.infoq.com/presentations/latency-pitfalls http://www.infoq.com/presentations/latency-pitfalls

bhauer 14y ago
Hello diroussel, Thanks for the feedback! We started the project with WeigHTTP, then starting with Round 2 we switched to Wrk [1] at the advice of other readers. Wrk provides latency measurements consisting of average, standard deviation, and maximum. See the earlier conversation about standard deviation here: https://news.ycombinator.com/item?id=5455972 https://news.ycombinator.com/item?id=5455972 If we had distribution data available, we would aim to provide that in some form. And perhaps the author of Wrk could add that in time. However, for the time being, I consider the matter somewhat academic. Not to be dismissive--I value your opinion--but I don't believe that would measurably impact my assessment of each framework's performance. Though, it would be fascinating to be able to validate my suspicion that Onion, being written in C, does not suffer even the tiny garbage collection pauses of the Java frameworks. [1] https://github.com/wg/wrk https://github.com/wg/wrk
- diroussel 14y ago
  Ok, I've raise an enhancement for wrk https://github.com/wg/wrk/issues/31 https://github.com/wg/wrk/issues/31 Perhaps you could upvote or something? Thanks for all the great work in these benchmarks. A useful resource.
tlarkworthy 14y ago
"average and std dev are only revelent if the distribution is Gaussian in distrition" technically not true. Knowledge of the second order moment (variance) lets you uniquely identify other distributions like Poisson, or uniform. Knowledge of even higher order moments lets you fit more complicated statistical models. Low variance is good, regardless of underlying distribution.

laumars 14y ago

There's two main mysql drivers for Go mymysql and go-mysql-driver. I've found the concurrency performance of the former to be abysmal when doing my benchmarking. Then the moment I switched to the latter, Go's performance went through the roof.