5 ms·

I really don't like these benchmarks. Its like benchmarking Fizzbuzz or something. Frameworks don't do anything. No one chooses a framework (at least I don't) b

I really don't like these benchmarks. Its like benchmarking Fizzbuzz or something. Frameworks don't do anything. No one chooses a framework (at least I don't) based on performance. You choose one framework over the other because you like the API and/or language. I myself am a framework author (giotto, a python framework that was not included in these benchmarks). If my framework had been included, I'm sure it would end up dead last. When I built it, I wasn't thinking about performance, I was focusing on building a framework that would result in applications that are easy to understand/debug and fast to write.

deleted 13y ago
[deleted]
dllthomas 13y ago
I agree that performance shouldn't dominate the decision, but there's no reason not to be informed by it - it can wind up mattering.
- phillmv 13y ago
  Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.
  - mrgoldenbrown 13y ago
    This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.
    - phillmv 13y ago
      >But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it. (If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".) This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.
      - cmircea 13y ago
        Moore's law hasn't made it moot. Running in the cloud is pretty slow and extremely expensive. Look at StackExchange for example - they used to handle a LOT of traffic on a handful of servers. Even these benchmarks say (or said) that the EC2 instance used is waaay slower than an i7 2600K.
  - oberhamsi 13y ago
    I think you ran into your own argument: if you are very read heavy (and lots of big sites are), it's all about caching and the DB becomes irrelevant. QED really hard to get right
  - rallison 13y ago
    The general point of these benchmarks is not to resemble a full production app, but to provide a baseline measurement. From the original blog post[1]: This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. We aim to know the upper bound being set on an application's performance per unit of hardware by each platform and framework. But we also want to exercise some of the frameworks' components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it's a reasonable starting point. So, yes, these benchmarks should not be the only factor in choosing a framework, but they do provide a possibly important data point (depending on the specific scenario). [1] http://www.techempower.com/blog/2013/03/28/frameworks-round-1/ http://www.techempower.com/blog/2013/03/28/frameworks-round-...
  - papsosouid 13y ago
    Except your assumption is complete nonsense. The more non-trivial your app is, the less the database is a bottle neck and the more the app is. The vast majority of web apps are extremely read heavy. Those apps benefit massively from caching, which completely removes the database as a bottleneck. This means you are often choosing between a language and framework combo that means paying for 50 instances vs one that means paying for 4 instances. That is a lot of money, and the fallacious notion that the slower language is more productive by virtue of being slow is silly.
lucian1900 13y ago
Not sure why this is downvoted so badly, the comment is largely correct. This benchmark is even less useful than alioth's shootout, I'm not sure why there is so much effort put into it :)
- papsosouid 13y ago
  The comment is largely the authors opinion, it can be neither correct nor incorrect. It is very possible that what he feels is important and unimportant does not generalize to everyone else.
  - voidlogic 13y ago
    Not to mention that he assumes all the faster langs take more effort. What if you could have a lang that is a best performance and be highly productive- he assume this is not possible. The Go code size is pretty small, in fact it might be smaller than the Rails code... I'm still trying to find all the Ruby files, Go is in one file...
    - kbenson 13y ago
      I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. Not sure yet if there are smaller ones (note that there are no other files required for those apps, period) In the same vein, Lua's OpenResty[4] looks good, as do Tornado[5], Flask[6] and Bottle[7] (although you need to tease the raw/ORM methods apart to get an idea for the last two). And of course, Sinatra[8]. There probably a lot more, especially for PHP, but I didn't feel like going through that list. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/mojolicious/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/dancer/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/kelp/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [4]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/openresty/app.lua https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [5]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/tornado/server.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [6]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/flask/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [7]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/bottle/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [8]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/sinatra/hello_world.rb https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
reactor 13y ago
If you think about it, it matters. Not that I always want the top performer, but definitely wont pick the last few.

deleted 13y ago

[deleted]

dllthomas 13y ago

I agree that performance shouldn't dominate the decision, but there's no reason not to be informed by it - it can wind up mattering.

phillmv 13y ago
Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.
- mrgoldenbrown 13y ago
  This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.
  - phillmv 13y ago
    >But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it. (If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".) This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.
    - cmircea 13y ago
      Moore's law hasn't made it moot. Running in the cloud is pretty slow and extremely expensive. Look at StackExchange for example - they used to handle a LOT of traffic on a handful of servers. Even these benchmarks say (or said) that the EC2 instance used is waaay slower than an i7 2600K.
- oberhamsi 13y ago
  I think you ran into your own argument: if you are very read heavy (and lots of big sites are), it's all about caching and the DB becomes irrelevant. QED really hard to get right
- rallison 13y ago
  The general point of these benchmarks is not to resemble a full production app, but to provide a baseline measurement. From the original blog post[1]: This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. We aim to know the upper bound being set on an application's performance per unit of hardware by each platform and framework. But we also want to exercise some of the frameworks' components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it's a reasonable starting point. So, yes, these benchmarks should not be the only factor in choosing a framework, but they do provide a possibly important data point (depending on the specific scenario). [1] http://www.techempower.com/blog/2013/03/28/frameworks-round-1/ http://www.techempower.com/blog/2013/03/28/frameworks-round-...
- papsosouid 13y ago
  Except your assumption is complete nonsense. The more non-trivial your app is, the less the database is a bottle neck and the more the app is. The vast majority of web apps are extremely read heavy. Those apps benefit massively from caching, which completely removes the database as a bottleneck. This means you are often choosing between a language and framework combo that means paying for 50 instances vs one that means paying for 4 instances. That is a lot of money, and the fallacious notion that the slower language is more productive by virtue of being slow is silly.

lucian1900 13y ago

Not sure why this is downvoted so badly, the comment is largely correct. This benchmark is even less useful than alioth's shootout, I'm not sure why there is so much effort put into it :)

papsosouid 13y ago
The comment is largely the authors opinion, it can be neither correct nor incorrect. It is very possible that what he feels is important and unimportant does not generalize to everyone else.
- voidlogic 13y ago
  Not to mention that he assumes all the faster langs take more effort. What if you could have a lang that is a best performance and be highly productive- he assume this is not possible. The Go code size is pretty small, in fact it might be smaller than the Rails code... I'm still trying to find all the Ruby files, Go is in one file...
  - kbenson 13y ago
    I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. Not sure yet if there are smaller ones (note that there are no other files required for those apps, period) In the same vein, Lua's OpenResty[4] looks good, as do Tornado[5], Flask[6] and Bottle[7] (although you need to tease the raw/ORM methods apart to get an idea for the last two). And of course, Sinatra[8]. There probably a lot more, especially for PHP, but I didn't feel like going through that list. [1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/mojolicious/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/dancer/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [3]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/kelp/app.pl https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [4]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/openresty/app.lua https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [5]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/tornado/server.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [6]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/flask/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [7]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/bottle/app.py https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast... [8]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/sinatra/hello_world.rb https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

reactor 13y ago

If you think about it, it matters. Not that I always want the top performer, but definitely wont pick the last few.