5 ms·

Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivi

by phillmv 13y ago

Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app. For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.

mrgoldenbrown 13y ago
This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.
- phillmv 13y ago
  >But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it. (If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".) This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.
  - cmircea 13y ago
    Moore's law hasn't made it moot. Running in the cloud is pretty slow and extremely expensive. Look at StackExchange for example - they used to handle a LOT of traffic on a handful of servers. Even these benchmarks say (or said) that the EC2 instance used is waaay slower than an i7 2600K.
oberhamsi 13y ago
I think you ran into your own argument: if you are very read heavy (and lots of big sites are), it's all about caching and the DB becomes irrelevant. QED really hard to get right
rallison 13y ago
The general point of these benchmarks is not to resemble a full production app, but to provide a baseline measurement. From the original blog post[1]: This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. We aim to know the upper bound being set on an application's performance per unit of hardware by each platform and framework. But we also want to exercise some of the frameworks' components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it's a reasonable starting point. So, yes, these benchmarks should not be the only factor in choosing a framework, but they do provide a possibly important data point (depending on the specific scenario). [1] http://www.techempower.com/blog/2013/03/28/frameworks-round-1/ http://www.techempower.com/blog/2013/03/28/frameworks-round-...
papsosouid 13y ago
Except your assumption is complete nonsense. The more non-trivial your app is, the less the database is a bottle neck and the more the app is. The vast majority of web apps are extremely read heavy. Those apps benefit massively from caching, which completely removes the database as a bottleneck. This means you are often choosing between a language and framework combo that means paying for 50 instances vs one that means paying for 4 instances. That is a lot of money, and the fallacious notion that the slower language is more productive by virtue of being slow is silly.

by phillmv 13y ago

mrgoldenbrown 13y ago

This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.

phillmv 13y ago
>But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it. (If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".) This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.
- cmircea 13y ago
  Moore's law hasn't made it moot. Running in the cloud is pretty slow and extremely expensive. Look at StackExchange for example - they used to handle a LOT of traffic on a handful of servers. Even these benchmarks say (or said) that the EC2 instance used is waaay slower than an i7 2600K.

oberhamsi 13y ago

I think you ran into your own argument: if you are very read heavy (and lots of big sites are), it's all about caching and the DB becomes irrelevant. QED really hard to get right

rallison 13y ago

The general point of these benchmarks is not to resemble a full production app, but to provide a baseline measurement. From the original blog post[1]: This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. We aim to know the upper bound being set on an application's performance per unit of hardware by each platform and framework. But we also want to exercise some of the frameworks' components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it's a reasonable starting point. So, yes, these benchmarks should not be the only factor in choosing a framework, but they do provide a possibly important data point (depending on the specific scenario). [1] http://www.techempower.com/blog/2013/03/28/frameworks-round-1/ http://www.techempower.com/blog/2013/03/28/frameworks-round-...

papsosouid 13y ago

Except your assumption is complete nonsense. The more non-trivial your app is, the less the database is a bottle neck and the more the app is. The vast majority of web apps are extremely read heavy. Those apps benefit massively from caching, which completely removes the database as a bottleneck. This means you are often choosing between a language and framework combo that means paying for 50 instances vs one that means paying for 4 instances. That is a lot of money, and the fallacious notion that the slower language is more productive by virtue of being slow is silly.