4 ms·

Ask HN: Should techempower.com framework benchmarks be taken seriously

4 points by wilsonfiifi 13y ago

Looking at the results for "Tornado" in the "Multiple queries" test of "Round 8", does this mean we have to ditch it in favour of "Flask" for example? Or should we take these results with a pinch of salt?

hardwaresofton 13y ago
You should take any results anywhere with a pinch of salt. However, it's not that benchmarks are meaningless -- review their methods. If you can think of a way to test that they're not following, or find some flaw in their testing methodology, then tell them about it, and have them fix it (or post what's wrong somewhere), but they are pretty up front about HOW they test, so results should not surprise you, given enough knowledge of the frameworks involved (if they test something that flask is great at and tornado is not, that will be obvious given knowledge of how the test was conducted)
eip 13y ago
Why would you switch from something at the very bottom of the list to something in the middle of the list? Why are you even using Python if you are worried about performance?
- wilsonfiifi 13y ago
  Python, IMHO, has a great collection of modules/frameworks for web application development (WTForms, SQLAlchemy, Jinja2, etc) so I'd prefer to have the "rendering/logic part" of the web application written in it. The performance challenges can be overcome by hooking up Python to "workers" written in Go, for instance, using Zeromq. However I would have liked to take advantage of Tornado's asynchronous requests handling thus my disappointment with its performance in the said benchmarks. I wanted to avoid using "uwsgi" with Flask because of an already complex technology stack i.e: Nginx + uwsgi + Flask + zeromq + Go + sql/nosql database(s)
- workhere-io 13y ago
  Perhaps the question should be, "Why are you even that worried about performance?" Instagram and Pinterest, two extremely high-traffic sites, run on Python. And as always, a little caching goes a long way.
kbenson 13y ago
The benchmarks optimize along one, or at least a very few, axis. There are many things of varying importance when developing your system/site, and they will most likely be different than anyone else's. For example, your design may scale well across multiple systems, and developer time is at a premium, so squeezing the most performance out of the given hardware may be much less important than increasing programmer productivity. In those cases, a higher level language with a reduced development time may be a better fit. Since the implementations in each language/framework are all available, you can look into the few that seem the most interesting base don your own criteria and get a quick idea of whether you think they would be a good fit for your criteria. In that respect, I think the benchmarks can be taken seriously, as long as you don't their rankings to be an indicator of what's best. That is, don't consider it a benchmark, consider it a very thorough survey of available frameworks.

4 points by wilsonfiifi 13y ago

hardwaresofton 13y ago

You should take any results anywhere with a pinch of salt. However, it's not that benchmarks are meaningless -- review their methods. If you can think of a way to test that they're not following, or find some flaw in their testing methodology, then tell them about it, and have them fix it (or post what's wrong somewhere), but they are pretty up front about HOW they test, so results should not surprise you, given enough knowledge of the frameworks involved (if they test something that flask is great at and tornado is not, that will be obvious given knowledge of how the test was conducted)

eip 13y ago

Why would you switch from something at the very bottom of the list to something in the middle of the list? Why are you even using Python if you are worried about performance?

wilsonfiifi 13y ago
Python, IMHO, has a great collection of modules/frameworks for web application development (WTForms, SQLAlchemy, Jinja2, etc) so I'd prefer to have the "rendering/logic part" of the web application written in it. The performance challenges can be overcome by hooking up Python to "workers" written in Go, for instance, using Zeromq. However I would have liked to take advantage of Tornado's asynchronous requests handling thus my disappointment with its performance in the said benchmarks. I wanted to avoid using "uwsgi" with Flask because of an already complex technology stack i.e: Nginx + uwsgi + Flask + zeromq + Go + sql/nosql database(s)
workhere-io 13y ago
Perhaps the question should be, "Why are you even that worried about performance?" Instagram and Pinterest, two extremely high-traffic sites, run on Python. And as always, a little caching goes a long way.

kbenson 13y ago

The benchmarks optimize along one, or at least a very few, axis. There are many things of varying importance when developing your system/site, and they will most likely be different than anyone else's. For example, your design may scale well across multiple systems, and developer time is at a premium, so squeezing the most performance out of the given hardware may be much less important than increasing programmer productivity. In those cases, a higher level language with a reduced development time may be a better fit. Since the implementations in each language/framework are all available, you can look into the few that seem the most interesting base don your own criteria and get a quick idea of whether you think they would be a good fit for your criteria. In that respect, I think the benchmarks can be taken seriously, as long as you don't their rankings to be an indicator of what's best. That is, don't consider it a benchmark, consider it a very thorough survey of available frameworks.