3 ms·

I'm not familiar with this site and benchmarks, but from reading comments it seems to be respected. Given that, I'm confused why this is called "web framework"

by blunte 8y ago

I'm not familiar with this site and benchmarks, but from reading comments it seems to be respected. Given that, I'm confused why this is called "web framework" benchmarks. It looks to me like it is comparing some actual frameworks (which rank very poorly) against minimalist, task-focused http servers (which rank highly).

aninteger 8y ago
Yes, exactly. I'd love to see some examples of a CRUD based API built on top of h2o. At least this appears to be possible with one of the higher performers (cppcms).
- ksec 8y ago
  Interesting I just submitted h20 2.3 Beta, but it seems to be not getting much interest on HN. It could have been called 3.0 with so many new features.
spullara 8y ago
I think that it allows you to evaluate the tradeoff between DIY services on raw http servers and using a framework that reduces development time but is much slower in production.
bhauer 8y ago
We use the word "framework" as a term of convenience covering the full spectrum from platforms, micro-frameworks, to full-stack frameworks. We are also liberal with accepting contributions from the community, which means we do indeed include several frameworks that are well outside the mainstream. That said, in tests such as Fortunes, you will see many full-stack frameworks demonstrating their capability to deliver myriad web application fundamentals (such as request routing, database connection pooling, ORM, XSS countermeasures, character encoding, data structures, and server-side templates) at very high performance levels. This project establishes a high-water mark of performance. Very few web applications process anything remotely close to the requests per second we're measuring. But using Fortunes as a proxy and applying a coefficient (e.g., 0.005) to adjust for real-world application sizing can give you a very rough but nevertheless potentially useful approximation of real-world expectations. For example, one can guess their application is 200 times more complex than our Fortunes test. Rough back-of-the-envelope math will then estimate a framework from the ultra-high performance tier (~200,000 fortunes/sec) might yield a generous ~1,000 real-world app requests per second on a modern Xeon server. Meanwhile, a framework in the middle tier (~10,000 fortunes/sec) would yield a more constraining ~50 real-world app requests per second. Again, this math is hand-waving and you can interpret the results in whatever way you prefer; you can dismiss the validity of approximating things so coarsely. But my experience is that real-world applications based on frameworks I've used—which are sprinkled among all tiers of our results—do align with this hand-waving approximation. Edit: I invite you to read the last section of the blog entry about this round [2] where I argue the same point in another way and share one of the tweets from an application developer whose real world application benefited from the performance improvements made to his favorite framework. [1] https://www.techempower.com/benchmarks/#section=motivation https://www.techempower.com/benchmarks/#section=motivation [2] https://www.techempower.com/blog/2018/06/06/framework-benchmarks-round-16/ https://www.techempower.com/blog/2018/06/06/framework-benchm...
- blunte 8y ago
  Thanks for the explanation. I see your reasoning. If there were a (asterisk) in the title: Web Framework Benchmarks (asterisk link), with the asterisk taking you to a page with something like your explanation above, that would be lovely.

by blunte 8y ago

aninteger 8y ago

Yes, exactly. I'd love to see some examples of a CRUD based API built on top of h2o. At least this appears to be possible with one of the higher performers (cppcms).

ksec 8y ago
Interesting I just submitted h20 2.3 Beta, but it seems to be not getting much interest on HN. It could have been called 3.0 with so many new features.

spullara 8y ago

I think that it allows you to evaluate the tradeoff between DIY services on raw http servers and using a framework that reduces development time but is much slower in production.

bhauer 8y ago

We use the word "framework" as a term of convenience covering the full spectrum from platforms, micro-frameworks, to full-stack frameworks. We are also liberal with accepting contributions from the community, which means we do indeed include several frameworks that are well outside the mainstream. That said, in tests such as Fortunes, you will see many full-stack frameworks demonstrating their capability to deliver myriad web application fundamentals (such as request routing, database connection pooling, ORM, XSS countermeasures, character encoding, data structures, and server-side templates) at very high performance levels. This project establishes a high-water mark of performance. Very few web applications process anything remotely close to the requests per second we're measuring. But using Fortunes as a proxy and applying a coefficient (e.g., 0.005) to adjust for real-world application sizing can give you a very rough but nevertheless potentially useful approximation of real-world expectations. For example, one can guess their application is 200 times more complex than our Fortunes test. Rough back-of-the-envelope math will then estimate a framework from the ultra-high performance tier (~200,000 fortunes/sec) might yield a generous ~1,000 real-world app requests per second on a modern Xeon server. Meanwhile, a framework in the middle tier (~10,000 fortunes/sec) would yield a more constraining ~50 real-world app requests per second. Again, this math is hand-waving and you can interpret the results in whatever way you prefer; you can dismiss the validity of approximating things so coarsely. But my experience is that real-world applications based on frameworks I've used—which are sprinkled among all tiers of our results—do align with this hand-waving approximation. Edit: I invite you to read the last section of the blog entry about this round [2] where I argue the same point in another way and share one of the tweets from an application developer whose real world application benefited from the performance improvements made to his favorite framework. [1] https://www.techempower.com/benchmarks/#section=motivation https://www.techempower.com/benchmarks/#section=motivation [2] https://www.techempower.com/blog/2018/06/06/framework-benchmarks-round-16/ https://www.techempower.com/blog/2018/06/06/framework-benchm...

blunte 8y ago
Thanks for the explanation. I see your reasoning. If there were a (asterisk) in the title: Web Framework Benchmarks (asterisk link), with the asterisk taking you to a page with something like your explanation above, that would be lovely.